2024-09-15T21:37:04.115 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T21:37:04.142 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T21:37:04.234 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906192' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 8921c8ef947807b717db041d85f0761b7914d5fc subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3393 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 294/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAVlUoCfTS9SogpUZ/+1TuQZ9K/xPGK+FJTo6zkFYNx82xzH5jfadNLgYsm3Ne+1w9RRktm/DrOMxOMxTKTjwXQ= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrGXIHKIwtq0KBCxYVaN8wCxFBDlPLTB9G2Q3xoW+1kMEkEYTtfkInNnTG+gB3HssZf0gs0hxXtkRKOAwwWLQw= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIohUzGDfOiMCOHvjiQ0NGe1SpTg1dgeyX0dd3q6zr4C1Mjnp/DzrYJ0296jhS8QIWyPOpYfJToUiqc3gF3ohTw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T21:37:04.234 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-15T21:37:04.235 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-15T21:37:04.235 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T21:37:04.236 INFO:teuthology.task.internal:Checking packages... 2024-09-15T21:37:04.259 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-15T21:37:04.259 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T21:37:04.259 INFO:teuthology.packaging:ref: None 2024-09-15T21:37:04.259 INFO:teuthology.packaging:tag: None 2024-09-15T21:37:04.259 INFO:teuthology.packaging:branch: squid 2024-09-15T21:37:04.259 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:37:04.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-15T21:37:04.460 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-326.g8921c8ef 2024-09-15T21:37:04.462 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T21:37:04.468 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T21:37:04.468 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T21:37:04.528 INFO:teuthology.task.internal:Saving configuration 2024-09-15T21:37:04.549 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T21:37:04.555 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T21:37:04.582 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 21:32:00.189168', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAVlUoCfTS9SogpUZ/+1TuQZ9K/xPGK+FJTo6zkFYNx82xzH5jfadNLgYsm3Ne+1w9RRktm/DrOMxOMxTKTjwXQ='} 2024-09-15T21:37:04.602 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 21:32:00.185905', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrGXIHKIwtq0KBCxYVaN8wCxFBDlPLTB9G2Q3xoW+1kMEkEYTtfkInNnTG+gB3HssZf0gs0hxXtkRKOAwwWLQw='} 2024-09-15T21:37:04.625 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 21:32:00.187669', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIohUzGDfOiMCOHvjiQ0NGe1SpTg1dgeyX0dd3q6zr4C1Mjnp/DzrYJ0296jhS8QIWyPOpYfJToUiqc3gF3ohTw='} 2024-09-15T21:37:04.625 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T21:37:04.632 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-15T21:37:04.632 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-15T21:37:04.632 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-15T21:37:04.632 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T21:37:04.722 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0abd157760>, signals=[15]) 2024-09-15T21:37:04.723 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T21:37:04.729 INFO:teuthology.task.internal:Opening connections... 2024-09-15T21:37:04.729 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:37:04.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:37:04.819 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-09-15T21:37:04.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:37:04.898 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T21:37:04.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:37:04.974 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T21:37:04.982 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-09-15T21:37:05.000 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-09-15T21:37:05.000 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T21:37:05.058 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-09-15T21:37:05.059 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-09-15T21:37:05.059 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T21:37:05.059 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-09-15T21:37:05.059 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T21:37:05.059 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T21:37:05.059 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T21:37:05.059 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-09-15T21:37:05.083 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-09-15T21:37:05.101 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-09-15T21:37:05.101 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T21:37:05.160 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-09-15T21:37:05.161 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T21:37:05.161 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T21:37:05.161 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T21:37:05.161 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-09-15T21:37:05.182 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-09-15T21:37:05.201 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-09-15T21:37:05.201 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-09-15T21:37:05.258 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-09-15T21:37:05.258 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T21:37:05.259 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T21:37:05.260 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-09-15T21:37:05.283 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T21:37:05.294 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T21:37:05.299 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T21:37:05.299 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-09-15T21:37:05.301 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-09-15T21:37:05.303 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-09-15T21:37:05.320 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T21:37:05.326 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T21:37:05.326 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-09-15T21:37:05.359 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-09-15T21:37:05.361 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-09-15T21:37:05.453 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T21:37:05.519 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T21:37:05.533 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-15T21:37:05.533 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-15T21:37:05.534 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-15T21:37:05.534 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T21:37:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:37:05.551 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-15T21:37:05.609 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-503.el9.x86_64 2024-09-15T21:37:05.609 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-503.el9.x86_64 2024-09-15T21:37:05.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-09-15T21:37:07.897 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-15T21:37:09.782 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-15T21:37:15.441 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-15T21:37:17.083 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 19 kB 00:00 2024-09-15T21:37:18.179 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T21:37:23.743 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T21:37:25.824 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T21:37:25.825 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T21:37:25.922 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-15T21:37:25.927 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-15T21:37:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-09-15T21:37:25.927 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-09-15T21:37:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:37:25.929 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 79 M 2024-09-15T21:37:25.929 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 126 M 2024-09-15T21:37:25.929 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-15T21:37:26.153 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 235 kB/s | 23 kB 00:00 2024-09-15T21:37:26.728 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 50 MB/s | 29 MB 00:00 2024-09-15T21:37:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-09-15T21:37:27.520 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 24 MB/s | 35 MB 00:01 2024-09-15T21:37:27.520 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:37:27.521 INFO:teuthology.orchestra.run.smithi028.stdout:Total 49 MB/s | 79 MB 00:01 2024-09-15T21:37:28.234 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-15T21:37:28.614 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-15T21:37:28.614 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-15T21:37:29.622 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-15T21:37:29.623 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-15T21:37:32.191 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-15T21:37:32.846 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T21:37:32.929 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T21:37:35.709 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T21:37:35.831 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T21:37:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T21:37:41.016 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T21:37:46.157 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T21:38:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T21:38:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T21:38:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T21:38:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T21:38:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T21:38:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T21:38:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:38:26.375 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-15T21:38:26.691 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T21:38:26.777 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T21:38:26.777 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T21:38:26.777 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T21:38:26.778 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T21:38:26.778 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T21:38:26.778 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-15T21:38:26.778 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T21:38:26.778 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T21:38:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T21:38:26.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T21:38:26.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-09-15T21:38:26.869 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-09-15T21:38:26.869 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 82.el9 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8274670 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T21:38:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:38:26.871 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T21:38:26.872 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T21:38:26.872 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-505.el9.x86_64 2024-09-15T21:38:26.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T21:38:27.497 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-09-15T21:38:28.870 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T21:38:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-09-15T21:38:28.896 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-15T21:38:28.912 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.frX7Oatevt 2024-09-15T21:38:28.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.frX7Oatevt 2024-09-15T21:38:28.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.frX7Oatevt 2024-09-15T21:38:29.125 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.frX7Oatevt is 6KB 2024-09-15T21:38:29.174 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.frX7Oatevt 2024-09-15T21:38:29.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T21:38:29.259 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T21:38:29.260 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T21:38:29.260 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T21:38:29.260 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T21:38:29.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T21:38:29.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-09-15T21:38:59.429 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-15T21:38:59.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-15T21:38:59.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:39:59.433 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T21:40:08.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-15T21:40:08.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:40:08.788 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-15T21:40:09.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-15T21:40:09.145 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-15T21:40:09.145 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-15T21:40:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-505.el9.x86_64 2024-09-15T21:40:09.162 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T21:40:09.162 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T21:40:09.162 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T21:40:10.163 DEBUG:teuthology.parallel:result is None 2024-09-15T21:40:10.164 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T21:40:10.807 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T21:40:10.807 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T21:40:10.811 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T21:40:10.813 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T21:40:10.832 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T21:40:11.050 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T21:40:11.198 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T21:40:11.199 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T21:40:11.201 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T21:40:11.204 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T21:40:11.368 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T21:40:11.462 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T21:40:11.463 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T21:40:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:40:11.481 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T21:40:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:40:11.499 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T21:40:11.502 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T21:40:11.524 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T21:40:11.580 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:40:11.586 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:40:11.599 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:40:11.601 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:40:11.608 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:40:11.619 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:40:11.621 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T21:40:11.627 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T21:40:11.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T21:40:11.630 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T21:40:11.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T21:40:11.660 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T21:40:11.669 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T21:40:11.670 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T21:40:11.699 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T21:40:11.702 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T21:40:11.718 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T21:40:11.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T21:40:11.851 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T21:40:11.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T21:40:11.978 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:40:11.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T21:40:12.045 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T21:40:12.091 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T21:40:12.151 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T21:40:12.196 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T21:40:12.239 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:40:12.239 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T21:40:12.307 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T21:40:12.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T21:40:12.410 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T21:40:12.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T21:40:12.537 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:40:12.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T21:40:12.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-09-15T21:40:12.607 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-09-15T21:40:12.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-09-15T21:40:12.643 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T21:40:12.657 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T21:40:12.683 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T21:40:13.172 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T21:40:13.179 INFO:teuthology.task.internal:Starting timer... 2024-09-15T21:40:13.179 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T21:40:13.189 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T21:40:13.198 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-15T21:40:13.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-09-15T21:40:13.256 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-09-15T21:40:13.258 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-09-15T21:40:13.314 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-09-15T21:40:13.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-09-15T21:40:13.371 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-09-15T21:40:13.373 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T21:40:13.373 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-09-15T21:40:13.394 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-09-15T21:40:13.395 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-09-15T21:40:13.426 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-09-15T21:40:13.426 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-09-15T21:40:13.451 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-09-15T21:40:13.451 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-09-15T21:40:13.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T21:40:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:40:13.482 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T21:40:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:40:13.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T21:40:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:40:13.538 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T21:40:13.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-09-15T21:40:13.570 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-09-15T21:40:13.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-09-15T21:40:13.634 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T21:40:13.644 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T21:40:13.647 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T21:40:13.760 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T21:40:13.776 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T21:40:13.777 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi174.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T21:44:17.599 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-09-15T21:44:17.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-15T21:44:17.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:44:17.684 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-15T21:44:17.779 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-15T21:44:17.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-15T21:44:17.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:44:17.859 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-15T21:44:17.938 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-15T21:44:17.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T21:44:17.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T21:44:18.017 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-15T21:44:18.101 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T21:44:18.102 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T21:44:18.112 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T21:44:18.112 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T21:44:18.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T21:44:18.115 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T21:44:18.116 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T21:44:18.118 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T21:44:18.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T21:44:18.148 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T21:44:18.150 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T21:44:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T21:44:18.166 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T21:44:18.173 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T21:44:18.188 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T21:44:18.192 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-09-15T21:44:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-09-15T21:44:18.204 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-09-15T21:44:18.205 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-09-15T21:44:18.214 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-09-15T21:44:18.219 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T21:44:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T21:44:18.225 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-09-15T21:44:18.233 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T21:44:18.236 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T21:44:18.240 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T21:44:18.254 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T21:44:18.286 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-09-15T21:44:18.289 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T21:44:18.290 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-09-15T21:44:18.290 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.290 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.290 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.290 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.291 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-15T21:44:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T21:44:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-15T21:44:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.316 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-09-15T21:44:18.319 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T21:44:18.319 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-09-15T21:44:18.320 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.320 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.320 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.320 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:44:18.321 INFO:teuthology.run_tasks:Running task install... 2024-09-15T21:44:18.329 DEBUG:teuthology.task.install:project ceph 2024-09-15T21:44:18.329 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-15T21:44:18.329 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-15T21:44:18.329 INFO:teuthology.task.install:Using flavor: default 2024-09-15T21:44:18.335 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T21:44:18.335 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-15T21:44:18.335 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T21:44:18.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:44:18.338 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T21:44:18.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:44:18.339 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T21:44:18.339 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:44:18.482 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T21:44:18.482 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T21:44:18.490 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T21:44:18.491 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T21:44:18.505 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T21:44:18.506 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T21:44:18.594 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T21:44:18.594 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:44:18.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T21:44:18.600 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T21:44:18.601 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:44:18.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T21:44:18.610 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T21:44:18.611 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:44:18.611 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T21:44:18.622 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T21:44:18.622 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T21:44:18.627 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T21:44:18.627 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T21:44:18.637 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T21:44:18.637 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T21:44:18.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T21:44:18.693 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T21:44:18.703 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T21:44:18.767 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T21:44:18.768 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T21:44:18.778 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T21:44:18.832 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-09-15T21:44:18.833 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-09-15T21:44:18.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-09-15T21:44:18.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-09-15T21:44:18.842 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-09-15T21:44:18.843 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-09-15T21:44:19.059 INFO:teuthology.orchestra.run.smithi203.stdout:45 files removed 2024-09-15T21:44:19.066 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-09-15T21:44:19.069 INFO:teuthology.orchestra.run.smithi174.stdout:45 files removed 2024-09-15T21:44:19.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T21:44:19.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T21:44:19.097 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T21:44:19.632 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 312 kB/s | 83 kB 00:00 2024-09-15T21:44:19.643 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 308 kB/s | 83 kB 00:00 2024-09-15T21:44:19.656 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 317 kB/s | 83 kB 00:00 2024-09-15T21:44:19.900 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 52 kB/s | 12 kB 00:00 2024-09-15T21:44:19.919 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 50 kB/s | 12 kB 00:00 2024-09-15T21:44:19.949 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-15T21:44:20.167 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-15T21:44:20.186 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-15T21:44:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-15T21:44:20.870 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-15T21:44:21.017 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-15T21:44:21.924 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-15T21:44:23.068 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-15T21:44:23.637 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-15T21:44:24.175 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-15T21:44:27.278 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-15T21:44:28.097 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-15T21:44:29.004 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-15T21:44:29.572 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 19 kB 00:00 2024-09-15T21:44:30.301 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.5 MB 00:02 2024-09-15T21:44:30.742 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-15T21:44:30.885 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-15T21:44:31.810 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 19 kB 00:00 2024-09-15T21:44:33.231 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-15T21:44:36.123 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T21:44:36.416 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-09-15T21:44:38.425 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T21:44:38.426 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T21:44:38.437 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T21:44:38.438 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T21:44:38.439 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T21:44:38.440 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T21:44:38.442 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T21:44:38.444 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T21:44:38.445 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T21:44:38.508 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-15T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-09-15T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-15T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-09-15T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-15T21:44:38.529 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T21:44:38.530 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T21:44:38.531 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T21:44:38.532 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T21:44:38.533 INFO:teuthology.orchestra.run.smithi203.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-15T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T21:44:38.534 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T21:44:38.535 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T21:44:38.536 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T21:44:38.537 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T21:44:38.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T21:44:38.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T21:44:38.540 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T21:44:38.541 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T21:44:38.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T21:44:38.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T21:44:38.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T21:44:38.545 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T21:44:38.546 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T21:44:38.547 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-09-15T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-15T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-09-15T21:44:38.548 INFO:teuthology.orchestra.run.smithi203.stdout:Install 148 Packages 2024-09-15T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 8 Packages 2024-09-15T21:44:38.549 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 226 M 2024-09-15T21:44:38.550 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-15T21:44:38.745 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T21:44:38.745 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T21:44:38.759 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T21:44:38.760 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T21:44:38.761 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T21:44:38.762 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T21:44:38.764 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T21:44:38.767 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T21:44:38.768 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T21:44:38.833 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-15T21:44:38.836 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T21:44:38.849 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-15T21:44:38.849 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-15T21:44:38.849 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-15T21:44:38.849 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-15T21:44:38.849 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T21:44:38.850 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T21:44:38.850 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T21:44:38.850 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T21:44:38.850 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T21:44:38.850 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T21:44:38.851 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T21:44:38.851 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T21:44:38.851 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T21:44:38.851 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T21:44:38.851 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T21:44:38.851 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T21:44:38.852 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T21:44:38.853 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T21:44:38.853 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T21:44:38.853 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T21:44:38.853 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T21:44:38.854 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T21:44:38.855 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T21:44:38.856 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T21:44:38.857 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T21:44:38.858 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T21:44:38.858 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T21:44:38.858 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T21:44:38.859 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T21:44:38.860 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T21:44:38.861 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T21:44:38.862 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T21:44:38.863 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T21:44:38.863 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T21:44:38.863 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T21:44:38.863 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T21:44:38.863 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T21:44:38.863 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T21:44:38.863 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T21:44:38.864 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T21:44:38.864 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T21:44:38.864 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T21:44:38.864 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T21:44:38.864 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T21:44:38.864 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T21:44:38.865 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T21:44:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T21:44:38.867 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T21:44:38.867 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T21:44:38.867 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T21:44:38.867 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T21:44:38.867 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T21:44:38.867 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T21:44:38.867 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T21:44:38.868 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T21:44:38.868 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T21:44:38.868 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T21:44:38.868 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T21:44:38.868 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T21:44:38.868 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T21:44:38.868 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:44:38.869 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-15T21:44:38.870 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-15T21:44:38.870 INFO:teuthology.orchestra.run.smithi174.stdout:Install 148 Packages 2024-09-15T21:44:38.870 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 8 Packages 2024-09-15T21:44:38.870 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:44:38.871 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 226 M 2024-09-15T21:44:38.871 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-15T21:44:39.258 INFO:teuthology.orchestra.run.smithi203.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 45 kB/s | 6.5 kB 00:00 2024-09-15T21:44:39.517 INFO:teuthology.orchestra.run.smithi203.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 3.4 MB/s | 896 kB 00:00 2024-09-15T21:44:39.600 INFO:teuthology.orchestra.run.smithi203.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.7 MB/s | 145 kB 00:00 2024-09-15T21:44:39.842 INFO:teuthology.orchestra.run.smithi174.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 29 kB/s | 6.5 kB 00:00 2024-09-15T21:44:40.009 INFO:teuthology.orchestra.run.smithi203.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 6.0 MB/s | 5.4 MB 00:00 2024-09-15T21:44:40.059 INFO:teuthology.orchestra.run.smithi203.stdout:(5/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 5.2 MB/s | 2.4 MB 00:00 2024-09-15T21:44:40.201 INFO:teuthology.orchestra.run.smithi203.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 7.9 MB/s | 1.5 MB 00:00 2024-09-15T21:44:40.551 INFO:teuthology.orchestra.run.smithi174.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 1.2 MB/s | 896 kB 00:00 2024-09-15T21:44:40.701 INFO:teuthology.orchestra.run.smithi203.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 7.3 MB/s | 4.7 MB 00:00 2024-09-15T21:44:40.709 INFO:teuthology.orchestra.run.smithi174.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 917 kB/s | 145 kB 00:00 2024-09-15T21:44:41.332 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T21:44:41.333 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T21:44:41.345 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T21:44:41.346 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T21:44:41.347 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T21:44:41.348 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T21:44:41.349 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T21:44:41.353 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T21:44:41.354 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T21:44:41.421 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-15T21:44:41.449 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-15T21:44:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-15T21:44:41.449 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-15T21:44:41.449 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-15T21:44:41.449 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T21:44:41.450 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T21:44:41.451 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T21:44:41.452 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T21:44:41.453 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T21:44:41.454 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T21:44:41.455 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T21:44:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T21:44:41.457 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T21:44:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T21:44:41.459 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T21:44:41.460 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T21:44:41.461 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T21:44:41.462 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T21:44:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T21:44:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T21:44:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T21:44:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T21:44:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T21:44:41.463 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-09-15T21:44:41.463 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T21:44:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:44:41.464 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-15T21:44:41.464 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-15T21:44:41.464 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-09-15T21:44:41.464 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 8 Packages 2024-09-15T21:44:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:44:41.465 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 226 M 2024-09-15T21:44:41.465 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-15T21:44:41.627 INFO:teuthology.orchestra.run.smithi203.stdout:(8/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 17 MB 00:01 2024-09-15T21:44:41.651 INFO:teuthology.orchestra.run.smithi174.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 2.5 MB/s | 2.4 MB 00:00 2024-09-15T21:44:41.752 INFO:teuthology.orchestra.run.smithi203.stdout:(9/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 10 MB/s | 11 MB 00:01 2024-09-15T21:44:41.768 INFO:teuthology.orchestra.run.smithi174.stdout:(5/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 2.5 MB/s | 5.4 MB 00:02 2024-09-15T21:44:41.777 INFO:teuthology.orchestra.run.smithi203.stdout:(10/156): ceph-selinux-19.1.1-326.g8921c8ef.el9 168 kB/s | 25 kB 00:00 2024-09-15T21:44:41.861 INFO:teuthology.orchestra.run.smithi203.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 2.7 MB/s | 228 kB 00:00 2024-09-15T21:44:41.945 INFO:teuthology.orchestra.run.smithi203.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 324 kB/s | 27 kB 00:00 2024-09-15T21:44:41.993 INFO:teuthology.orchestra.run.smithi174.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 4.5 MB/s | 1.5 MB 00:00 2024-09-15T21:44:42.003 INFO:teuthology.orchestra.run.smithi203.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 543 kB/s | 32 kB 00:00 2024-09-15T21:44:42.212 INFO:teuthology.orchestra.run.smithi203.stdout:(14/156): ceph-common-19.1.1-326.g8921c8ef.el9. 6.8 MB/s | 21 MB 00:03 2024-09-15T21:44:42.245 INFO:teuthology.orchestra.run.smithi203.stdout:(15/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 3.0 MB/s | 751 kB 00:00 2024-09-15T21:44:42.304 INFO:teuthology.orchestra.run.smithi203.stdout:(16/156): libcephsqlite-19.1.1-326.g8921c8ef.el 1.7 MB/s | 163 kB 00:00 2024-09-15T21:44:42.329 INFO:teuthology.orchestra.run.smithi028.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2024-09-15T21:44:42.346 INFO:teuthology.orchestra.run.smithi203.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 1.2 MB/s | 127 kB 00:00 2024-09-15T21:44:42.593 INFO:teuthology.orchestra.run.smithi174.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 5.7 MB/s | 4.7 MB 00:00 2024-09-15T21:44:42.897 INFO:teuthology.orchestra.run.smithi203.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 855 kB/s | 506 kB 00:00 2024-09-15T21:44:43.005 INFO:teuthology.orchestra.run.smithi203.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 417 kB/s | 45 kB 00:00 2024-09-15T21:44:43.089 INFO:teuthology.orchestra.run.smithi203.stdout:(20/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 7.2 MB/s | 5.3 MB 00:00 2024-09-15T21:44:43.114 INFO:teuthology.orchestra.run.smithi203.stdout:(21/156): python3-ceph-common-19.1.1-326.g8921c 1.2 MB/s | 136 kB 00:00 2024-09-15T21:44:43.157 INFO:teuthology.orchestra.run.smithi203.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 2.4 MB/s | 161 kB 00:00 2024-09-15T21:44:43.172 INFO:teuthology.orchestra.run.smithi028.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 1.0 MB/s | 896 kB 00:00 2024-09-15T21:44:43.181 INFO:teuthology.orchestra.run.smithi203.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 4.7 MB/s | 320 kB 00:00 2024-09-15T21:44:43.240 INFO:teuthology.orchestra.run.smithi203.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.7 MB/s | 99 kB 00:00 2024-09-15T21:44:43.265 INFO:teuthology.orchestra.run.smithi203.stdout:(25/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 2.7 MB/s | 300 kB 00:00 2024-09-15T21:44:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.4 MB/s | 145 kB 00:00 2024-09-15T21:44:43.299 INFO:teuthology.orchestra.run.smithi203.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.4 MB/s | 85 kB 00:00 2024-09-15T21:44:43.376 INFO:teuthology.orchestra.run.smithi203.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.2 MB/s | 170 kB 00:00 2024-09-15T21:44:43.457 INFO:teuthology.orchestra.run.smithi203.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 391 kB/s | 32 kB 00:00 2024-09-15T21:44:43.557 INFO:teuthology.orchestra.run.smithi203.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.4 MB/s | 146 kB 00:00 2024-09-15T21:44:43.927 INFO:teuthology.orchestra.run.smithi174.stdout:(8/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 8.0 MB/s | 11 MB 00:01 2024-09-15T21:44:44.027 INFO:teuthology.orchestra.run.smithi174.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 253 kB/s | 25 kB 00:00 2024-09-15T21:44:44.216 INFO:teuthology.orchestra.run.smithi203.stdout:(30/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 3.3 MB/s | 3.1 MB 00:00 2024-09-15T21:44:44.405 INFO:teuthology.orchestra.run.smithi028.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 2.1 MB/s | 2.4 MB 00:01 2024-09-15T21:44:44.424 INFO:teuthology.orchestra.run.smithi203.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 4.1 MB/s | 3.6 MB 00:00 2024-09-15T21:44:44.516 INFO:teuthology.orchestra.run.smithi203.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 2.7 MB/s | 249 kB 00:00 2024-09-15T21:44:44.608 INFO:teuthology.orchestra.run.smithi203.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 537 kB/s | 49 kB 00:00 2024-09-15T21:44:44.692 INFO:teuthology.orchestra.run.smithi203.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 197 kB/s | 16 kB 00:00 2024-09-15T21:44:44.784 INFO:teuthology.orchestra.run.smithi203.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 3.0 MB/s | 285 kB 00:00 2024-09-15T21:44:45.076 INFO:teuthology.orchestra.run.smithi203.stdout:(36/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 2.6 MB/s | 765 kB 00:00 2024-09-15T21:44:45.097 INFO:teuthology.orchestra.run.smithi028.stdout:(5/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-09-15T21:44:45.168 INFO:teuthology.orchestra.run.smithi203.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 199 kB/s | 18 kB 00:00 2024-09-15T21:44:45.260 INFO:teuthology.orchestra.run.smithi203.stdout:(38/156): fuse-2.9.9-16.el9.x86_64.rpm 871 kB/s | 80 kB 00:00 2024-09-15T21:44:45.360 INFO:teuthology.orchestra.run.smithi203.stdout:(39/156): libgfortran-11.5.0-2.el9.x86_64.rpm 7.8 MB/s | 801 kB 00:00 2024-09-15T21:44:45.403 INFO:teuthology.orchestra.run.smithi203.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-15T21:44:45.453 INFO:teuthology.orchestra.run.smithi203.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-09-15T21:44:45.528 INFO:teuthology.orchestra.run.smithi203.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-09-15T21:44:45.578 INFO:teuthology.orchestra.run.smithi203.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-15T21:44:45.621 INFO:teuthology.orchestra.run.smithi203.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-09-15T21:44:45.663 INFO:teuthology.orchestra.run.smithi203.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-09-15T21:44:45.705 INFO:teuthology.orchestra.run.smithi203.stdout:(46/156): python3-urllib3-1.26.5-6.el9.noarch.r 5.0 MB/s | 215 kB 00:00 2024-09-15T21:44:45.797 INFO:teuthology.orchestra.run.smithi203.stdout:(47/156): ceph-mgr-diskprediction-local-19.1.1- 4.7 MB/s | 7.4 MB 00:01 2024-09-15T21:44:46.005 INFO:teuthology.orchestra.run.smithi203.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-09-15T21:44:46.064 INFO:teuthology.orchestra.run.smithi203.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-09-15T21:44:46.123 INFO:teuthology.orchestra.run.smithi203.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-09-15T21:44:46.148 INFO:teuthology.orchestra.run.smithi203.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 259 kB/s | 22 kB 00:00 2024-09-15T21:44:46.181 INFO:teuthology.orchestra.run.smithi203.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-09-15T21:44:46.207 INFO:teuthology.orchestra.run.smithi203.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 557 kB/s | 33 kB 00:00 2024-09-15T21:44:46.266 INFO:teuthology.orchestra.run.smithi203.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-09-15T21:44:46.302 INFO:teuthology.orchestra.run.smithi174.stdout:(10/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 3.9 MB/s | 17 MB 00:04 2024-09-15T21:44:46.316 INFO:teuthology.orchestra.run.smithi203.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-15T21:44:46.366 INFO:teuthology.orchestra.run.smithi203.stdout:(56/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-09-15T21:44:46.400 INFO:teuthology.orchestra.run.smithi203.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-09-15T21:44:46.425 INFO:teuthology.orchestra.run.smithi203.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 769 kB/s | 45 kB 00:00 2024-09-15T21:44:46.452 INFO:teuthology.orchestra.run.smithi174.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 1.5 MB/s | 228 kB 00:00 2024-09-15T21:44:46.468 INFO:teuthology.orchestra.run.smithi203.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 662 kB 00:00 2024-09-15T21:44:46.493 INFO:teuthology.orchestra.run.smithi203.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 8.5 MB/s | 581 kB 00:00 2024-09-15T21:44:46.518 INFO:teuthology.orchestra.run.smithi203.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 486 kB/s | 24 kB 00:00 2024-09-15T21:44:46.536 INFO:teuthology.orchestra.run.smithi174.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 325 kB/s | 27 kB 00:00 2024-09-15T21:44:46.544 INFO:teuthology.orchestra.run.smithi203.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-09-15T21:44:46.577 INFO:teuthology.orchestra.run.smithi203.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-15T21:44:46.619 INFO:teuthology.orchestra.run.smithi203.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-09-15T21:44:46.620 INFO:teuthology.orchestra.run.smithi174.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 381 kB/s | 32 kB 00:00 2024-09-15T21:44:46.645 INFO:teuthology.orchestra.run.smithi203.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 573 kB/s | 38 kB 00:00 2024-09-15T21:44:46.705 INFO:teuthology.orchestra.run.smithi028.stdout:(6/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 1.2 MB/s | 5.4 MB 00:04 2024-09-15T21:44:46.845 INFO:teuthology.orchestra.run.smithi174.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 3.3 MB/s | 751 kB 00:00 2024-09-15T21:44:46.861 INFO:teuthology.orchestra.run.smithi203.stdout:(66/156): openblas-openmp-0.3.26-2.el9.x86_64.r 20 MB/s | 4.9 MB 00:00 2024-09-15T21:44:46.911 INFO:teuthology.orchestra.run.smithi174.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.4 MB/s | 163 kB 00:00 2024-09-15T21:44:46.920 INFO:teuthology.orchestra.run.smithi203.stdout:(67/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-09-15T21:44:46.962 INFO:teuthology.orchestra.run.smithi203.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-15T21:44:46.978 INFO:teuthology.orchestra.run.smithi174.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 1.9 MB/s | 127 kB 00:00 2024-09-15T21:44:46.997 INFO:teuthology.orchestra.run.smithi028.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 2.5 MB/s | 4.7 MB 00:01 2024-09-15T21:44:47.004 INFO:teuthology.orchestra.run.smithi203.stdout:(69/156): postgresql-contrib-13.16-1.el9.x86_64 11 MB/s | 910 kB 00:00 2024-09-15T21:44:47.030 INFO:teuthology.orchestra.run.smithi203.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-15T21:44:47.105 INFO:teuthology.orchestra.run.smithi203.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-15T21:44:47.129 INFO:teuthology.orchestra.run.smithi174.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 3.3 MB/s | 506 kB 00:00 2024-09-15T21:44:47.163 INFO:teuthology.orchestra.run.smithi203.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-09-15T21:44:47.264 INFO:teuthology.orchestra.run.smithi203.stdout:(73/156): postgresql-server-13.16-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-09-15T21:44:47.356 INFO:teuthology.orchestra.run.smithi203.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 668 kB/s | 61 kB 00:00 2024-09-15T21:44:47.423 INFO:teuthology.orchestra.run.smithi203.stdout:(75/156): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-09-15T21:44:47.448 INFO:teuthology.orchestra.run.smithi203.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-09-15T21:44:47.482 INFO:teuthology.orchestra.run.smithi203.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-09-15T21:44:47.507 INFO:teuthology.orchestra.run.smithi203.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 437 kB/s | 26 kB 00:00 2024-09-15T21:44:47.533 INFO:teuthology.orchestra.run.smithi203.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-09-15T21:44:47.558 INFO:teuthology.orchestra.run.smithi203.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-15T21:44:47.583 INFO:teuthology.orchestra.run.smithi203.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 692 kB/s | 35 kB 00:00 2024-09-15T21:44:47.651 INFO:teuthology.orchestra.run.smithi203.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.6 MB/s | 250 kB 00:00 2024-09-15T21:44:47.759 INFO:teuthology.orchestra.run.smithi203.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-09-15T21:44:47.785 INFO:teuthology.orchestra.run.smithi203.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-09-15T21:44:47.826 INFO:teuthology.orchestra.run.smithi203.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-09-15T21:44:47.852 INFO:teuthology.orchestra.run.smithi203.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 631 kB/s | 42 kB 00:00 2024-09-15T21:44:48.336 INFO:teuthology.orchestra.run.smithi203.stdout:(87/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 7.4 MB/s | 49 MB 00:06 2024-09-15T21:44:48.337 INFO:teuthology.orchestra.run.smithi174.stdout:(18/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 4.4 MB/s | 5.3 MB 00:01 2024-09-15T21:44:48.361 INFO:teuthology.orchestra.run.smithi203.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 312 kB/s | 159 kB 00:00 2024-09-15T21:44:48.386 INFO:teuthology.orchestra.run.smithi203.stdout:(89/156): python3-protobuf-3.14.0-13.el9.noarch 481 kB/s | 269 kB 00:00 2024-09-15T21:44:48.387 INFO:teuthology.orchestra.run.smithi174.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 908 kB/s | 45 kB 00:00 2024-09-15T21:44:48.411 INFO:teuthology.orchestra.run.smithi203.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-15T21:44:48.437 INFO:teuthology.orchestra.run.smithi203.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-15T21:44:48.437 INFO:teuthology.orchestra.run.smithi174.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 2.7 MB/s | 136 kB 00:00 2024-09-15T21:44:48.487 INFO:teuthology.orchestra.run.smithi203.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-09-15T21:44:48.496 INFO:teuthology.orchestra.run.smithi174.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 2.7 MB/s | 161 kB 00:00 2024-09-15T21:44:48.546 INFO:teuthology.orchestra.run.smithi203.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 759 kB/s | 44 kB 00:00 2024-09-15T21:44:48.571 INFO:teuthology.orchestra.run.smithi203.stdout:(94/156): python3-pyasn1-modules-0.4.8-6.el9.no 1.2 MB/s | 279 kB 00:00 2024-09-15T21:44:48.572 INFO:teuthology.orchestra.run.smithi174.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.2 MB/s | 320 kB 00:00 2024-09-15T21:44:48.605 INFO:teuthology.orchestra.run.smithi203.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-15T21:44:48.630 INFO:teuthology.orchestra.run.smithi203.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 611 kB/s | 36 kB 00:00 2024-09-15T21:44:48.645 INFO:teuthology.orchestra.run.smithi174.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 4.0 MB/s | 300 kB 00:00 2024-09-15T21:44:48.655 INFO:teuthology.orchestra.run.smithi203.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-09-15T21:44:48.704 INFO:teuthology.orchestra.run.smithi174.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.7 MB/s | 99 kB 00:00 2024-09-15T21:44:48.714 INFO:teuthology.orchestra.run.smithi203.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 991 kB/s | 58 kB 00:00 2024-09-15T21:44:48.739 INFO:teuthology.orchestra.run.smithi203.stdout:(99/156): socat-1.7.4.1-6.el9.x86_64.rpm 2.7 MB/s | 304 kB 00:00 2024-09-15T21:44:48.754 INFO:teuthology.orchestra.run.smithi174.stdout:(25/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-15T21:44:48.790 INFO:teuthology.orchestra.run.smithi203.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.2 MB/s | 96 kB 00:00 2024-09-15T21:44:48.815 INFO:teuthology.orchestra.run.smithi203.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 844 kB/s | 64 kB 00:00 2024-09-15T21:44:48.849 INFO:teuthology.orchestra.run.smithi203.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 763 kB/s | 45 kB 00:00 2024-09-15T21:44:49.041 INFO:teuthology.orchestra.run.smithi203.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-09-15T21:44:49.108 INFO:teuthology.orchestra.run.smithi203.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 8.1 MB/s | 548 kB 00:00 2024-09-15T21:44:49.150 INFO:teuthology.orchestra.run.smithi203.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 7.2 MB/s | 308 kB 00:00 2024-09-15T21:44:49.175 INFO:teuthology.orchestra.run.smithi203.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-15T21:44:49.197 INFO:teuthology.orchestra.run.smithi028.stdout:(8/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 4.8 MB/s | 11 MB 00:02 2024-09-15T21:44:49.213 INFO:teuthology.orchestra.run.smithi174.stdout:(26/156): ceph-common-19.1.1-326.g8921c8ef.el9. 2.2 MB/s | 21 MB 00:09 2024-09-15T21:44:49.239 INFO:teuthology.orchestra.run.smithi028.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 610 kB/s | 25 kB 00:00 2024-09-15T21:44:49.242 INFO:teuthology.orchestra.run.smithi203.stdout:(107/156): lua-devel-5.4.4-4.el9.x86_64.rpm 52 kB/s | 22 kB 00:00 2024-09-15T21:44:49.280 INFO:teuthology.orchestra.run.smithi174.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.5 MB/s | 170 kB 00:00 2024-09-15T21:44:49.293 INFO:teuthology.orchestra.run.smithi203.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-09-15T21:44:49.326 INFO:teuthology.orchestra.run.smithi203.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-15T21:44:49.330 INFO:teuthology.orchestra.run.smithi174.stdout:(28/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 5.4 MB/s | 3.1 MB 00:00 2024-09-15T21:44:49.355 INFO:teuthology.orchestra.run.smithi174.stdout:(29/156): ceph-grafana-dashboards-19.1.1-326.g8 423 kB/s | 32 kB 00:00 2024-09-15T21:44:49.360 INFO:teuthology.orchestra.run.smithi203.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-15T21:44:49.381 INFO:teuthology.orchestra.run.smithi174.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.8 MB/s | 146 kB 00:00 2024-09-15T21:44:49.393 INFO:teuthology.orchestra.run.smithi203.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-09-15T21:44:49.502 INFO:teuthology.orchestra.run.smithi203.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.6 MB/s | 839 kB 00:00 2024-09-15T21:44:49.728 INFO:teuthology.orchestra.run.smithi203.stdout:(113/156): libarrow-9.0.0-11.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-09-15T21:44:49.769 INFO:teuthology.orchestra.run.smithi203.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 2.0 MB/s | 548 kB 00:00 2024-09-15T21:44:49.795 INFO:teuthology.orchestra.run.smithi203.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 648 kB/s | 43 kB 00:00 2024-09-15T21:44:49.820 INFO:teuthology.orchestra.run.smithi203.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-09-15T21:44:49.846 INFO:teuthology.orchestra.run.smithi203.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-09-15T21:44:49.879 INFO:teuthology.orchestra.run.smithi203.stdout:(118/156): protobuf-compiler-3.14.0-13.el9.x86_ 838 kB/s | 863 kB 00:01 2024-09-15T21:44:49.905 INFO:teuthology.orchestra.run.smithi203.stdout:(119/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-09-15T21:44:49.938 INFO:teuthology.orchestra.run.smithi203.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-09-15T21:44:49.963 INFO:teuthology.orchestra.run.smithi203.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.2 MB/s | 220 kB 00:00 2024-09-15T21:44:49.981 INFO:teuthology.orchestra.run.smithi174.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 5.7 MB/s | 3.6 MB 00:00 2024-09-15T21:44:50.005 INFO:teuthology.orchestra.run.smithi203.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-09-15T21:44:50.031 INFO:teuthology.orchestra.run.smithi203.stdout:(123/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-09-15T21:44:50.048 INFO:teuthology.orchestra.run.smithi174.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 3.7 MB/s | 249 kB 00:00 2024-09-15T21:44:50.056 INFO:teuthology.orchestra.run.smithi203.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-09-15T21:44:50.081 INFO:teuthology.orchestra.run.smithi203.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-09-15T21:44:50.107 INFO:teuthology.orchestra.run.smithi203.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-09-15T21:44:50.132 INFO:teuthology.orchestra.run.smithi203.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-09-15T21:44:50.157 INFO:teuthology.orchestra.run.smithi203.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-09-15T21:44:50.183 INFO:teuthology.orchestra.run.smithi203.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T21:44:50.217 INFO:teuthology.orchestra.run.smithi203.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-15T21:44:50.258 INFO:teuthology.orchestra.run.smithi203.stdout:(131/156): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-09-15T21:44:50.284 INFO:teuthology.orchestra.run.smithi203.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 861 kB/s | 58 kB 00:00 2024-09-15T21:44:50.326 INFO:teuthology.orchestra.run.smithi203.stdout:(133/156): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-09-15T21:44:50.351 INFO:teuthology.orchestra.run.smithi203.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-09-15T21:44:50.377 INFO:teuthology.orchestra.run.smithi203.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-15T21:44:50.402 INFO:teuthology.orchestra.run.smithi203.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-09-15T21:44:50.427 INFO:teuthology.orchestra.run.smithi203.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-15T21:44:50.453 INFO:teuthology.orchestra.run.smithi203.stdout:(138/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-09-15T21:44:50.457 INFO:teuthology.orchestra.run.smithi174.stdout:(33/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 7.6 MB/s | 49 MB 00:06 2024-09-15T21:44:50.486 INFO:teuthology.orchestra.run.smithi203.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-15T21:44:50.489 INFO:teuthology.orchestra.run.smithi028.stdout:(10/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 4.5 MB/s | 17 MB 00:03 2024-09-15T21:44:50.503 INFO:teuthology.orchestra.run.smithi203.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-09-15T21:44:50.507 INFO:teuthology.orchestra.run.smithi174.stdout:(34/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 107 kB/s | 49 kB 00:00 2024-09-15T21:44:50.529 INFO:teuthology.orchestra.run.smithi203.stdout:(141/156): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-09-15T21:44:50.532 INFO:teuthology.orchestra.run.smithi174.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 220 kB/s | 16 kB 00:00 2024-09-15T21:44:50.554 INFO:teuthology.orchestra.run.smithi203.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 1.1 MB/s | 74 kB 00:00 2024-09-15T21:44:50.556 INFO:teuthology.orchestra.run.smithi028.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.4 MB/s | 228 kB 00:00 2024-09-15T21:44:50.579 INFO:teuthology.orchestra.run.smithi203.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-15T21:44:50.598 INFO:teuthology.orchestra.run.smithi028.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 654 kB/s | 27 kB 00:00 2024-09-15T21:44:50.605 INFO:teuthology.orchestra.run.smithi203.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 800 kB/s | 20 kB 00:00 2024-09-15T21:44:50.630 INFO:teuthology.orchestra.run.smithi203.stdout:(145/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-15T21:44:50.631 INFO:teuthology.orchestra.run.smithi028.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 953 kB/s | 32 kB 00:00 2024-09-15T21:44:50.641 INFO:teuthology.orchestra.run.smithi174.stdout:(36/156): ceph-mgr-diskprediction-local-19.1.1- 5.9 MB/s | 7.4 MB 00:01 2024-09-15T21:44:50.664 INFO:teuthology.orchestra.run.smithi203.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-15T21:44:50.666 INFO:teuthology.orchestra.run.smithi174.stdout:(37/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 1.8 MB/s | 285 kB 00:00 2024-09-15T21:44:50.689 INFO:teuthology.orchestra.run.smithi203.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-15T21:44:50.691 INFO:teuthology.orchestra.run.smithi174.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 366 kB/s | 18 kB 00:00 2024-09-15T21:44:50.733 INFO:teuthology.orchestra.run.smithi174.stdout:(39/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 3.7 MB/s | 765 kB 00:00 2024-09-15T21:44:50.739 INFO:teuthology.orchestra.run.smithi203.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-15T21:44:50.740 INFO:teuthology.orchestra.run.smithi028.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 6.8 MB/s | 751 kB 00:00 2024-09-15T21:44:50.791 INFO:teuthology.orchestra.run.smithi028.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 3.2 MB/s | 163 kB 00:00 2024-09-15T21:44:50.841 INFO:teuthology.orchestra.run.smithi028.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 2.5 MB/s | 127 kB 00:00 2024-09-15T21:44:50.842 INFO:teuthology.orchestra.run.smithi174.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 455 kB/s | 80 kB 00:00 2024-09-15T21:44:50.867 INFO:teuthology.orchestra.run.smithi174.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-15T21:44:50.893 INFO:teuthology.orchestra.run.smithi174.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-15T21:44:50.914 INFO:teuthology.orchestra.run.smithi203.stdout:(149/156): NetworkManager-1.49.90-1.el9.x86_64. 13 MB/s | 2.3 MB 00:00 2024-09-15T21:44:50.933 INFO:teuthology.orchestra.run.smithi028.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 5.4 MB/s | 506 kB 00:00 2024-09-15T21:44:50.976 INFO:teuthology.orchestra.run.smithi174.stdout:(43/156): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-09-15T21:44:50.981 INFO:teuthology.orchestra.run.smithi203.stdout:(150/156): NetworkManager-libnm-1.49.90-1.el9.x 27 MB/s | 1.8 MB 00:00 2024-09-15T21:44:51.001 INFO:teuthology.orchestra.run.smithi174.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 982 kB/s | 106 kB 00:00 2024-09-15T21:44:51.025 INFO:teuthology.orchestra.run.smithi203.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 982 kB/s | 41 kB 00:00 2024-09-15T21:44:51.026 INFO:teuthology.orchestra.run.smithi174.stdout:(45/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-15T21:44:51.052 INFO:teuthology.orchestra.run.smithi174.stdout:(46/156): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-09-15T21:44:51.077 INFO:teuthology.orchestra.run.smithi174.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-09-15T21:44:51.082 INFO:teuthology.orchestra.run.smithi203.stdout:(152/156): librados2-19.1.1-326.g8921c8ef.el9.x 8.2 MB/s | 3.4 MB 00:00 2024-09-15T21:44:51.119 INFO:teuthology.orchestra.run.smithi174.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-09-15T21:44:51.141 INFO:teuthology.orchestra.run.smithi203.stdout:(153/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 7.0 MB/s | 3.2 MB 00:00 2024-09-15T21:44:51.166 INFO:teuthology.orchestra.run.smithi203.stdout:(154/156): NetworkManager-tui-1.49.90-1.el9.x86 1.7 MB/s | 249 kB 00:00 2024-09-15T21:44:51.291 INFO:teuthology.orchestra.run.smithi203.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 6.5 MB/s | 1.4 MB 00:00 2024-09-15T21:44:51.294 INFO:teuthology.orchestra.run.smithi174.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 611 kB/s | 107 kB 00:00 2024-09-15T21:44:51.353 INFO:teuthology.orchestra.run.smithi174.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 372 kB/s | 22 kB 00:00 2024-09-15T21:44:51.378 INFO:teuthology.orchestra.run.smithi174.stdout:(51/156): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-09-15T21:44:51.400 INFO:teuthology.orchestra.run.smithi203.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 9.3 MB/s | 2.4 MB 00:00 2024-09-15T21:44:51.407 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:44:51.407 INFO:teuthology.orchestra.run.smithi203.stdout:Total 18 MB/s | 226 MB 00:12 2024-09-15T21:44:51.420 INFO:teuthology.orchestra.run.smithi174.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 943 kB 00:00 2024-09-15T21:44:51.437 INFO:teuthology.orchestra.run.smithi174.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-09-15T21:44:51.479 INFO:teuthology.orchestra.run.smithi174.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 439 kB/s | 18 kB 00:00 2024-09-15T21:44:51.558 INFO:teuthology.orchestra.run.smithi028.stdout:(18/156): ceph-common-19.1.1-326.g8921c8ef.el9. 2.2 MB/s | 21 MB 00:09 2024-09-15T21:44:51.562 INFO:teuthology.orchestra.run.smithi174.stdout:(55/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-09-15T21:44:51.579 INFO:teuthology.orchestra.run.smithi174.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-09-15T21:44:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 910 kB/s | 45 kB 00:00 2024-09-15T21:44:51.621 INFO:teuthology.orchestra.run.smithi174.stdout:(57/156): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-09-15T21:44:51.646 INFO:teuthology.orchestra.run.smithi174.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 677 kB/s | 45 kB 00:00 2024-09-15T21:44:51.672 INFO:teuthology.orchestra.run.smithi174.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-09-15T21:44:51.675 INFO:teuthology.orchestra.run.smithi028.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 2.0 MB/s | 136 kB 00:00 2024-09-15T21:44:51.705 INFO:teuthology.orchestra.run.smithi174.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 9.7 MB/s | 581 kB 00:00 2024-09-15T21:44:51.730 INFO:teuthology.orchestra.run.smithi174.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 419 kB/s | 24 kB 00:00 2024-09-15T21:44:51.742 INFO:teuthology.orchestra.run.smithi028.stdout:(21/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 6.6 MB/s | 5.3 MB 00:00 2024-09-15T21:44:51.764 INFO:teuthology.orchestra.run.smithi174.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-09-15T21:44:51.767 INFO:teuthology.orchestra.run.smithi028.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.7 MB/s | 161 kB 00:00 2024-09-15T21:44:51.790 INFO:teuthology.orchestra.run.smithi174.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-15T21:44:51.809 INFO:teuthology.orchestra.run.smithi028.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 4.7 MB/s | 320 kB 00:00 2024-09-15T21:44:51.823 INFO:teuthology.orchestra.run.smithi174.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-09-15T21:44:51.849 INFO:teuthology.orchestra.run.smithi174.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 655 kB/s | 38 kB 00:00 2024-09-15T21:44:51.859 INFO:teuthology.orchestra.run.smithi028.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 3.2 MB/s | 300 kB 00:00 2024-09-15T21:44:51.885 INFO:teuthology.orchestra.run.smithi028.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.3 MB/s | 99 kB 00:00 2024-09-15T21:44:51.910 INFO:teuthology.orchestra.run.smithi028.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-15T21:44:51.960 INFO:teuthology.orchestra.run.smithi028.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 3.3 MB/s | 170 kB 00:00 2024-09-15T21:44:51.974 INFO:teuthology.orchestra.run.smithi174.stdout:(66/156): openblas-openmp-0.3.26-2.el9.x86_64.r 33 MB/s | 4.9 MB 00:00 2024-09-15T21:44:52.013 INFO:teuthology.orchestra.run.smithi028.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 612 kB/s | 32 kB 00:00 2024-09-15T21:44:52.057 INFO:teuthology.orchestra.run.smithi174.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-15T21:44:52.063 INFO:teuthology.orchestra.run.smithi028.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.9 MB/s | 146 kB 00:00 2024-09-15T21:44:52.083 INFO:teuthology.orchestra.run.smithi174.stdout:(68/156): librdkafka-1.6.1-102.el9.x86_64.rpm 1.4 MB/s | 662 kB 00:00 2024-09-15T21:44:52.150 INFO:teuthology.orchestra.run.smithi174.stdout:(69/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-09-15T21:44:52.183 INFO:teuthology.orchestra.run.smithi174.stdout:(70/156): postgresql-contrib-13.16-1.el9.x86_64 7.1 MB/s | 910 kB 00:00 2024-09-15T21:44:52.275 INFO:teuthology.orchestra.run.smithi174.stdout:(71/156): postgresql-private-libs-13.16-1.el9.x 715 kB/s | 137 kB 00:00 2024-09-15T21:44:52.280 INFO:teuthology.orchestra.run.smithi028.stdout:(30/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 7.9 MB/s | 3.1 MB 00:00 2024-09-15T21:44:52.317 INFO:teuthology.orchestra.run.smithi174.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-09-15T21:44:52.392 INFO:teuthology.orchestra.run.smithi174.stdout:(73/156): postgresql-server-13.16-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-09-15T21:44:52.417 INFO:teuthology.orchestra.run.smithi174.stdout:(74/156): python3-attrs-20.3.0-7.el9.noarch.rpm 616 kB/s | 87 kB 00:00 2024-09-15T21:44:52.443 INFO:teuthology.orchestra.run.smithi174.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-15T21:44:52.502 INFO:teuthology.orchestra.run.smithi174.stdout:(76/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-09-15T21:44:52.572 INFO:teuthology.orchestra.run.smithi028.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 7.0 MB/s | 3.6 MB 00:00 2024-09-15T21:44:52.577 INFO:teuthology.orchestra.run.smithi174.stdout:(77/156): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-09-15T21:44:52.602 INFO:teuthology.orchestra.run.smithi174.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-09-15T21:44:52.627 INFO:teuthology.orchestra.run.smithi174.stdout:(79/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-15T21:44:52.639 INFO:teuthology.orchestra.run.smithi028.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 3.7 MB/s | 249 kB 00:00 2024-09-15T21:44:52.652 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-15T21:44:52.652 INFO:teuthology.orchestra.run.smithi174.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 254 kB/s | 19 kB 00:00 2024-09-15T21:44:52.678 INFO:teuthology.orchestra.run.smithi174.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-15T21:44:52.680 INFO:teuthology.orchestra.run.smithi028.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 1.2 MB/s | 49 kB 00:00 2024-09-15T21:44:52.703 INFO:teuthology.orchestra.run.smithi174.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-09-15T21:44:52.723 INFO:teuthology.orchestra.run.smithi028.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 396 kB/s | 16 kB 00:00 2024-09-15T21:44:52.745 INFO:teuthology.orchestra.run.smithi174.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.6 MB/s | 250 kB 00:00 2024-09-15T21:44:52.789 INFO:teuthology.orchestra.run.smithi028.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.2 MB/s | 285 kB 00:00 2024-09-15T21:44:52.804 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-15T21:44:52.804 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-15T21:44:52.812 INFO:teuthology.orchestra.run.smithi174.stdout:(84/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-09-15T21:44:52.837 INFO:teuthology.orchestra.run.smithi174.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 842 kB/s | 77 kB 00:00 2024-09-15T21:44:52.871 INFO:teuthology.orchestra.run.smithi174.stdout:(86/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 222 kB 00:00 2024-09-15T21:44:52.887 INFO:teuthology.orchestra.run.smithi174.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 560 kB/s | 42 kB 00:00 2024-09-15T21:44:52.907 INFO:teuthology.orchestra.run.smithi028.stdout:(36/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 6.4 MB/s | 765 kB 00:00 2024-09-15T21:44:52.913 INFO:teuthology.orchestra.run.smithi174.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-09-15T21:44:52.948 INFO:teuthology.orchestra.run.smithi028.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 441 kB/s | 18 kB 00:00 2024-09-15T21:44:52.955 INFO:teuthology.orchestra.run.smithi174.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-09-15T21:44:52.980 INFO:teuthology.orchestra.run.smithi174.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 760 kB/s | 51 kB 00:00 2024-09-15T21:44:53.005 INFO:teuthology.orchestra.run.smithi174.stdout:(91/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-15T21:44:53.030 INFO:teuthology.orchestra.run.smithi174.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 713 kB/s | 54 kB 00:00 2024-09-15T21:44:53.064 INFO:teuthology.orchestra.run.smithi174.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-09-15T21:44:53.106 INFO:teuthology.orchestra.run.smithi174.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 587 kB/s | 44 kB 00:00 2024-09-15T21:44:53.124 INFO:teuthology.orchestra.run.smithi028.stdout:(38/156): ceph-mgr-diskprediction-local-19.1.1- 8.8 MB/s | 7.4 MB 00:00 2024-09-15T21:44:53.149 INFO:teuthology.orchestra.run.smithi028.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 399 kB/s | 80 kB 00:00 2024-09-15T21:44:53.164 INFO:teuthology.orchestra.run.smithi174.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 615 kB/s | 36 kB 00:00 2024-09-15T21:44:53.215 INFO:teuthology.orchestra.run.smithi174.stdout:(96/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-15T21:44:53.241 INFO:teuthology.orchestra.run.smithi174.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-15T21:44:53.282 INFO:teuthology.orchestra.run.smithi028.stdout:(40/156): libgfortran-11.5.0-2.el9.x86_64.rpm 4.9 MB/s | 801 kB 00:00 2024-09-15T21:44:53.308 INFO:teuthology.orchestra.run.smithi028.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-15T21:44:53.399 INFO:teuthology.orchestra.run.smithi174.stdout:(98/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-09-15T21:44:53.399 INFO:teuthology.orchestra.run.smithi028.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-15T21:44:53.424 INFO:teuthology.orchestra.run.smithi174.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 315 kB/s | 58 kB 00:00 2024-09-15T21:44:53.458 INFO:teuthology.orchestra.run.smithi174.stdout:(100/156): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-15T21:44:53.458 INFO:teuthology.orchestra.run.smithi028.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-15T21:44:53.483 INFO:teuthology.orchestra.run.smithi174.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-15T21:44:53.508 INFO:teuthology.orchestra.run.smithi174.stdout:(102/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 879 kB/s | 96 kB 00:00 2024-09-15T21:44:53.525 INFO:teuthology.orchestra.run.smithi028.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-09-15T21:44:53.533 INFO:teuthology.orchestra.run.smithi174.stdout:(103/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-09-15T21:44:53.559 INFO:teuthology.orchestra.run.smithi028.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 4.9 MB/s | 1.2 MB 00:00 2024-09-15T21:44:53.592 INFO:teuthology.orchestra.run.smithi174.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-15T21:44:53.601 INFO:teuthology.orchestra.run.smithi028.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-09-15T21:44:53.626 INFO:teuthology.orchestra.run.smithi174.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 9.0 MB/s | 308 kB 00:00 2024-09-15T21:44:53.651 INFO:teuthology.orchestra.run.smithi174.stdout:(106/156): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-09-15T21:44:53.668 INFO:teuthology.orchestra.run.smithi028.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-09-15T21:44:53.676 INFO:teuthology.orchestra.run.smithi174.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-09-15T21:44:53.710 INFO:teuthology.orchestra.run.smithi174.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-15T21:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 5.0 MB/s | 689 kB 00:00 2024-09-15T21:44:53.735 INFO:teuthology.orchestra.run.smithi174.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-15T21:44:53.760 INFO:teuthology.orchestra.run.smithi174.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-15T21:44:53.810 INFO:teuthology.orchestra.run.smithi174.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-15T21:44:53.830 INFO:teuthology.orchestra.run.smithi028.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-09-15T21:44:53.853 INFO:teuthology.orchestra.run.smithi174.stdout:(112/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-09-15T21:44:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 4.6 MB/s | 943 kB 00:00 2024-09-15T21:44:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 304 kB/s | 22 kB 00:00 2024-09-15T21:44:53.920 INFO:teuthology.orchestra.run.smithi174.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-15T21:44:53.945 INFO:teuthology.orchestra.run.smithi174.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-15T21:44:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-09-15T21:44:53.970 INFO:teuthology.orchestra.run.smithi174.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-15T21:44:53.986 INFO:teuthology.orchestra.run.smithi028.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-15T21:44:54.004 INFO:teuthology.orchestra.run.smithi174.stdout:(116/156): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-09-15T21:44:54.037 INFO:teuthology.orchestra.run.smithi174.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 210 kB/s | 14 kB 00:00 2024-09-15T21:44:54.044 INFO:teuthology.orchestra.run.smithi028.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-09-15T21:44:54.062 INFO:teuthology.orchestra.run.smithi174.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-09-15T21:44:54.096 INFO:teuthology.orchestra.run.smithi174.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-15T21:44:54.103 INFO:teuthology.orchestra.run.smithi028.stdout:(55/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-09-15T21:44:54.128 INFO:teuthology.orchestra.run.smithi028.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 133 kB/s | 11 kB 00:00 2024-09-15T21:44:54.130 INFO:teuthology.orchestra.run.smithi174.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-09-15T21:44:54.155 INFO:teuthology.orchestra.run.smithi174.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-15T21:44:54.170 INFO:teuthology.orchestra.run.smithi028.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-09-15T21:44:54.197 INFO:teuthology.orchestra.run.smithi174.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-09-15T21:44:54.222 INFO:teuthology.orchestra.run.smithi174.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-15T21:44:54.237 INFO:teuthology.orchestra.run.smithi028.stdout:(58/156): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-09-15T21:44:54.247 INFO:teuthology.orchestra.run.smithi174.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 710 kB/s | 18 kB 00:00 2024-09-15T21:44:54.271 INFO:teuthology.orchestra.run.smithi028.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 318 kB/s | 45 kB 00:00 2024-09-15T21:44:54.306 INFO:teuthology.orchestra.run.smithi174.stdout:(125/156): libarrow-9.0.0-11.el9.x86_64.rpm 6.7 MB/s | 4.4 MB 00:00 2024-09-15T21:44:54.331 INFO:teuthology.orchestra.run.smithi174.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-09-15T21:44:54.357 INFO:teuthology.orchestra.run.smithi174.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-09-15T21:44:54.382 INFO:teuthology.orchestra.run.smithi174.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-09-15T21:44:54.408 INFO:teuthology.orchestra.run.smithi174.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T21:44:54.441 INFO:teuthology.orchestra.run.smithi174.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-15T21:44:54.475 INFO:teuthology.orchestra.run.smithi174.stdout:(131/156): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-09-15T21:44:54.491 INFO:teuthology.orchestra.run.smithi174.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-09-15T21:44:54.508 INFO:teuthology.orchestra.run.smithi174.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 990 kB/s | 16 kB 00:00 2024-09-15T21:44:54.534 INFO:teuthology.orchestra.run.smithi174.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-09-15T21:44:54.559 INFO:teuthology.orchestra.run.smithi174.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-09-15T21:44:54.576 INFO:teuthology.orchestra.run.smithi174.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 1.7 MB/s | 28 kB 00:00 2024-09-15T21:44:54.593 INFO:teuthology.orchestra.run.smithi174.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 922 kB/s | 31 kB 00:00 2024-09-15T21:44:54.614 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-15T21:44:54.615 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-15T21:44:54.618 INFO:teuthology.orchestra.run.smithi174.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T21:44:54.643 INFO:teuthology.orchestra.run.smithi174.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-09-15T21:44:54.685 INFO:teuthology.orchestra.run.smithi174.stdout:(140/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-09-15T21:44:54.710 INFO:teuthology.orchestra.run.smithi174.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-09-15T21:44:54.736 INFO:teuthology.orchestra.run.smithi174.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 808 kB/s | 74 kB 00:00 2024-09-15T21:44:54.763 INFO:teuthology.orchestra.run.smithi028.stdout:(60/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 8.8 MB/s | 49 MB 00:05 2024-09-15T21:44:54.788 INFO:teuthology.orchestra.run.smithi028.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 47 kB/s | 24 kB 00:00 2024-09-15T21:44:54.813 INFO:teuthology.orchestra.run.smithi028.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 1.0 MB/s | 581 kB 00:00 2024-09-15T21:44:54.849 INFO:teuthology.orchestra.run.smithi174.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 647 kB/s | 90 kB 00:00 2024-09-15T21:44:54.855 INFO:teuthology.orchestra.run.smithi028.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-15T21:44:54.878 INFO:teuthology.orchestra.run.smithi174.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-09-15T21:44:54.880 INFO:teuthology.orchestra.run.smithi028.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-09-15T21:44:54.904 INFO:teuthology.orchestra.run.smithi174.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 371 kB/s | 20 kB 00:00 2024-09-15T21:44:54.906 INFO:teuthology.orchestra.run.smithi028.stdout:(65/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 334 kB/s | 47 kB 00:00 2024-09-15T21:44:54.931 INFO:teuthology.orchestra.run.smithi028.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 509 kB/s | 38 kB 00:00 2024-09-15T21:44:54.937 INFO:teuthology.orchestra.run.smithi174.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-15T21:44:54.962 INFO:teuthology.orchestra.run.smithi174.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 1.8 MB/s | 427 kB 00:00 2024-09-15T21:44:55.029 INFO:teuthology.orchestra.run.smithi174.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-15T21:44:55.031 INFO:teuthology.orchestra.run.smithi028.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 33 MB/s | 4.9 MB 00:00 2024-09-15T21:44:55.074 INFO:teuthology.orchestra.run.smithi028.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-15T21:44:55.107 INFO:teuthology.orchestra.run.smithi028.stdout:(69/156): postgresql-contrib-13.16-1.el9.x86_64 12 MB/s | 910 kB 00:00 2024-09-15T21:44:55.140 INFO:teuthology.orchestra.run.smithi028.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 2.0 MB/s | 137 kB 00:00 2024-09-15T21:44:55.180 INFO:teuthology.orchestra.run.smithi174.stdout:(149/156): NetworkManager-1.49.90-1.el9.x86_64. 16 MB/s | 2.3 MB 00:00 2024-09-15T21:44:55.249 INFO:teuthology.orchestra.run.smithi028.stdout:(71/156): postgresql-server-13.16-1.el9.x86_64. 41 MB/s | 5.8 MB 00:00 2024-09-15T21:44:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-09-15T21:44:55.313 INFO:teuthology.orchestra.run.smithi174.stdout:(150/156): librados2-19.1.1-326.g8921c8ef.el9.x 9.1 MB/s | 3.4 MB 00:00 2024-09-15T21:44:55.316 INFO:teuthology.orchestra.run.smithi028.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-15T21:44:55.355 INFO:teuthology.orchestra.run.smithi174.stdout:(151/156): NetworkManager-libnm-1.49.90-1.el9.x 10 MB/s | 1.8 MB 00:00 2024-09-15T21:44:55.375 INFO:teuthology.orchestra.run.smithi028.stdout:(74/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2024-09-15T21:44:55.380 INFO:teuthology.orchestra.run.smithi174.stdout:(152/156): NetworkManager-team-1.49.90-1.el9.x8 611 kB/s | 41 kB 00:00 2024-09-15T21:44:55.391 INFO:teuthology.orchestra.run.smithi028.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 815 kB/s | 61 kB 00:00 2024-09-15T21:44:55.447 INFO:teuthology.orchestra.run.smithi174.stdout:(153/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 6.5 MB/s | 3.2 MB 00:00 2024-09-15T21:44:55.481 INFO:teuthology.orchestra.run.smithi174.stdout:(154/156): NetworkManager-tui-1.49.90-1.el9.x86 1.9 MB/s | 249 kB 00:00 2024-09-15T21:44:55.484 INFO:teuthology.orchestra.run.smithi028.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 6.0 MB 00:00 2024-09-15T21:44:55.509 INFO:teuthology.orchestra.run.smithi028.stdout:(77/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-15T21:44:55.514 INFO:teuthology.orchestra.run.smithi174.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 10 MB/s | 1.4 MB 00:00 2024-09-15T21:44:55.534 INFO:teuthology.orchestra.run.smithi028.stdout:(78/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-09-15T21:44:55.551 INFO:teuthology.orchestra.run.smithi028.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 384 kB/s | 26 kB 00:00 2024-09-15T21:44:55.593 INFO:teuthology.orchestra.run.smithi028.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 327 kB/s | 19 kB 00:00 2024-09-15T21:44:55.618 INFO:teuthology.orchestra.run.smithi028.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-15T21:44:55.643 INFO:teuthology.orchestra.run.smithi028.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 377 kB/s | 35 kB 00:00 2024-09-15T21:44:55.656 INFO:teuthology.orchestra.run.smithi174.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-09-15T21:44:55.663 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:44:55.663 INFO:teuthology.orchestra.run.smithi174.stdout:Total 13 MB/s | 226 MB 00:16 2024-09-15T21:44:55.702 INFO:teuthology.orchestra.run.smithi028.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-09-15T21:44:55.727 INFO:teuthology.orchestra.run.smithi028.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-09-15T21:44:55.794 INFO:teuthology.orchestra.run.smithi028.stdout:(85/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-09-15T21:44:55.819 INFO:teuthology.orchestra.run.smithi028.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 460 kB/s | 42 kB 00:00 2024-09-15T21:44:55.845 INFO:teuthology.orchestra.run.smithi028.stdout:(87/156): python3-packaging-20.9-5.el9.noarch.r 542 kB/s | 77 kB 00:00 2024-09-15T21:44:55.878 INFO:teuthology.orchestra.run.smithi028.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 269 kB 00:00 2024-09-15T21:44:55.903 INFO:teuthology.orchestra.run.smithi028.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 159 kB 00:00 2024-09-15T21:44:55.962 INFO:teuthology.orchestra.run.smithi028.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-15T21:44:55.987 INFO:teuthology.orchestra.run.smithi028.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-09-15T21:44:56.021 INFO:teuthology.orchestra.run.smithi028.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 458 kB/s | 54 kB 00:00 2024-09-15T21:44:56.088 INFO:teuthology.orchestra.run.smithi028.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 625 kB/s | 42 kB 00:00 2024-09-15T21:44:56.113 INFO:teuthology.orchestra.run.smithi028.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 481 kB/s | 44 kB 00:00 2024-09-15T21:44:56.155 INFO:teuthology.orchestra.run.smithi028.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-09-15T21:44:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 537 kB/s | 36 kB 00:00 2024-09-15T21:44:56.231 INFO:teuthology.orchestra.run.smithi028.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 631 kB/s | 47 kB 00:00 2024-09-15T21:44:56.256 INFO:teuthology.orchestra.run.smithi028.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 4.0 MB/s | 304 kB 00:00 2024-09-15T21:44:56.423 INFO:teuthology.orchestra.run.smithi028.stdout:(99/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-15T21:44:56.448 INFO:teuthology.orchestra.run.smithi028.stdout:(100/156): uuid-1.6.2-55.el9.x86_64.rpm 266 kB/s | 58 kB 00:00 2024-09-15T21:44:56.473 INFO:teuthology.orchestra.run.smithi028.stdout:(101/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 441 kB/s | 96 kB 00:00 2024-09-15T21:44:56.498 INFO:teuthology.orchestra.run.smithi028.stdout:(102/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 847 kB/s | 64 kB 00:00 2024-09-15T21:44:56.524 INFO:teuthology.orchestra.run.smithi028.stdout:(103/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-09-15T21:44:56.574 INFO:teuthology.orchestra.run.smithi028.stdout:(104/156): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-09-15T21:44:56.641 INFO:teuthology.orchestra.run.smithi028.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-09-15T21:44:56.666 INFO:teuthology.orchestra.run.smithi028.stdout:(106/156): gperftools-libs-2.9.1-3.el9.x86_64.r 3.3 MB/s | 308 kB 00:00 2024-09-15T21:44:56.691 INFO:teuthology.orchestra.run.smithi028.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 386 kB/s | 19 kB 00:00 2024-09-15T21:44:56.717 INFO:teuthology.orchestra.run.smithi028.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-15T21:44:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-15T21:44:56.775 INFO:teuthology.orchestra.run.smithi028.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-15T21:44:56.809 INFO:teuthology.orchestra.run.smithi028.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2024-09-15T21:44:56.893 INFO:teuthology.orchestra.run.smithi028.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-15T21:44:56.894 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-15T21:44:56.918 INFO:teuthology.orchestra.run.smithi028.stdout:(113/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-09-15T21:44:56.968 INFO:teuthology.orchestra.run.smithi028.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 869 kB/s | 43 kB 00:00 2024-09-15T21:44:56.993 INFO:teuthology.orchestra.run.smithi028.stdout:(115/156): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-09-15T21:44:57.019 INFO:teuthology.orchestra.run.smithi028.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 648 kB/s | 32 kB 00:00 2024-09-15T21:44:57.044 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-15T21:44:57.044 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-15T21:44:57.044 INFO:teuthology.orchestra.run.smithi028.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 282 kB/s | 14 kB 00:00 2024-09-15T21:44:57.077 INFO:teuthology.orchestra.run.smithi028.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-09-15T21:44:57.111 INFO:teuthology.orchestra.run.smithi028.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-09-15T21:44:57.178 INFO:teuthology.orchestra.run.smithi028.stdout:(120/156): libarrow-9.0.0-11.el9.x86_64.rpm 8.6 MB/s | 4.4 MB 00:00 2024-09-15T21:44:57.203 INFO:teuthology.orchestra.run.smithi028.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-09-15T21:44:57.228 INFO:teuthology.orchestra.run.smithi028.stdout:(122/156): python3-google-auth-2.34.0-1.el9.noa 1.8 MB/s | 220 kB 00:00 2024-09-15T21:44:57.254 INFO:teuthology.orchestra.run.smithi028.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-15T21:44:57.287 INFO:teuthology.orchestra.run.smithi028.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 532 kB/s | 18 kB 00:00 2024-09-15T21:44:57.313 INFO:teuthology.orchestra.run.smithi028.stdout:(125/156): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-09-15T21:44:57.329 INFO:teuthology.orchestra.run.smithi028.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 551 kB/s | 23 kB 00:00 2024-09-15T21:44:57.346 INFO:teuthology.orchestra.run.smithi028.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 581 kB/s | 19 kB 00:00 2024-09-15T21:44:57.363 INFO:teuthology.orchestra.run.smithi028.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 584 kB/s | 20 kB 00:00 2024-09-15T21:44:57.389 INFO:teuthology.orchestra.run.smithi028.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T21:44:57.439 INFO:teuthology.orchestra.run.smithi028.stdout:(130/156): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-09-15T21:44:57.464 INFO:teuthology.orchestra.run.smithi028.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-09-15T21:44:57.489 INFO:teuthology.orchestra.run.smithi028.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-09-15T21:44:57.515 INFO:teuthology.orchestra.run.smithi028.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 663 kB/s | 16 kB 00:00 2024-09-15T21:44:57.548 INFO:teuthology.orchestra.run.smithi028.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-15T21:44:57.574 INFO:teuthology.orchestra.run.smithi028.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-15T21:44:57.599 INFO:teuthology.orchestra.run.smithi028.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-09-15T21:44:57.624 INFO:teuthology.orchestra.run.smithi028.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-09-15T21:44:57.649 INFO:teuthology.orchestra.run.smithi028.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-09-15T21:44:57.683 INFO:teuthology.orchestra.run.smithi028.stdout:(139/156): python3-kubernetes-26.1.0-2.el9.noar 3.0 MB/s | 1.0 MB 00:00 2024-09-15T21:44:57.708 INFO:teuthology.orchestra.run.smithi028.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-09-15T21:44:57.734 INFO:teuthology.orchestra.run.smithi028.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 427 kB/s | 36 kB 00:00 2024-09-15T21:44:57.759 INFO:teuthology.orchestra.run.smithi028.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 987 kB/s | 74 kB 00:00 2024-09-15T21:44:57.784 INFO:teuthology.orchestra.run.smithi028.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-09-15T21:44:57.809 INFO:teuthology.orchestra.run.smithi028.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T21:44:57.834 INFO:teuthology.orchestra.run.smithi028.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-09-15T21:44:57.859 INFO:teuthology.orchestra.run.smithi028.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-15T21:44:57.885 INFO:teuthology.orchestra.run.smithi028.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-15T21:44:57.960 INFO:teuthology.orchestra.run.smithi028.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-15T21:44:58.402 INFO:teuthology.orchestra.run.smithi028.stdout:(149/156): librados2-19.1.1-326.g8921c8ef.el9.x 6.3 MB/s | 3.4 MB 00:00 2024-09-15T21:44:58.460 INFO:teuthology.orchestra.run.smithi028.stdout:(150/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 5.5 MB/s | 3.2 MB 00:00 2024-09-15T21:44:58.586 INFO:teuthology.orchestra.run.smithi028.stdout:(151/156): NetworkManager-1.49.90-1.el9.x86_64. 3.7 MB/s | 2.3 MB 00:00 2024-09-15T21:44:58.611 INFO:teuthology.orchestra.run.smithi028.stdout:(152/156): NetworkManager-team-1.49.90-1.el9.x8 271 kB/s | 41 kB 00:00 2024-09-15T21:44:58.694 INFO:teuthology.orchestra.run.smithi028.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 2.2 MB/s | 249 kB 00:00 2024-09-15T21:44:58.838 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-15T21:44:58.838 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-15T21:44:58.861 INFO:teuthology.orchestra.run.smithi028.stdout:(154/156): NetworkManager-libnm-1.49.90-1.el9.x 3.9 MB/s | 1.8 MB 00:00 2024-09-15T21:44:58.895 INFO:teuthology.orchestra.run.smithi028.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 4.8 MB/s | 1.4 MB 00:00 2024-09-15T21:44:59.495 INFO:teuthology.orchestra.run.smithi028.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 3.0 MB/s | 2.4 MB 00:00 2024-09-15T21:44:59.501 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:44:59.501 INFO:teuthology.orchestra.run.smithi028.stdout:Total 13 MB/s | 226 MB 00:18 2024-09-15T21:45:00.722 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-15T21:45:00.890 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-15T21:45:00.890 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-15T21:45:01.018 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-15T21:45:01.146 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T21:45:01.264 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T21:45:01.422 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T21:45:01.464 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T21:45:01.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T21:45:01.549 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T21:45:01.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T21:45:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:02.121 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:02.150 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:02.225 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:02.255 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T21:45:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T21:45:02.460 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T21:45:02.488 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:02.592 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:02.614 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T21:45:02.677 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T21:45:02.707 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T21:45:02.733 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T21:45:02.925 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T21:45:03.075 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T21:45:03.198 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T21:45:03.214 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-15T21:45:03.214 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-15T21:45:03.298 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:03.387 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:03.613 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:03.724 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T21:45:03.822 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T21:45:03.859 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T21:45:03.902 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T21:45:03.976 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T21:45:04.020 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T21:45:04.060 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T21:45:04.104 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T21:45:04.137 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T21:45:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T21:45:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T21:45:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T21:45:04.326 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T21:45:04.361 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T21:45:04.438 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T21:45:04.472 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T21:45:04.524 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T21:45:04.553 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T21:45:04.624 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T21:45:04.676 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T21:45:04.760 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T21:45:04.892 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T21:45:04.954 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T21:45:05.378 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-15T21:45:05.478 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T21:45:05.502 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T21:45:05.558 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T21:45:05.597 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T21:45:05.631 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T21:45:05.638 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T21:45:05.677 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T21:45:05.684 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T21:45:05.718 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T21:45:05.719 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T21:45:05.745 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T21:45:05.786 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T21:45:05.887 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T21:45:05.964 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T21:45:06.019 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T21:45:06.091 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T21:45:06.118 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T21:45:06.123 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T21:45:06.159 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T21:45:06.206 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T21:45:06.220 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:06.242 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T21:45:06.285 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T21:45:06.325 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:06.336 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T21:45:06.363 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:06.454 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:06.482 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T21:45:06.484 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T21:45:06.633 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T21:45:06.680 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T21:45:06.708 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:06.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T21:45:06.782 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T21:45:06.811 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:06.834 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T21:45:06.887 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T21:45:06.897 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T21:45:06.920 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T21:45:06.931 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T21:45:06.937 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T21:45:06.971 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T21:45:07.005 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T21:45:07.038 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T21:45:07.078 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T21:45:07.114 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T21:45:07.230 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T21:45:07.249 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T21:45:07.296 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T21:45:07.324 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T21:45:07.358 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T21:45:07.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T21:45:07.397 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T21:45:07.429 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T21:45:07.448 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:07.459 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T21:45:07.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T21:45:07.526 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:07.527 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T21:45:07.569 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T21:45:07.612 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T21:45:07.698 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T21:45:07.742 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T21:45:07.746 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:07.773 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T21:45:07.865 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T21:45:07.876 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T21:45:07.924 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T21:45:07.955 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T21:45:07.992 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T21:45:08.035 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T21:45:08.100 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T21:45:08.144 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T21:45:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T21:45:08.220 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T21:45:08.252 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T21:45:08.289 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T21:45:08.365 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T21:45:08.419 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T21:45:08.450 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T21:45:08.494 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T21:45:08.572 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T21:45:08.605 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T21:45:08.649 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T21:45:08.658 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T21:45:08.677 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T21:45:08.706 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T21:45:08.740 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T21:45:08.747 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T21:45:08.800 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T21:45:08.842 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T21:45:08.875 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T21:45:08.887 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T21:45:08.932 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T21:45:09.009 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T21:45:09.021 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T21:45:09.060 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T21:45:09.067 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T21:45:09.260 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T21:45:09.299 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T21:45:09.376 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T21:45:09.592 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T21:45:09.673 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T21:45:09.701 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T21:45:09.750 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T21:45:09.788 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T21:45:09.830 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T21:45:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-15T21:45:09.966 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T21:45:09.989 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T21:45:10.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T21:45:10.030 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T21:45:10.065 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T21:45:10.068 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T21:45:10.115 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T21:45:10.131 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T21:45:10.140 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T21:45:10.149 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T21:45:10.190 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T21:45:10.204 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T21:45:10.235 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T21:45:10.272 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T21:45:10.318 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T21:45:10.354 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T21:45:10.397 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T21:45:10.464 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T21:45:10.548 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T21:45:10.601 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T21:45:10.649 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:10.684 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T21:45:10.757 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:10.783 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:10.876 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:10.913 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T21:45:10.958 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T21:45:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T21:45:11.184 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T21:45:11.208 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T21:45:11.211 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:11.300 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T21:45:11.317 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:11.339 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T21:45:11.345 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T21:45:11.364 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T21:45:11.388 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T21:45:11.389 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T21:45:11.417 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T21:45:11.447 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T21:45:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T21:45:11.473 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T21:45:11.480 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T21:45:11.508 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T21:45:11.541 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T21:45:11.599 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T21:45:11.625 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T21:45:11.637 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T21:45:11.666 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T21:45:11.700 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T21:45:11.739 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T21:45:11.771 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T21:45:11.801 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T21:45:11.818 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T21:45:11.836 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T21:45:11.851 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T21:45:11.868 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T21:45:11.888 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T21:45:11.918 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T21:45:11.928 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T21:45:11.952 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T21:45:11.978 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T21:45:12.010 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T21:45:12.040 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T21:45:12.045 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T21:45:12.046 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:12.083 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T21:45:12.114 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T21:45:12.121 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:12.218 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T21:45:12.274 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T21:45:12.358 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T21:45:12.471 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T21:45:12.529 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T21:45:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T21:45:12.566 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T21:45:12.601 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T21:45:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T21:45:12.636 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T21:45:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T21:45:12.703 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T21:45:12.754 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T21:45:12.794 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T21:45:12.837 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T21:45:12.870 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T21:45:12.905 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T21:45:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T21:45:13.010 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T21:45:13.037 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T21:45:13.054 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T21:45:13.074 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T21:45:13.087 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T21:45:13.186 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T21:45:13.242 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T21:45:13.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T21:45:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T21:45:13.374 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T21:45:13.415 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T21:45:13.482 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T21:45:13.530 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T21:45:13.574 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T21:45:13.602 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T21:45:13.607 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T21:45:13.654 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T21:45:13.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T21:45:13.725 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T21:45:13.731 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T21:45:13.801 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T21:45:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T21:45:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T21:45:14.394 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T21:45:14.510 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T21:45:14.556 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T21:45:14.650 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T21:45:14.684 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T21:45:14.728 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T21:45:14.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T21:45:14.816 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T21:45:14.980 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T21:45:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T21:45:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T21:45:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T21:45:15.215 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T21:45:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T21:45:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T21:45:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T21:45:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T21:45:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T21:45:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T21:45:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T21:45:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T21:45:15.735 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T21:45:15.763 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:15.802 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T21:45:15.815 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T21:45:15.843 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T21:45:15.892 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T21:45:15.900 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T21:45:15.900 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T21:45:15.901 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T21:45:15.901 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T21:45:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T21:45:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T21:45:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T21:45:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T21:45:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T21:45:16.203 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T21:45:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T21:45:16.247 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T21:45:16.336 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T21:45:16.346 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T21:45:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T21:45:16.379 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T21:45:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T21:45:16.415 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T21:45:16.459 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T21:45:16.498 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T21:45:16.540 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T21:45:16.570 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T21:45:16.604 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T21:45:16.637 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T21:45:16.699 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T21:45:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T21:45:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T21:45:16.870 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T21:45:16.900 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T21:45:16.903 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T21:45:16.941 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T21:45:17.007 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T21:45:17.013 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T21:45:17.043 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T21:45:17.060 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T21:45:17.819 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T21:45:17.864 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T21:45:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T21:45:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T21:45:18.071 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T21:45:18.116 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T21:45:18.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T21:45:18.242 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T21:45:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T21:45:18.473 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T21:45:18.552 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T21:45:19.241 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T21:45:19.364 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T21:45:19.387 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:19.423 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:19.972 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:20.009 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T21:45:20.093 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T21:45:20.093 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T21:45:20.094 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T21:45:20.094 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:20.127 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T21:45:20.682 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T21:45:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T21:45:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T21:45:20.834 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T21:45:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T21:45:21.196 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T21:45:21.280 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T21:45:21.310 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T21:45:21.346 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T21:45:21.856 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T21:45:21.899 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T21:45:21.970 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T21:45:22.001 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T21:45:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T21:45:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T21:45:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T21:45:25.193 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T21:45:25.196 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T21:45:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T21:45:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T21:45:27.695 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-09-15T21:45:27.695 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-09-15T21:45:27.695 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-09-15T21:45:27.695 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-09-15T21:45:27.695 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-09-15T21:45:27.696 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-09-15T21:45:27.696 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-09-15T21:45:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:27.869 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T21:45:27.885 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T21:45:27.885 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:27.885 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T21:45:27.885 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:29.372 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T21:45:29.399 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T21:45:29.399 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:29.399 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T21:45:29.400 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:45:29.400 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:45:29.400 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:29.633 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T21:45:29.656 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T21:45:29.657 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:29.657 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T21:45:29.657 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:45:29.657 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:45:29.657 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T21:45:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T21:45:30.124 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:30.125 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T21:45:30.125 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:45:30.125 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:45:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:30.239 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T21:45:30.297 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T21:45:30.352 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T21:45:30.541 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T21:45:30.594 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T21:45:30.673 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T21:45:30.761 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T21:45:30.902 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T21:45:30.953 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T21:45:30.984 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T21:45:31.012 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T21:45:31.467 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T21:45:31.495 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T21:45:31.517 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T21:45:31.517 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-09-15T21:45:31.517 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-09-15T21:45:31.517 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-09-15T21:45:31.517 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-09-15T21:45:31.517 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-09-15T21:45:31.517 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-09-15T21:45:31.518 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-09-15T21:45:31.518 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:31.535 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T21:45:31.606 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T21:45:31.643 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T21:45:31.686 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T21:45:31.702 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T21:45:31.702 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:31.702 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T21:45:31.703 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:32.699 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T21:45:32.737 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T21:45:33.164 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T21:45:33.191 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T21:45:33.191 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:33.191 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T21:45:33.191 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:45:33.191 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:45:33.192 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:33.423 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T21:45:33.447 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T21:45:33.447 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:33.447 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T21:45:33.447 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:45:33.447 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:45:33.447 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T21:45:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T21:45:33.842 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T21:45:33.882 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T21:45:33.906 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T21:45:33.906 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:33.906 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T21:45:33.906 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:45:33.906 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:45:33.907 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:34.001 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T21:45:34.012 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T21:45:34.031 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T21:45:34.056 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T21:45:34.056 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:34.056 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T21:45:34.056 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:45:34.056 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:45:34.056 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:34.069 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T21:45:34.123 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T21:45:34.124 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T21:45:34.136 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T21:45:34.244 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T21:45:34.298 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T21:45:34.311 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T21:45:34.366 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T21:45:34.444 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T21:45:34.532 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T21:45:34.673 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T21:45:34.725 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T21:45:34.756 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T21:45:34.784 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T21:45:35.188 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T21:45:35.215 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T21:45:35.227 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T21:45:35.248 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T21:45:35.260 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T21:45:35.284 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T21:45:35.285 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:35.285 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T21:45:35.285 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:45:35.285 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:45:35.285 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:35.309 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T21:45:35.327 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T21:45:35.338 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T21:45:35.351 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T21:45:35.351 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:35.351 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T21:45:35.351 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:35.412 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T21:45:35.435 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T21:45:35.435 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:35.435 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T21:45:35.436 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T21:45:35.436 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T21:45:35.436 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:35.737 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T21:45:35.762 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T21:45:35.762 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:35.762 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T21:45:35.762 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:45:35.762 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:45:35.762 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:36.384 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T21:45:36.423 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T21:45:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-09-15T21:45:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:37.385 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T21:45:37.415 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T21:45:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T21:45:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T21:45:37.504 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:37.504 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T21:45:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:37.519 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T21:45:37.678 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T21:45:37.712 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T21:45:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T21:45:37.734 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:37.734 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T21:45:37.734 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:45:37.734 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:45:37.734 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:37.785 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T21:45:37.797 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T21:45:37.893 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T21:45:37.941 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T21:45:38.866 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T21:45:38.904 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T21:45:38.929 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T21:45:38.929 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:38.929 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T21:45:38.929 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:45:38.929 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:45:38.929 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:38.971 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T21:45:38.994 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T21:45:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T21:45:38.995 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:39.040 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T21:45:39.046 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T21:45:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T21:45:39.069 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:39.069 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T21:45:39.069 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:45:39.069 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:45:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:39.070 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T21:45:39.070 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:39.070 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T21:45:39.070 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T21:45:39.070 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T21:45:39.070 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:39.316 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T21:45:39.341 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T21:45:39.341 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:39.341 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T21:45:39.341 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:45:39.341 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:45:39.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:39.380 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T21:45:39.404 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T21:45:39.404 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:39.404 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T21:45:39.405 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:45:39.405 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:45:39.405 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T21:45:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T21:45:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T21:45:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:45:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:45:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:39.947 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T21:45:40.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T21:45:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T21:45:40.262 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T21:45:40.317 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T21:45:40.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T21:45:40.398 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T21:45:40.408 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T21:45:40.449 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T21:45:40.481 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T21:45:40.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T21:45:40.508 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T21:45:40.629 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T21:45:40.689 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T21:45:40.696 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T21:45:40.724 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T21:45:40.748 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T21:45:40.751 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T21:45:40.782 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T21:45:40.807 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T21:45:40.845 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T21:45:41.192 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T21:45:41.225 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T21:45:41.257 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T21:45:41.321 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T21:45:41.347 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T21:45:41.434 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:41.477 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:41.499 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:42.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T21:45:42.442 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T21:45:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T21:45:42.483 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T21:45:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T21:45:42.566 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T21:45:42.595 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T21:45:42.621 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T21:45:42.655 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T21:45:42.655 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T21:45:42.740 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T21:45:42.740 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T21:45:42.833 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T21:45:42.857 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T21:45:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:43.044 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:43.045 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T21:45:43.135 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T21:45:43.207 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T21:45:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T21:45:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T21:45:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T21:45:43.860 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T21:45:43.886 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T21:45:43.886 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:43.886 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T21:45:43.886 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:45:43.886 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:45:43.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:43.935 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T21:45:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T21:45:43.959 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T21:45:44.000 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T21:45:44.041 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T21:45:44.049 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T21:45:44.073 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T21:45:44.123 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T21:45:44.129 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T21:45:44.310 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T21:45:44.363 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T21:45:44.393 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T21:45:44.423 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T21:45:44.469 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T21:45:45.084 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:45.107 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T21:45:45.124 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:45.137 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T21:45:45.146 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T21:45:45.163 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:45.163 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T21:45:45.163 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:45:45.163 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:45:45.163 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:45.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T21:45:45.229 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T21:45:45.229 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:45.229 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T21:45:45.230 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:45.280 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T21:45:45.305 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T21:45:45.305 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:45.305 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T21:45:45.305 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T21:45:45.306 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T21:45:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:45.623 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T21:45:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T21:45:45.648 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:45:45.648 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T21:45:45.648 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:45:45.648 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:45:45.648 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:45:45.979 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T21:45:46.070 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T21:45:46.111 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T21:45:46.193 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T21:45:46.223 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T21:45:46.248 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T21:45:46.274 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T21:45:46.275 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T21:45:46.358 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T21:45:46.358 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T21:45:46.426 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T21:45:46.450 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T21:45:46.456 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:46.456 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:46.636 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:46.636 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T21:45:46.720 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T21:45:46.783 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T21:45:49.065 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T21:45:49.066 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T21:45:49.067 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T21:45:49.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T21:45:49.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T21:45:49.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T21:45:49.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T21:45:49.081 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T21:45:49.082 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T21:45:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T21:45:49.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T21:45:49.085 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T21:45:49.085 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T21:45:49.085 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T21:45:49.085 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T21:45:49.085 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T21:45:49.085 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T21:45:49.086 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T21:45:49.087 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T21:45:49.088 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T21:45:49.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T21:45:49.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T21:45:49.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T21:45:49.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T21:45:49.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T21:45:49.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T21:45:49.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T21:45:49.092 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T21:45:49.092 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T21:45:49.092 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T21:45:49.093 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T21:45:49.094 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T21:45:49.096 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T21:45:49.096 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T21:45:49.096 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T21:45:49.096 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T21:45:49.096 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T21:45:49.096 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T21:45:49.096 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T21:45:49.097 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T21:45:49.097 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T21:45:50.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T21:45:50.552 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T21:45:50.601 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T21:45:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T21:45:50.659 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T21:45:50.859 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T21:45:50.897 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T21:45:50.928 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T21:45:50.957 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T21:45:51.003 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T21:45:51.514 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:51.514 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:51.514 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-09-15T21:45:51.514 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:51.514 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:51.514 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.515 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T21:45:51.517 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.518 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T21:45:51.519 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T21:45:51.520 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T21:45:51.521 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T21:45:51.522 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T21:45:51.522 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T21:45:51.522 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.522 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.523 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.523 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T21:45:51.523 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T21:45:51.523 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T21:45:51.523 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T21:45:51.524 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T21:45:51.525 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T21:45:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T21:45:51.527 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T21:45:51.528 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T21:45:51.529 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T21:45:51.529 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T21:45:51.529 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T21:45:51.529 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T21:45:51.529 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T21:45:51.529 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:45:51.529 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-15T21:45:51.645 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:51.684 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:51.705 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T21:45:51.937 DEBUG:teuthology.parallel:result is None 2024-09-15T21:45:52.184 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:52.184 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T21:45:52.184 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T21:45:52.184 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T21:45:52.184 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T21:45:52.184 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T21:45:52.185 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T21:45:52.186 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T21:45:52.199 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T21:45:52.200 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T21:45:52.201 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T21:45:52.202 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T21:45:52.203 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T21:45:52.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T21:45:52.205 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T21:45:52.206 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T21:45:52.207 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T21:45:52.208 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T21:45:52.209 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T21:45:52.210 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T21:45:52.616 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T21:45:52.713 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T21:45:52.760 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T21:45:52.849 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T21:45:52.882 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T21:45:52.908 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T21:45:52.934 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T21:45:52.934 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T21:45:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T21:45:53.027 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T21:45:53.102 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T21:45:53.127 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T21:45:53.132 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:53.132 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:53.320 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T21:45:53.321 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T21:45:53.405 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T21:45:53.481 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:55.043 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:55.043 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:55.043 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-09-15T21:45:55.043 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:55.043 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:55.043 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T21:45:55.044 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.045 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T21:45:55.046 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.047 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T21:45:55.048 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T21:45:55.049 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T21:45:55.050 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T21:45:55.051 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T21:45:55.052 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T21:45:55.053 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T21:45:55.054 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T21:45:55.055 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T21:45:55.056 INFO:teuthology.orchestra.run.smithi174.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T21:45:55.056 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T21:45:55.056 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T21:45:55.056 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T21:45:55.056 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:45:55.056 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-15T21:45:55.490 DEBUG:teuthology.parallel:result is None 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T21:45:58.883 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T21:45:58.884 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T21:45:58.885 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T21:45:58.885 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T21:45:58.885 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T21:45:58.885 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T21:45:58.885 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T21:45:58.885 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T21:45:58.897 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T21:45:58.897 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T21:45:58.897 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T21:45:58.898 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T21:45:58.899 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T21:45:58.900 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T21:45:58.901 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T21:45:58.902 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T21:45:58.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T21:45:58.904 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T21:45:58.905 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T21:45:58.906 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T21:45:58.907 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T21:45:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T21:45:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T21:45:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T21:45:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T21:45:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T21:45:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T21:45:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.291 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.292 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.293 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T21:46:01.294 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.295 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T21:46:01.296 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T21:46:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T21:46:01.299 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T21:46:01.300 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T21:46:01.301 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T21:46:01.302 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T21:46:01.304 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T21:46:01.304 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T21:46:01.304 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T21:46:01.304 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T21:46:01.304 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T21:46:01.304 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T21:46:01.304 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T21:46:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:46:01.305 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-15T21:46:01.702 DEBUG:teuthology.parallel:result is None 2024-09-15T21:46:01.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:46:01.855 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T21:46:01.878 INFO:teuthology.orchestra.run.smithi028.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T21:46:01.879 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T21:46:01.879 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T21:46:01.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:46:02.031 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T21:46:02.057 INFO:teuthology.orchestra.run.smithi174.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T21:46:02.057 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T21:46:02.058 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T21:46:02.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:46:02.199 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T21:46:02.226 INFO:teuthology.orchestra.run.smithi203.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T21:46:02.226 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T21:46:02.226 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T21:46:02.228 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T21:46:02.228 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:46:02.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T21:46:02.259 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:46:02.259 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T21:46:02.293 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:46:02.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T21:46:02.324 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T21:46:02.325 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:46:02.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T21:46:02.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T21:46:02.421 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:46:02.421 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T21:46:02.451 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T21:46:02.519 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:46:02.519 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T21:46:02.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T21:46:02.621 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T21:46:02.621 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:46:02.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T21:46:02.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T21:46:02.717 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:46:02.718 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T21:46:02.748 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T21:46:02.818 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:46:02.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T21:46:02.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T21:46:02.917 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T21:46:02.918 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:46:02.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T21:46:02.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T21:46:03.016 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:46:03.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T21:46:03.045 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T21:46:03.113 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:46:03.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T21:46:03.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T21:46:03.212 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-15T21:46:03.304 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-15T21:46:03.304 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:46:03.305 INFO:tasks.cephadm:Cluster fsid is e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:46:03.305 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-15T21:46:03.305 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.174', 'mon.c': '172.21.15.203'} 2024-09-15T21:46:03.305 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-09-15T21:46:03.306 INFO:tasks.cephadm:First mgr is x 2024-09-15T21:46:03.306 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-15T21:46:03.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-09-15T21:46:03.338 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-09-15T21:46:03.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-09-15T21:46:03.399 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-15T21:46:03.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:46:03.541 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56081/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-326-g8921c8ef', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-326.g8921c8ef'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-15 16:06:42.385367', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-15T21:46:03.684 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 8921c8ef947807b717db041d85f0761b7914d5fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:46:03.685 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-15T21:46:03.686 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-15T21:46:03.686 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.082 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 21:46 /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.082 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.467 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 21:46 /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.467 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.842 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 21:46 /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.842 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.863 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.883 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T21:46:04.925 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc on all hosts... 2024-09-15T21:46:04.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T21:46:04.929 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T21:46:04.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T21:46:06.550 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T21:46:06.594 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T21:46:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T21:46:45.186 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:46:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T21:46:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T21:46:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-09-15T21:46:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T21:46:45.186 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T21:46:45.186 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:46:46.035 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T21:46:46.035 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T21:46:46.035 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T21:46:46.035 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-09-15T21:46:46.035 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T21:46:46.035 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-15T21:46:46.035 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T21:46:49.839 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-15T21:46:49.839 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T21:46:49.840 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T21:46:49.840 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-09-15T21:46:49.840 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T21:46:49.840 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-15T21:46:49.840 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-15T21:46:49.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-09-15T21:46:49.902 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-09-15T21:46:49.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-09-15T21:46:49.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-09-15T21:46:49.994 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-09-15T21:46:50.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-09-15T21:46:50.049 INFO:tasks.cephadm:Writing seed config... 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-15T21:46:50.053 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-09-15T21:46:50.054 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-15T21:46:50.055 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-15T21:46:50.056 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-09-15T21:46:50.057 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:46:50.057 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-15T21:46:50.075 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e78b7764-73ab-11ef-bceb-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-15T21:46:50.076 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service 2024-09-15T21:46:50.118 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.x.service 2024-09-15T21:46:50.161 INFO:tasks.cephadm:Bootstrapping... 2024-09-15T21:46:50.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc -v bootstrap --fsid e78b7764-73ab-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-15T21:46:50.374 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:46:50.375 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc', '-v', 'bootstrap', '--fsid', 'e78b7764-73ab-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-09-15T21:46:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-15T21:46:50.375 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-09-15T21:46:50.399 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 5.2.2 2024-09-15T21:46:50.399 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-09-15T21:46:50.399 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-09-15T21:46:50.407 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-15T21:46:50.407 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-15T21:46:50.413 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-15T21:46:50.414 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-15T21:46:50.421 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-09-15T21:46:50.427 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-09-15T21:46:50.428 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-09-15T21:46:50.428 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-09-15T21:46:50.451 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 5.2.2 2024-09-15T21:46:50.451 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-15T21:46:50.451 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-09-15T21:46:50.451 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-09-15T21:46:50.458 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-15T21:46:50.458 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-15T21:46:50.464 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-15T21:46:50.464 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-15T21:46:50.471 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-09-15T21:46:50.478 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-09-15T21:46:50.478 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-09-15T21:46:50.478 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-09-15T21:46:50.479 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:46:50.479 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 140399302829056 on /run/cephadm/e78b7764-73ab-11ef-bceb-c7b262605968.lock 2024-09-15T21:46:50.479 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 140399302829056 acquired on /run/cephadm/e78b7764-73ab-11ef-bceb-c7b262605968.lock 2024-09-15T21:46:50.479 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-09-15T21:46:50.479 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-09-15T21:46:50.479 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-09-15T21:46:50.483 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-09-15T21:46:50.483 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-09-15T21:46:50.485 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-15T21:46:50.485 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-15T21:46:50.486 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-09-15T21:46:50.488 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-15T21:46:50.488 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-15T21:46:50.488 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-15T21:46:50.488 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-15T21:46:50.488 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-15T21:46:50.488 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-15T21:46:50.489 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-09-15T21:46:50.489 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-09-15T21:46:50.489 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-15T21:46:50.489 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-15T21:46:50.490 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T21:46:50.979 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-15T21:46:50.979 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T21:46:50.979 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-09-15T21:46:50.979 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-15T21:46:50.979 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:c0574da0c3174f3dc49daa274908f97d06456bd6a3cfab87e94022c1369a83a6 2024-09-15T21:46:50.979 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-15T21:46:50.979 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-15T21:46:51.824 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-15T21:46:51.824 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-15T21:46:51.825 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-09-15T21:46:55.924 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-09-15T21:46:55.924 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-09-15T21:47:04.322 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDSVedmgHzAMxAACSYYXD3F1BmMeuiIHvx1EA== 2024-09-15T21:47:05.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDYVedmRHwmKRAAuoKKXMXKS2o/IeAA+vE5Wg== 2024-09-15T21:47:06.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDZVedm4BrpIRAA5KX7O4MOj61ozYPY2I9hdQ== 2024-09-15T21:47:06.250 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:47:07.248 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-09-15T21:47:08.287 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-09-15T21:47:08.501 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-15T21:47:08.669 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-15T21:47:08.847 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e78b7764-73ab-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-e78b7764-73ab-11ef-bceb-c7b262605968.target. 2024-09-15T21:47:08.847 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e78b7764-73ab-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-e78b7764-73ab-11ef-bceb-c7b262605968.target. 2024-09-15T21:47:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a 2024-09-15T21:47:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-15T21:47:09.319 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e78b7764-73ab-11ef-bceb-c7b262605968.target.wants/ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-e78b7764-73ab-11ef-bceb-c7b262605968@.service. 2024-09-15T21:47:09.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:09 smithi028 podman[30581]: 2024-09-15 21:47:09.645515039 +0000 UTC m=+0.016676311 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:47:10.209 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:10 smithi028 podman[30581]: 2024-09-15 21:47:10.089494331 +0000 UTC m=+0.460655590 container create 6649b742f94ce84aa97b5234f59e50b89dd1f51561e20b73ad5883a0114315b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-15T21:47:10.532 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T21:47:10.532 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-15T21:47:10.540 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T21:47:10.540 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-15T21:47:10.540 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-15T21:47:10.540 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-09-15T21:47:10.540 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-09-15T21:47:10.540 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-09-15T21:47:10.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:10 smithi028 podman[30581]: 2024-09-15 21:47:10.408300408 +0000 UTC m=+0.779461679 container init 6649b742f94ce84aa97b5234f59e50b89dd1f51561e20b73ad5883a0114315b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-15T21:47:10.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:10 smithi028 podman[30581]: 2024-09-15 21:47:10.411862722 +0000 UTC m=+0.783023982 container start 6649b742f94ce84aa97b5234f59e50b89dd1f51561e20b73ad5883a0114315b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-15T21:47:10.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:10 smithi028 ceph-mon[30601]: mkfs e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:10.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:10 smithi028 ceph-mon[30601]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-15T21:47:10.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:10 smithi028 bash[30581]: 6649b742f94ce84aa97b5234f59e50b89dd1f51561e20b73ad5883a0114315b5 2024-09-15T21:47:10.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:10 smithi028 systemd[1]: Started Ceph mon.a for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:47:11.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-09-15T21:47:11.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:11.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-15T21:47:11.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.581645s) 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-15T21:47:11.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.957+0000 7f44dea26640 1 Processor -- start 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.957+0000 7f44dea26640 1 -- start start 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.957+0000 7f44dea26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d8107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.958+0000 7f44dea26640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81080d0 con 0x7f44d8107700 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.958+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d8107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.958+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d8107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32806/0 (socket says 172.21.15.28:32806) 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.958+0000 7f44d7fff640 1 -- 172.21.15.28:0/2581170676 learned_addr learned my addr 172.21.15.28:0/2581170676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.958+0000 7f44d7fff640 1 -- 172.21.15.28:0/2581170676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d8108900 con 0x7f44d8107700 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.960+0000 7f44d7fff640 1 --2- 172.21.15.28:0/2581170676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d8107b00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009820 tx=0x7f44c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2dee02121542dd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.960+0000 7f44d6ffd640 1 -- 172.21.15.28:0/2581170676 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44c8037070 con 0x7f44d8107700 2024-09-15T21:47:11.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.960+0000 7f44d6ffd640 1 -- 172.21.15.28:0/2581170676 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c8032440 con 0x7f44d8107700 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.960+0000 7f44d6ffd640 1 -- 172.21.15.28:0/2581170676 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44c8032740 con 0x7f44d8107700 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.961+0000 7f44dea26640 1 -- 172.21.15.28:0/2581170676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 msgr2=0x7f44d8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.961+0000 7f44dea26640 1 --2- 172.21.15.28:0/2581170676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d8107b00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009820 tx=0x7f44c802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.961+0000 7f44dea26640 1 -- 172.21.15.28:0/2581170676 shutdown_connections 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.961+0000 7f44dea26640 1 --2- 172.21.15.28:0/2581170676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d8107b00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.961+0000 7f44dea26640 1 -- 172.21.15.28:0/2581170676 >> 172.21.15.28:0/2581170676 conn(0x7f44d8102ed0 msgr2=0x7f44d81052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.961+0000 7f44dea26640 1 -- 172.21.15.28:0/2581170676 shutdown_connections 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.961+0000 7f44dea26640 1 -- 172.21.15.28:0/2581170676 wait complete. 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.962+0000 7f44dea26640 1 Processor -- start 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.962+0000 7f44dea26640 1 -- start start 2024-09-15T21:47:11.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.962+0000 7f44dea26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.962+0000 7f44dea26640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44d8108ea0 con 0x7f44d8107700 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.962+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.962+0000 7f44d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32822/0 (socket says 172.21.15.28:32822) 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.962+0000 7f44d7fff640 1 -- 172.21.15.28:0/1033759344 learned_addr learned my addr 172.21.15.28:0/1033759344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44d7fff640 1 -- 172.21.15.28:0/1033759344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d807e0d0 con 0x7f44d8107700 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44d7fff640 1 --2- 172.21.15.28:0/1033759344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d807db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f44c80090b0 tx=0x7f44c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44d57fa640 1 -- 172.21.15.28:0/1033759344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44c8041070 con 0x7f44d8107700 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44d57fa640 1 -- 172.21.15.28:0/1033759344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c803c7a0 con 0x7f44d8107700 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44d57fa640 1 -- 172.21.15.28:0/1033759344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44c8037040 con 0x7f44d8107700 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d807c2c0 con 0x7f44d8107700 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d807c800 con 0x7f44d8107700 2024-09-15T21:47:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44d57fa640 1 -- 172.21.15.28:0/1033759344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f44c8044070 con 0x7f44d8107700 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.963+0000 7f44d57fa640 1 -- 172.21.15.28:0/1033759344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f44c803d3f0 con 0x7f44d8107700 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.964+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d81094f0 con 0x7f44d8107700 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:10.966+0000 7f44d57fa640 1 -- 172.21.15.28:0/1033759344 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f44c8046450 con 0x7f44d8107700 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.028+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f44d807cf60 con 0x7f44d8107700 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44d57fa640 1 -- 172.21.15.28:0/1033759344 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f44c8046630 con 0x7f44d8107700 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 msgr2=0x7f44d807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44dea26640 1 --2- 172.21.15.28:0/1033759344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d807db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f44c80090b0 tx=0x7f44c80046c0 comp rx=0 tx=0).stop 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 shutdown_connections 2024-09-15T21:47:11.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44dea26640 1 --2- 172.21.15.28:0/1033759344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44d8107700 0x7f44d807db90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:11.614 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 >> 172.21.15.28:0/1033759344 conn(0x7f44d8102ed0 msgr2=0x7f44d818f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:11.614 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 shutdown_connections 2024-09-15T21:47:11.614 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.029+0000 7f44dea26640 1 -- 172.21.15.28:0/1033759344 wait complete. 2024-09-15T21:47:11.614 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-09-15T21:47:11.614 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-09-15T21:47:11.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: monmap epoch 1 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: last_changed 2024-09-15T21:47:06.538730+0000 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: min_mon_release 19 (squid) 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: election_strategy: 1 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: fsmap 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: osdmap e1: 0 total, 0 up, 0 in 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: mgrmap e1: no daemons active 2024-09-15T21:47:11.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:11 smithi028 ceph-mon[30601]: from='client.? 172.21.15.28:0/1033759344' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-15T21:47:12.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:12.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-09-15T21:47:12.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:12.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-09-15T21:47:12.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-15T21:47:12.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.979+0000 7f55496d8640 1 Processor -- start 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.979+0000 7f55496d8640 1 -- start start 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.980+0000 7f55496d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f5544107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.980+0000 7f55496d8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55441080d0 con 0x7f5544107700 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.980+0000 7f5542ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f5544107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.980+0000 7f5542ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f5544107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32828/0 (socket says 172.21.15.28:32828) 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.980+0000 7f5542ffd640 1 -- 172.21.15.28:0/2005784054 learned_addr learned my addr 172.21.15.28:0/2005784054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.980+0000 7f5542ffd640 1 -- 172.21.15.28:0/2005784054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5544108900 con 0x7f5544107700 2024-09-15T21:47:12.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.981+0000 7f5542ffd640 1 --2- 172.21.15.28:0/2005784054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f5544107b00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5534009820 tx=0x7f553402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a7d8466e324c80 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.981+0000 7f5541ffb640 1 -- 172.21.15.28:0/2005784054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5534037070 con 0x7f5544107700 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.981+0000 7f5541ffb640 1 -- 172.21.15.28:0/2005784054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5534032440 con 0x7f5544107700 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.981+0000 7f5541ffb640 1 -- 172.21.15.28:0/2005784054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5534032740 con 0x7f5544107700 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.981+0000 7f55496d8640 1 -- 172.21.15.28:0/2005784054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 msgr2=0x7f5544107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.981+0000 7f55496d8640 1 --2- 172.21.15.28:0/2005784054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f5544107b00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5534009820 tx=0x7f553402a270 comp rx=0 tx=0).stop 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 -- 172.21.15.28:0/2005784054 shutdown_connections 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 --2- 172.21.15.28:0/2005784054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f5544107b00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 -- 172.21.15.28:0/2005784054 >> 172.21.15.28:0/2005784054 conn(0x7f5544102ed0 msgr2=0x7f55441052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 -- 172.21.15.28:0/2005784054 shutdown_connections 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 -- 172.21.15.28:0/2005784054 wait complete. 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 Processor -- start 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 -- start start 2024-09-15T21:47:12.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f554419af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.982+0000 7f55496d8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5544108ea0 con 0x7f5544107700 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f5542ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f554419af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f5542ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f554419af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32830/0 (socket says 172.21.15.28:32830) 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f5542ffd640 1 -- 172.21.15.28:0/1496064974 learned_addr learned my addr 172.21.15.28:0/1496064974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f5542ffd640 1 -- 172.21.15.28:0/1496064974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554419b450 con 0x7f5544107700 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f5542ffd640 1 --2- 172.21.15.28:0/1496064974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f554419af10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f55340090b0 tx=0x7f55340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f552bfff640 1 -- 172.21.15.28:0/1496064974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5534041070 con 0x7f5544107700 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f552bfff640 1 -- 172.21.15.28:0/1496064974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f553403c7a0 con 0x7f5544107700 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f554419b6e0 con 0x7f5544107700 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f552bfff640 1 -- 172.21.15.28:0/1496064974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5534037040 con 0x7f5544107700 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f554419e3d0 con 0x7f5544107700 2024-09-15T21:47:12.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.983+0000 7f552bfff640 1 -- 172.21.15.28:0/1496064974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5534044070 con 0x7f5544107700 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.984+0000 7f552bfff640 1 -- 172.21.15.28:0/1496064974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f553403d3f0 con 0x7f5544107700 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.984+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55441094f0 con 0x7f5544107700 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:11.986+0000 7f552bfff640 1 -- 172.21.15.28:0/1496064974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f5534046450 con 0x7f5544107700 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.046+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f554419faa0 con 0x7f5544107700 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.049+0000 7f552bfff640 1 -- 172.21.15.28:0/1496064974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7f5534046630 con 0x7f5544107700 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.050+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 msgr2=0x7f554419af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.050+0000 7f55496d8640 1 --2- 172.21.15.28:0/1496064974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f554419af10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f55340090b0 tx=0x7f55340046c0 comp rx=0 tx=0).stop 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.050+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 shutdown_connections 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.050+0000 7f55496d8640 1 --2- 172.21.15.28:0/1496064974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5544107700 0x7f554419af10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.050+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 >> 172.21.15.28:0/1496064974 conn(0x7f5544102ed0 msgr2=0x7f554418f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.050+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 shutdown_connections 2024-09-15T21:47:12.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:12.050+0000 7f55496d8640 1 -- 172.21.15.28:0/1496064974 wait complete. 2024-09-15T21:47:12.609 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-09-15T21:47:12.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:12 smithi028 ceph-mon[30601]: from='client.? 172.21.15.28:0/1496064974' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-15T21:47:12.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:12 smithi028 ceph-mon[30601]: from='client.? 172.21.15.28:0/1496064974' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-15T21:47:13.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:13 smithi028 ceph-mon[30601]: from='client.? 172.21.15.28:0/3668448249' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.095+0000 7fb90bea3640 1 Processor -- start 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.095+0000 7fb90bea3640 1 -- start start 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.095+0000 7fb90bea3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb904107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.095+0000 7fb90bea3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9041080d0 con 0x7fb904107700 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.095+0000 7fb909c3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb904107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.095+0000 7fb909c3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb904107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32846/0 (socket says 172.21.15.28:32846) 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.095+0000 7fb909c3f640 1 -- 172.21.15.28:0/120426309 learned_addr learned my addr 172.21.15.28:0/120426309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:13.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.096+0000 7fb909c3f640 1 -- 172.21.15.28:0/120426309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb904108900 con 0x7fb904107700 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.096+0000 7fb909c3f640 1 --2- 172.21.15.28:0/120426309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb904107b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb900009820 tx=0x7fb90002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e250c674c1045a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.096+0000 7fb908c3d640 1 -- 172.21.15.28:0/120426309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900037070 con 0x7fb904107700 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.096+0000 7fb908c3d640 1 -- 172.21.15.28:0/120426309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb900032900 con 0x7fb904107700 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.096+0000 7fb908c3d640 1 -- 172.21.15.28:0/120426309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900030850 con 0x7fb904107700 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.097+0000 7fb90bea3640 1 -- 172.21.15.28:0/120426309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 msgr2=0x7fb904107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.097+0000 7fb90bea3640 1 --2- 172.21.15.28:0/120426309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb904107b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb900009820 tx=0x7fb90002a270 comp rx=0 tx=0).stop 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.097+0000 7fb90bea3640 1 -- 172.21.15.28:0/120426309 shutdown_connections 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.097+0000 7fb90bea3640 1 --2- 172.21.15.28:0/120426309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb904107b00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.097+0000 7fb90bea3640 1 -- 172.21.15.28:0/120426309 >> 172.21.15.28:0/120426309 conn(0x7fb904102ed0 msgr2=0x7fb9041052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.097+0000 7fb90bea3640 1 -- 172.21.15.28:0/120426309 shutdown_connections 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.097+0000 7fb90bea3640 1 -- 172.21.15.28:0/120426309 wait complete. 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb90bea3640 1 Processor -- start 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb90bea3640 1 -- start start 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb90bea3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb90419b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb90bea3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb90410bcb0 con 0x7fb904107700 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb909c3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb90419b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb909c3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb90419b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32850/0 (socket says 172.21.15.28:32850) 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb909c3f640 1 -- 172.21.15.28:0/3668448249 learned_addr learned my addr 172.21.15.28:0/3668448249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:13.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.098+0000 7fb909c3f640 1 -- 172.21.15.28:0/3668448249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90419bc60 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb909c3f640 1 --2- 172.21.15.28:0/3668448249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb90419b720 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb900009940 tx=0x7fb900032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb8faffd640 1 -- 172.21.15.28:0/3668448249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900042070 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb8faffd640 1 -- 172.21.15.28:0/3668448249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb9000308c0 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb90419bef0 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb8faffd640 1 -- 172.21.15.28:0/3668448249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb900037040 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb90419ebe0 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb8faffd640 1 -- 172.21.15.28:0/3668448249 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb900045070 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.099+0000 7fb8faffd640 1 -- 172.21.15.28:0/3668448249 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb9000463b0 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.100+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb90410be30 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.102+0000 7fb8faffd640 1 -- 172.21.15.28:0/3668448249 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fb9000327b0 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.162+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb904107b80 con 0x7fb904107700 2024-09-15T21:47:13.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.162+0000 7fb8faffd640 1 -- 172.21.15.28:0/3668448249 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fb904107b80 con 0x7fb904107700 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.163+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 msgr2=0x7fb90419b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.163+0000 7fb90bea3640 1 --2- 172.21.15.28:0/3668448249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb90419b720 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb900009940 tx=0x7fb900032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.163+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 shutdown_connections 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.163+0000 7fb90bea3640 1 --2- 172.21.15.28:0/3668448249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb904107700 0x7fb90419b720 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.163+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 >> 172.21.15.28:0/3668448249 conn(0x7fb904102ed0 msgr2=0x7fb904103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.163+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 shutdown_connections 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:13.163+0000 7fb90bea3640 1 -- 172.21.15.28:0/3668448249 wait complete. 2024-09-15T21:47:13.731 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-09-15T21:47:13.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:13 smithi028 systemd[1]: Stopping Ceph mon.a for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:47:14.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:13 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a[30597]: 2024-09-15T21:47:13.979+0000 7f2ce942d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T21:47:14.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:13 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a[30597]: 2024-09-15T21:47:13.979+0000 7f2ce942d640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-15T21:47:14.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:14 smithi028 podman[30843]: 2024-09-15 21:47:14.226467113 +0000 UTC m=+0.337064797 container died 6649b742f94ce84aa97b5234f59e50b89dd1f51561e20b73ad5883a0114315b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-15T21:47:14.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:14 smithi028 podman[30843]: 2024-09-15 21:47:14.57966162 +0000 UTC m=+0.690259301 container cleanup 6649b742f94ce84aa97b5234f59e50b89dd1f51561e20b73ad5883a0114315b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-15T21:47:14.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:14 smithi028 bash[30843]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a 2024-09-15T21:47:15.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:14 smithi028 podman[30855]: 2024-09-15 21:47:14.988930343 +0000 UTC m=+0.759712160 container remove 6649b742f94ce84aa97b5234f59e50b89dd1f51561e20b73ad5883a0114315b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731) 2024-09-15T21:47:15.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-15T21:47:15.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 systemd[1]: Stopped Ceph mon.a for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:47:15.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 systemd[1]: Starting Ceph mon.a for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:47:15.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 podman[30950]: 2024-09-15 21:47:15.37310115 +0000 UTC m=+0.109542366 container create ac21064d7e14df08b7a48a1791ce380ea06ec39fb586131c2b056ccb7fe24f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T21:47:15.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 podman[30950]: 2024-09-15 21:47:15.291985465 +0000 UTC m=+0.028426676 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:47:15.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 podman[30950]: 2024-09-15 21:47:15.583802785 +0000 UTC m=+0.320243999 container init ac21064d7e14df08b7a48a1791ce380ea06ec39fb586131c2b056ccb7fe24f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 podman[30950]: 2024-09-15 21:47:15.586981339 +0000 UTC m=+0.323422551 container start ac21064d7e14df08b7a48a1791ce380ea06ec39fb586131c2b056ccb7fe24f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: set uid:gid to 167:167 (ceph:ceph) 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: pidfile_write: ignore empty --pid-file 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: load: jerasure load: lrc 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: RocksDB version: 7.9.2 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Git sha 0 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-15T21:47:15.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: DB SUMMARY 2024-09-15T21:47:15.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: DB Session ID: 8ZSB8XCSALPB0TYMWWZK 2024-09-15T21:47:15.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: CURRENT file: CURRENT 2024-09-15T21:47:15.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: IDENTITY file: IDENTITY 2024-09-15T21:47:15.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-15T21:47:15.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-15T21:47:15.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105716 ; 2024-09-15T21:47:15.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.error_if_exists: 0 2024-09-15T21:47:15.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.create_if_missing: 0 2024-09-15T21:47:15.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.paranoid_checks: 1 2024-09-15T21:47:15.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-15T21:47:15.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-15T21:47:15.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-15T21:47:15.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.env: 0x56508805a940 2024-09-15T21:47:15.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.fs: PosixFileSystem 2024-09-15T21:47:15.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.info_log: 0x56508a4816e0 2024-09-15T21:47:15.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_file_opening_threads: 16 2024-09-15T21:47:15.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.statistics: (nil) 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.use_fsync: 0 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_log_file_size: 0 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.keep_log_file_num: 1000 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.recycle_log_file_num: 0 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.allow_fallocate: 1 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.allow_mmap_reads: 0 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.allow_mmap_writes: 0 2024-09-15T21:47:15.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.use_direct_reads: 0 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.create_missing_column_families: 0 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.db_log_dir: 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.wal_dir: 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-15T21:47:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.advise_random_on_open: 1 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.db_write_buffer_size: 0 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.write_buffer_manager: 0x56508a490dc0 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.rate_limiter: (nil) 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.wal_recovery_mode: 2 2024-09-15T21:47:15.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.enable_thread_tracking: 0 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.enable_pipelined_write: 0 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.unordered_write: 0 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.row_cache: None 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.wal_filter: None 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.allow_ingest_behind: 0 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.two_write_queues: 0 2024-09-15T21:47:15.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.manual_wal_flush: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.wal_compression: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.atomic_flush: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.log_readahead_size: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.best_efforts_recovery: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.allow_data_in_errors: 0 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.db_host_id: __hostname__ 2024-09-15T21:47:15.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_background_jobs: 2 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_background_compactions: -1 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_subcompactions: 1 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_total_wal_size: 0 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-15T21:47:15.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_open_files: -1 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bytes_per_sync: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_readahead_size: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_background_flushes: -1 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Compression algorithms supported: 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kZSTD supported: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kXpressCompression supported: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kBZip2Compression supported: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kLZ4Compression supported: 1 2024-09-15T21:47:15.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kZlibCompression supported: 1 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kLZ4HCCompression supported: 1 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: kSnappyCompression supported: 1 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.merge_operator: 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_filter: None 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_filter_factory: None 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.sst_partitioner_factory: None 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-15T21:47:15.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56508a481320) 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-09-15T21:47:15.678 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x56508a4751f0 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-09-15T21:47:15.679 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-09-15T21:47:15.680 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.write_buffer_size: 33554432 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_write_buffer_number: 2 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression: NoCompression 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression: Disabled 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.prefix_extractor: nullptr 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.num_levels: 7 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-15T21:47:15.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.level: 32767 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.strategy: 0 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-15T21:47:15.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.enabled: false 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.target_file_size_base: 67108864 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-15T21:47:15.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.arena_block_size: 1048576 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-15T21:47:15.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.disable_auto_compactions: 0 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-15T21:47:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.inplace_update_support: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.bloom_locality: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.max_successive_merges: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.paranoid_file_checks: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.force_consistency_checks: 1 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.report_bg_io_stats: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.ttl: 2592000 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-15T21:47:15.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.enable_blob_files: false 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.min_blob_size: 0 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.blob_file_size: 268435456 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.blob_file_starting_level: 0 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-15T21:47:15.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d15277bb-54b2-496a-acef-e03c893cd313 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726436835614364, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726436835615382, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100716, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 98610, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14417, "raw_average_key_size": 49, "raw_value_size": 90824, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726436835, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d15277bb-54b2-496a-acef-e03c893cd313", "db_session_id": "8ZSB8XCSALPB0TYMWWZK", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726436835615483, "job": 1, "event": "recovery_finished"} 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56508a4a2e00 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: DB pointer 0x56508a59e000 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-15T21:47:15.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-15T21:47:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 100.22 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 100.22 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-15T21:47:15.690 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 29.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 29.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x56508a4751f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-15T21:47:15.691 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???) e1 preinit fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???).mds e1 new map 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???).mds e1 print_map 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout: btime 2024-09-15T21:47:10:448594+0000 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-09-15T21:47:15.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-15T21:47:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T21:47:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T21:47:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T21:47:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-15T21:47:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 bash[30950]: ac21064d7e14df08b7a48a1791ce380ea06ec39fb586131c2b056ccb7fe24f4e 2024-09-15T21:47:15.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 systemd[1]: Started Ceph mon.a for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:47:15.693 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-15T21:47:15.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: monmap epoch 1 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: last_changed 2024-09-15T21:47:06.538730+0000 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: min_mon_release 19 (squid) 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: election_strategy: 1 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: fsmap 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: osdmap e1: 0 total, 0 up, 0 in 2024-09-15T21:47:15.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:15 smithi028 ceph-mon[30969]: mgrmap e1: no daemons active 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.199+0000 7f9820188640 1 Processor -- start 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.200+0000 7f9820188640 1 -- start start 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.200+0000 7f9820188640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f9818107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.200+0000 7f9820188640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98181080d0 con 0x7f9818107700 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.200+0000 7f981df24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f9818107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.200+0000 7f981df24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f9818107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32852/0 (socket says 172.21.15.28:32852) 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.200+0000 7f981df24640 1 -- 172.21.15.28:0/2733253094 learned_addr learned my addr 172.21.15.28:0/2733253094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.201+0000 7f981df24640 1 -- 172.21.15.28:0/2733253094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9818108900 con 0x7f9818107700 2024-09-15T21:47:16.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.202+0000 7f981df24640 1 --2- 172.21.15.28:0/2733253094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f9818107b00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9814009820 tx=0x7f981402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9bc42b6469a928 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f981cf22640 1 -- 172.21.15.28:0/2733253094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9814037070 con 0x7f9818107700 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f981cf22640 1 -- 172.21.15.28:0/2733253094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9814032900 con 0x7f9818107700 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f981cf22640 1 -- 172.21.15.28:0/2733253094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9814030850 con 0x7f9818107700 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f9820188640 1 -- 172.21.15.28:0/2733253094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 msgr2=0x7f9818107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f9820188640 1 --2- 172.21.15.28:0/2733253094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f9818107b00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9814009820 tx=0x7f981402a270 comp rx=0 tx=0).stop 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f9820188640 1 -- 172.21.15.28:0/2733253094 shutdown_connections 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f9820188640 1 --2- 172.21.15.28:0/2733253094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f9818107b00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.203+0000 7f9820188640 1 -- 172.21.15.28:0/2733253094 >> 172.21.15.28:0/2733253094 conn(0x7f9818102ed0 msgr2=0x7f98181052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f9820188640 1 -- 172.21.15.28:0/2733253094 shutdown_connections 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f9820188640 1 -- 172.21.15.28:0/2733253094 wait complete. 2024-09-15T21:47:16.849 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f9820188640 1 Processor -- start 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f9820188640 1 -- start start 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f9820188640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f981819b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f9820188640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f981810bcb0 con 0x7f9818107700 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f981df24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f981819b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.204+0000 7f981df24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f981819b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32854/0 (socket says 172.21.15.28:32854) 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f981df24640 1 -- 172.21.15.28:0/999923103 learned_addr learned my addr 172.21.15.28:0/999923103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f981df24640 1 -- 172.21.15.28:0/999923103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981819bc60 con 0x7f9818107700 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f981df24640 1 --2- 172.21.15.28:0/999923103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f981819b720 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9814009940 tx=0x7f9814032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f980effd640 1 -- 172.21.15.28:0/999923103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9814042070 con 0x7f9818107700 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f980effd640 1 -- 172.21.15.28:0/999923103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f98140308c0 con 0x7f9818107700 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981819bef0 con 0x7f9818107700 2024-09-15T21:47:16.850 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f980effd640 1 -- 172.21.15.28:0/999923103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9814037040 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f981819ebe0 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.205+0000 7f980effd640 1 -- 172.21.15.28:0/999923103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9814045070 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.206+0000 7f980effd640 1 -- 172.21.15.28:0/999923103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9814047b70 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.206+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f981810be30 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.209+0000 7f980effd640 1 -- 172.21.15.28:0/999923103 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f98140327b0 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.268+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f981819f1d0 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.270+0000 7f980effd640 1 -- 172.21.15.28:0/999923103 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f981403c760 con 0x7f9818107700 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.271+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 msgr2=0x7f981819b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.271+0000 7f9820188640 1 --2- 172.21.15.28:0/999923103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f981819b720 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9814009940 tx=0x7f9814032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.271+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 shutdown_connections 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.271+0000 7f9820188640 1 --2- 172.21.15.28:0/999923103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818107700 0x7f981819b720 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:16.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.271+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 >> 172.21.15.28:0/999923103 conn(0x7f9818102ed0 msgr2=0x7f9818103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:16.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.271+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 shutdown_connections 2024-09-15T21:47:16.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:16.271+0000 7f9820188640 1 -- 172.21.15.28:0/999923103 wait complete. 2024-09-15T21:47:16.852 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-15T21:47:16.874 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-15T21:47:16.874 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-09-15T21:47:16.874 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-15T21:47:16.875 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-15T21:47:17.136 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.x 2024-09-15T21:47:17.136 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.x.service: Unit ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.x.service not loaded. 2024-09-15T21:47:17.291 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e78b7764-73ab-11ef-bceb-c7b262605968.target.wants/ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-e78b7764-73ab-11ef-bceb-c7b262605968@.service. 2024-09-15T21:47:17.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:17 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/999923103' entity='client.admin' 2024-09-15T21:47:17.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 systemd[1]: Starting Ceph mgr.x for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:47:17.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 podman[31183]: 2024-09-15 21:47:17.525743041 +0000 UTC m=+0.104245116 container create c43e87b526bf4acc75aa920549c4f22375d342913565265d1f35a09f2a907315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , ceph=True) 2024-09-15T21:47:17.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 podman[31183]: 2024-09-15 21:47:17.437702 +0000 UTC m=+0.016204084 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T21:47:17.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 podman[31183]: 2024-09-15 21:47:17.727974397 +0000 UTC m=+0.306476474 container init c43e87b526bf4acc75aa920549c4f22375d342913565265d1f35a09f2a907315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731) 2024-09-15T21:47:17.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 podman[31183]: 2024-09-15 21:47:17.730764416 +0000 UTC m=+0.309266491 container start c43e87b526bf4acc75aa920549c4f22375d342913565265d1f35a09f2a907315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-8921c8e) 2024-09-15T21:47:17.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 bash[31183]: c43e87b526bf4acc75aa920549c4f22375d342913565265d1f35a09f2a907315 2024-09-15T21:47:17.802 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T21:47:17.803 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-15T21:47:17.810 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T21:47:17.810 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-15T21:47:17.810 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-15T21:47:17.810 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-09-15T21:47:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T21:47:17.820 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-15T21:47:17.827 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T21:47:17.828 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-15T21:47:17.828 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-15T21:47:17.828 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-15T21:47:17.828 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-09-15T21:47:17.828 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-09-15T21:47:18.080 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 systemd[1]: Started Ceph mgr.x for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:47:18.080 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:17.861+0000 7f4a8c99e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:47:18.080 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:17 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:17.920+0000 7f4a8c99e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:47:18.081 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:18.077+0000 7f4a8c99e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:47:18.335 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:18.147+0000 7f4a8c99e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:47:18.335 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:18.305+0000 7f4a8c99e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:47:18.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:18 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2624247472' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T21:47:18.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:18.456+0000 7f4a8c99e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:47:18.943 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:18.787+0000 7f4a8c99e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:47:18.944 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:18.850+0000 7f4a8c99e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:47:18.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:18.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-15T21:47:18.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:47:18.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T21:47:18.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:18.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T21:47:18.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T21:47:18.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T21:47:10:448594+0000", 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-15T21:47:18.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T21:47:10.449233+0000", 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T21:47:18.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.251+0000 7f35ad314640 1 Processor -- start 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.251+0000 7f35ad314640 1 -- start start 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.251+0000 7f35ad314640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a8106170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.251+0000 7f35ad314640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35a8106740 con 0x7f35a8105d70 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.252+0000 7f35a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a8106170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.252+0000 7f35a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a8106170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32886/0 (socket says 172.21.15.28:32886) 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.252+0000 7f35a6ffd640 1 -- 172.21.15.28:0/1810461804 learned_addr learned my addr 172.21.15.28:0/1810461804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:18.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.252+0000 7f35a6ffd640 1 -- 172.21.15.28:0/1810461804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a8106f70 con 0x7f35a8105d70 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.253+0000 7f35a6ffd640 1 --2- 172.21.15.28:0/1810461804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a8106170 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3598009820 tx=0x7f359802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67b5e3140e08f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.253+0000 7f35a5ffb640 1 -- 172.21.15.28:0/1810461804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598037070 con 0x7f35a8105d70 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.253+0000 7f35a5ffb640 1 -- 172.21.15.28:0/1810461804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3598032900 con 0x7f35a8105d70 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.253+0000 7f35a5ffb640 1 -- 172.21.15.28:0/1810461804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598030850 con 0x7f35a8105d70 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.254+0000 7f35ad314640 1 -- 172.21.15.28:0/1810461804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 msgr2=0x7f35a8106170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.254+0000 7f35ad314640 1 --2- 172.21.15.28:0/1810461804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a8106170 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3598009820 tx=0x7f359802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.254+0000 7f35ad314640 1 -- 172.21.15.28:0/1810461804 shutdown_connections 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.254+0000 7f35ad314640 1 --2- 172.21.15.28:0/1810461804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a8106170 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:18.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.254+0000 7f35ad314640 1 -- 172.21.15.28:0/1810461804 >> 172.21.15.28:0/1810461804 conn(0x7f35a8101520 msgr2=0x7f35a8103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.254+0000 7f35ad314640 1 -- 172.21.15.28:0/1810461804 shutdown_connections 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.254+0000 7f35ad314640 1 -- 172.21.15.28:0/1810461804 wait complete. 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.255+0000 7f35ad314640 1 Processor -- start 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.255+0000 7f35ad314640 1 -- start start 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.255+0000 7f35ad314640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.255+0000 7f35ad314640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35a8107750 con 0x7f35a8105d70 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.255+0000 7f35a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.255+0000 7f35a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32898/0 (socket says 172.21.15.28:32898) 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.255+0000 7f35a6ffd640 1 -- 172.21.15.28:0/2624247472 learned_addr learned my addr 172.21.15.28:0/2624247472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.256+0000 7f35a6ffd640 1 -- 172.21.15.28:0/2624247472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a819de40 con 0x7f35a8105d70 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.256+0000 7f35a6ffd640 1 --2- 172.21.15.28:0/2624247472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a819d900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3598009940 tx=0x7f3598032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:18.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.256+0000 7f359ffff640 1 -- 172.21.15.28:0/2624247472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598042070 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.256+0000 7f359ffff640 1 -- 172.21.15.28:0/2624247472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f35980308c0 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.256+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a819e0d0 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.256+0000 7f359ffff640 1 -- 172.21.15.28:0/2624247472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598037040 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.256+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a81a0dc0 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.257+0000 7f359ffff640 1 -- 172.21.15.28:0/2624247472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3598045070 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.257+0000 7f359ffff640 1 -- 172.21.15.28:0/2624247472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3598047bc0 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.257+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a8106170 con 0x7f35a8105d70 2024-09-15T21:47:18.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.259+0000 7f359ffff640 1 -- 172.21.15.28:0/2624247472 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f35980327b0 con 0x7f35a8105d70 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.322+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f35a81a1330 con 0x7f35a8105d70 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.322+0000 7f359ffff640 1 -- 172.21.15.28:0/2624247472 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f359803c870 con 0x7f35a8105d70 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.323+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 msgr2=0x7f35a819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.323+0000 7f35ad314640 1 --2- 172.21.15.28:0/2624247472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a819d900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3598009940 tx=0x7f3598032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.323+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 shutdown_connections 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.323+0000 7f35ad314640 1 --2- 172.21.15.28:0/2624247472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a8105d70 0x7f35a819d900 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.323+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 >> 172.21.15.28:0/2624247472 conn(0x7f35a8101520 msgr2=0x7f35a8101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.323+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 shutdown_connections 2024-09-15T21:47:18.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:18.323+0000 7f35ad314640 1 -- 172.21.15.28:0/2624247472 wait complete. 2024-09-15T21:47:18.969 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-09-15T21:47:19.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:19 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:19.045+0000 7f4a8c99e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:47:20.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:19 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:19.965+0000 7f4a8c99e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:47:20.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:20.035+0000 7f4a8c99e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:47:21.066 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:20.787+0000 7f4a8c99e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:47:21.066 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:20.907+0000 7f4a8c99e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:47:21.066 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.064+0000 7f4a8c99e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:47:21.360 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.128+0000 7f4a8c99e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:47:21.360 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.286+0000 7f4a8c99e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:47:21.361 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.358+0000 7f4a8c99e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:47:21.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/877950113' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T21:47:21.662 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.423+0000 7f4a8c99e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:47:21.662 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.593+0000 7f4a8c99e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:47:21.662 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.660+0000 7f4a8c99e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:47:21.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.730+0000 7f4a8c99e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:47:21.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.817+0000 7f4a8c99e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:47:21.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.879+0000 7f4a8c99e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:47:22.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:22.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-15T21:47:22.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:47:22.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:22.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T21:47:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T21:47:10:448594+0000", 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T21:47:22.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T21:47:10.449233+0000", 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.382+0000 7fefdc280640 1 Processor -- start 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.383+0000 7fefdc280640 1 -- start start 2024-09-15T21:47:22.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.383+0000 7fefdc280640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd41069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.383+0000 7fefdc280640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4106f90 con 0x7fefd41065c0 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.383+0000 7fefda01c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd41069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.383+0000 7fefda01c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd41069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46696/0 (socket says 172.21.15.28:46696) 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.383+0000 7fefda01c640 1 -- 172.21.15.28:0/3467588783 learned_addr learned my addr 172.21.15.28:0/3467588783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.383+0000 7fefda01c640 1 -- 172.21.15.28:0/3467588783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd41077c0 con 0x7fefd41065c0 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.384+0000 7fefda01c640 1 --2- 172.21.15.28:0/3467588783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd41069c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009820 tx=0x7fefd002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608d75a14c170ed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.385+0000 7fefd901a640 1 -- 172.21.15.28:0/3467588783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefd0037070 con 0x7fefd41065c0 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.385+0000 7fefd901a640 1 -- 172.21.15.28:0/3467588783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fefd0032900 con 0x7fefd41065c0 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.385+0000 7fefd901a640 1 -- 172.21.15.28:0/3467588783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefd0030850 con 0x7fefd41065c0 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.385+0000 7fefdc280640 1 -- 172.21.15.28:0/3467588783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 msgr2=0x7fefd41069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.385+0000 7fefdc280640 1 --2- 172.21.15.28:0/3467588783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd41069c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009820 tx=0x7fefd002a270 comp rx=0 tx=0).stop 2024-09-15T21:47:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 -- 172.21.15.28:0/3467588783 shutdown_connections 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 --2- 172.21.15.28:0/3467588783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd41069c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 -- 172.21.15.28:0/3467588783 >> 172.21.15.28:0/3467588783 conn(0x7fefd4101d30 msgr2=0x7fefd4104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 -- 172.21.15.28:0/3467588783 shutdown_connections 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 -- 172.21.15.28:0/3467588783 wait complete. 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 Processor -- start 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 -- start start 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd419b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefdc280640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefd410ab70 con 0x7fefd41065c0 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefda01c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd419b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefda01c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd419b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46708/0 (socket says 172.21.15.28:46708) 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.386+0000 7fefda01c640 1 -- 172.21.15.28:0/877950113 learned_addr learned my addr 172.21.15.28:0/877950113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefda01c640 1 -- 172.21.15.28:0/877950113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd419bc60 con 0x7fefd41065c0 2024-09-15T21:47:22.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefda01c640 1 --2- 172.21.15.28:0/877950113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd419b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009940 tx=0x7fefd0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefcb7fe640 1 -- 172.21.15.28:0/877950113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefd0042070 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefcb7fe640 1 -- 172.21.15.28:0/877950113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fefd00308c0 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd419bef0 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefcb7fe640 1 -- 172.21.15.28:0/877950113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefd0037040 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd419ebe0 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefcb7fe640 1 -- 172.21.15.28:0/877950113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fefd0045070 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.387+0000 7fefcb7fe640 1 -- 172.21.15.28:0/877950113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fefd00463b0 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.388+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd410acf0 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.390+0000 7fefcb7fe640 1 -- 172.21.15.28:0/877950113 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fefd00327b0 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.453+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fefd402d950 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.454+0000 7fefcb7fe640 1 -- 172.21.15.28:0/877950113 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7fefd0047440 con 0x7fefd41065c0 2024-09-15T21:47:22.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.455+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 msgr2=0x7fefd419b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.455+0000 7fefdc280640 1 --2- 172.21.15.28:0/877950113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd419b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009940 tx=0x7fefd0032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.455+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 shutdown_connections 2024-09-15T21:47:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.455+0000 7fefdc280640 1 --2- 172.21.15.28:0/877950113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefd41065c0 0x7fefd419b720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.455+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 >> 172.21.15.28:0/877950113 conn(0x7fefd4101d30 msgr2=0x7fefd4102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.455+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 shutdown_connections 2024-09-15T21:47:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:21.455+0000 7fefdc280640 1 -- 172.21.15.28:0/877950113 wait complete. 2024-09-15T21:47:22.076 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-09-15T21:47:22.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:21.940+0000 7f4a8c99e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:47:23.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:22.770+0000 7f4a8c99e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:47:23.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: Activating manager daemon x 2024-09-15T21:47:23.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: mgrmap e2: x(active, starting, since 0.00299575s) 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: Manager daemon x is now available 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' 2024-09-15T21:47:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:22 smithi028 ceph-mon[30969]: from='mgr.14100 172.21.15.28:0/3970250052' entity='mgr.x' 2024-09-15T21:47:25.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:24 smithi028 ceph-mon[30969]: mgrmap e3: x(active, since 1.00777s) 2024-09-15T21:47:25.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:24 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/741451699' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:25.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T21:47:25.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:25.320 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T21:47:25.320 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T21:47:25.320 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T21:47:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T21:47:10:448594+0000", 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T21:47:25.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T21:47:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T21:47:10.449233+0000", 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.490+0000 7fc38dace640 1 Processor -- start 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.490+0000 7fc38dace640 1 -- start start 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.490+0000 7fc38dace640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38810a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.490+0000 7fc38dace640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc38807bed0 con 0x7fc388108130 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.490+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38810a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.490+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38810a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46782/0 (socket says 172.21.15.28:46782) 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.490+0000 7fc3877fe640 1 -- 172.21.15.28:0/3234400325 learned_addr learned my addr 172.21.15.28:0/3234400325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.491+0000 7fc3877fe640 1 -- 172.21.15.28:0/3234400325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38810aa60 con 0x7fc388108130 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.491+0000 7fc3877fe640 1 --2- 172.21.15.28:0/3234400325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38810a520 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc378009820 tx=0x7fc37802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f39a597467bac6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.491+0000 7fc386ffd640 1 -- 172.21.15.28:0/3234400325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc378037070 con 0x7fc388108130 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc386ffd640 1 -- 172.21.15.28:0/3234400325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc378032900 con 0x7fc388108130 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc386ffd640 1 -- 172.21.15.28:0/3234400325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc378030850 con 0x7fc388108130 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc38dace640 1 -- 172.21.15.28:0/3234400325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 msgr2=0x7fc38810a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc38dace640 1 --2- 172.21.15.28:0/3234400325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38810a520 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc378009820 tx=0x7fc37802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc38dace640 1 -- 172.21.15.28:0/3234400325 shutdown_connections 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc38dace640 1 --2- 172.21.15.28:0/3234400325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38810a520 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc38dace640 1 -- 172.21.15.28:0/3234400325 >> 172.21.15.28:0/3234400325 conn(0x7fc388101d30 msgr2=0x7fc388104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc38dace640 1 -- 172.21.15.28:0/3234400325 shutdown_connections 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.492+0000 7fc38dace640 1 -- 172.21.15.28:0/3234400325 wait complete. 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.493+0000 7fc38dace640 1 Processor -- start 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.493+0000 7fc38dace640 1 -- start start 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.493+0000 7fc38dace640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38819b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.493+0000 7fc38dace640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc38810bb00 con 0x7fc388108130 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.493+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38819b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.493+0000 7fc3877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38819b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46790/0 (socket says 172.21.15.28:46790) 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.493+0000 7fc3877fe640 1 -- 172.21.15.28:0/741451699 learned_addr learned my addr 172.21.15.28:0/741451699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.494+0000 7fc3877fe640 1 -- 172.21.15.28:0/741451699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38819bc00 con 0x7fc388108130 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.494+0000 7fc3877fe640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38819b6c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3780097f0 tx=0x7fc378030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.494+0000 7fc3857fa640 1 -- 172.21.15.28:0/741451699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc378037040 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.494+0000 7fc3857fa640 1 -- 172.21.15.28:0/741451699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc378040070 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.494+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38819be90 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.494+0000 7fc3857fa640 1 -- 172.21.15.28:0/741451699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc37803a390 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.494+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc38819eb80 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.495+0000 7fc3857fa640 1 -- 172.21.15.28:0/741451699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 3) ==== 46344+0+0 (secure 0 0 0) 0x7fc37803a630 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.495+0000 7fc3857fa640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fc374039050 0x7fc37403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.495+0000 7fc3857fa640 1 -- 172.21.15.28:0/741451699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc37806c1e0 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.495+0000 7fc37ffff640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fc374039050 0x7fc37403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.495+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc388108590 con 0x7fc388108130 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.496+0000 7fc37ffff640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fc374039050 0x7fc37403b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc3700098c0 tx=0x7fc370006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:25.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.501+0000 7fc3857fa640 1 -- 172.21.15.28:0/741451699 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc37803a920 con 0x7fc388108130 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.746+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc38819f120 con 0x7fc388108130 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.746+0000 7fc3857fa640 1 -- 172.21.15.28:0/741451699 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7fc378060070 con 0x7fc388108130 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fc374039050 msgr2=0x7fc37403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fc374039050 0x7fc37403b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc3700098c0 tx=0x7fc370006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 msgr2=0x7fc38819b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38819b6c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3780097f0 tx=0x7fc378030d80 comp rx=0 tx=0).stop 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 shutdown_connections 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fc374039050 0x7fc37403b510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 --2- 172.21.15.28:0/741451699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc388108130 0x7fc38819b6c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.749+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 >> 172.21.15.28:0/741451699 conn(0x7fc388101d30 msgr2=0x7fc388102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.750+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 shutdown_connections 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:24.750+0000 7fc38dace640 1 -- 172.21.15.28:0/741451699 wait complete. 2024-09-15T21:47:25.335 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-09-15T21:47:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:25 smithi028 ceph-mon[30969]: mgrmap e4: x(active, since 2s) 2024-09-15T21:47:26.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:26.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-09-15T21:47:26.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:26.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-09-15T21:47:26.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-15T21:47:26.429 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.705+0000 7fec9dedb640 1 Processor -- start 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.705+0000 7fec9dedb640 1 -- start start 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.705+0000 7fec9dedb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec981069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.705+0000 7fec9dedb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec98106f90 con 0x7fec981065c0 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.706+0000 7fec977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec981069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.706+0000 7fec977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec981069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46794/0 (socket says 172.21.15.28:46794) 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.706+0000 7fec977fe640 1 -- 172.21.15.28:0/2555821972 learned_addr learned my addr 172.21.15.28:0/2555821972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.706+0000 7fec977fe640 1 -- 172.21.15.28:0/2555821972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec981077c0 con 0x7fec981065c0 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec977fe640 1 --2- 172.21.15.28:0/2555821972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec981069c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fec88009820 tx=0x7fec8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468f511ebbf15d4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec967fc640 1 -- 172.21.15.28:0/2555821972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec88037070 con 0x7fec981065c0 2024-09-15T21:47:26.430 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec967fc640 1 -- 172.21.15.28:0/2555821972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fec88032900 con 0x7fec981065c0 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec967fc640 1 -- 172.21.15.28:0/2555821972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec88030850 con 0x7fec981065c0 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec9dedb640 1 -- 172.21.15.28:0/2555821972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 msgr2=0x7fec981069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec9dedb640 1 --2- 172.21.15.28:0/2555821972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec981069c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fec88009820 tx=0x7fec8802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec9dedb640 1 -- 172.21.15.28:0/2555821972 shutdown_connections 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec9dedb640 1 --2- 172.21.15.28:0/2555821972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec981069c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.707+0000 7fec9dedb640 1 -- 172.21.15.28:0/2555821972 >> 172.21.15.28:0/2555821972 conn(0x7fec98101d30 msgr2=0x7fec98104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.708+0000 7fec9dedb640 1 -- 172.21.15.28:0/2555821972 shutdown_connections 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.708+0000 7fec9dedb640 1 -- 172.21.15.28:0/2555821972 wait complete. 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.708+0000 7fec9dedb640 1 Processor -- start 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.708+0000 7fec9dedb640 1 -- start start 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.708+0000 7fec9dedb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec9807ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:26.431 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.708+0000 7fec9dedb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec9810ab70 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec9807ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec9807ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46800/0 (socket says 172.21.15.28:46800) 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec977fe640 1 -- 172.21.15.28:0/150650213 learned_addr learned my addr 172.21.15.28:0/150650213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec977fe640 1 -- 172.21.15.28:0/150650213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9819c2a0 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec977fe640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec9807ba70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fec88009940 tx=0x7fec88032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec94ff9640 1 -- 172.21.15.28:0/150650213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec88042070 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec94ff9640 1 -- 172.21.15.28:0/150650213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fec880308c0 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec981a08b0 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec94ff9640 1 -- 172.21.15.28:0/150650213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec88037040 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.709+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec981a0c50 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.710+0000 7fec94ff9640 1 -- 172.21.15.28:0/150650213 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fec88045070 con 0x7fec981065c0 2024-09-15T21:47:26.432 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.710+0000 7fec94ff9640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fec800391b0 0x7fec8003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.710+0000 7fec94ff9640 1 -- 172.21.15.28:0/150650213 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fec8806c3e0 con 0x7fec981065c0 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.711+0000 7fec96ffd640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fec800391b0 0x7fec8003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.711+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec9810acf0 con 0x7fec981065c0 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.711+0000 7fec96ffd640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fec800391b0 0x7fec8003b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0098c0 tx=0x7fec8c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.716+0000 7fec94ff9640 1 -- 172.21.15.28:0/150650213 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fec8802ed90 con 0x7fec981065c0 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.892+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fec981a0ed0 con 0x7fec981065c0 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.893+0000 7fec94ff9640 1 -- 172.21.15.28:0/150650213 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+454 (secure 0 0 0) 0x7fec981a0ed0 con 0x7fec981065c0 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.895+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fec800391b0 msgr2=0x7fec8003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.895+0000 7fec9dedb640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fec800391b0 0x7fec8003b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0098c0 tx=0x7fec8c006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.895+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 msgr2=0x7fec9807ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.895+0000 7fec9dedb640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec9807ba70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fec88009940 tx=0x7fec88032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:26.433 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.896+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 shutdown_connections 2024-09-15T21:47:26.434 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.896+0000 7fec9dedb640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fec800391b0 0x7fec8003b670 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:26.434 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.896+0000 7fec9dedb640 1 --2- 172.21.15.28:0/150650213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec981065c0 0x7fec9807ba70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:26.434 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.896+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 >> 172.21.15.28:0/150650213 conn(0x7fec98101d30 msgr2=0x7fec98102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:26.434 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.896+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 shutdown_connections 2024-09-15T21:47:26.434 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:25.896+0000 7fec9dedb640 1 -- 172.21.15.28:0/150650213 wait complete. 2024-09-15T21:47:26.434 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-09-15T21:47:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:26 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/150650213' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-15T21:47:28.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:27 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4212427379' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-15T21:47:28.100 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:27 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ignoring --setuser ceph since I am not root 2024-09-15T21:47:28.100 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:27 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ignoring --setgroup ceph since I am not root 2024-09-15T21:47:28.101 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:27 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:27.931+0000 7f4672dce040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:47:28.101 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:27 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:27.990+0000 7f4672dce040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:47:28.350 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:28.145+0000 7f4672dce040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:47:28.351 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:28.214+0000 7f4672dce040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:47:28.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.852+0000 7fd8ede6a640 1 Processor -- start 2024-09-15T21:47:28.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.852+0000 7fd8ede6a640 1 -- start start 2024-09-15T21:47:28.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.852+0000 7fd8ede6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e8107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:28.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.852+0000 7fd8ede6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e81080d0 con 0x7fd8e8107700 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.852+0000 7fd8e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e8107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.852+0000 7fd8e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e8107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46802/0 (socket says 172.21.15.28:46802) 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.852+0000 7fd8e77fe640 1 -- 172.21.15.28:0/4212200249 learned_addr learned my addr 172.21.15.28:0/4212200249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.853+0000 7fd8e77fe640 1 -- 172.21.15.28:0/4212200249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e8108900 con 0x7fd8e8107700 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.853+0000 7fd8e77fe640 1 --2- 172.21.15.28:0/4212200249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e8107b00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009820 tx=0x7fd8d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c24bcba2f467d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8e67fc640 1 -- 172.21.15.28:0/4212200249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8d8037070 con 0x7fd8e8107700 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8e67fc640 1 -- 172.21.15.28:0/4212200249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd8d8032900 con 0x7fd8e8107700 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8e67fc640 1 -- 172.21.15.28:0/4212200249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8d8030850 con 0x7fd8e8107700 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212200249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 msgr2=0x7fd8e8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8ede6a640 1 --2- 172.21.15.28:0/4212200249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e8107b00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009820 tx=0x7fd8d802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212200249 shutdown_connections 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8ede6a640 1 --2- 172.21.15.28:0/4212200249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e8107b00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212200249 >> 172.21.15.28:0/4212200249 conn(0x7fd8e8102ed0 msgr2=0x7fd8e81052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212200249 shutdown_connections 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.854+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212200249 wait complete. 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.855+0000 7fd8ede6a640 1 Processor -- start 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.855+0000 7fd8ede6a640 1 -- start start 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.855+0000 7fd8ede6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e819b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.855+0000 7fd8ede6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e810bcb0 con 0x7fd8e8107700 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.855+0000 7fd8e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e819b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.855+0000 7fd8e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e819b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46818/0 (socket says 172.21.15.28:46818) 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.855+0000 7fd8e77fe640 1 -- 172.21.15.28:0/4212427379 learned_addr learned my addr 172.21.15.28:0/4212427379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.856+0000 7fd8e77fe640 1 -- 172.21.15.28:0/4212427379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e819bc60 con 0x7fd8e8107700 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.856+0000 7fd8e77fe640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e819b720 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009940 tx=0x7fd8d8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:28.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.856+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8d8042070 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.856+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd8d80308c0 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.856+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8d8037040 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.856+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e819bef0 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.856+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e819ebe0 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.857+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fd8d8045070 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.857+0000 7fd8e4ff9640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fd8d0039160 0x7fd8d003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.857+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e810be30 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.857+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd8d806c1e0 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.857+0000 7fd8e6ffd640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fd8d0039160 0x7fd8d003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.858+0000 7fd8e6ffd640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fd8d0039160 0x7fd8d003b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0098c0 tx=0x7fd8dc006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:26.862+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd8d803cd90 con 0x7fd8e8107700 2024-09-15T21:47:28.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.086+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fd8e819edb0 con 0x7fd8e8107700 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.782+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fd8d802fd90 con 0x7fd8e8107700 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.784+0000 7fd8e4ff9640 1 -- 172.21.15.28:0/4212427379 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7fd8d80327b0 con 0x7fd8e8107700 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fd8d0039160 msgr2=0x7fd8d003b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fd8d0039160 0x7fd8d003b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc0098c0 tx=0x7fd8dc006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 msgr2=0x7fd8e819b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e819b720 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009940 tx=0x7fd8d8032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 shutdown_connections 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7fd8d0039160 0x7fd8d003b620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 --2- 172.21.15.28:0/4212427379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8e8107700 0x7fd8e819b720 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 >> 172.21.15.28:0/4212427379 conn(0x7fd8e8102ed0 msgr2=0x7fd8e8103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 shutdown_connections 2024-09-15T21:47:28.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:27.786+0000 7fd8ede6a640 1 -- 172.21.15.28:0/4212427379 wait complete. 2024-09-15T21:47:28.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:28.371+0000 7f4672dce040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:47:28.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:28.524+0000 7f4672dce040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:47:28.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4212427379' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-15T21:47:28.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-mon[30969]: mgrmap e5: x(active, since 5s) 2024-09-15T21:47:28.977 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:28.859+0000 7f4672dce040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:47:28.977 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:28 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:28.922+0000 7f4672dce040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:47:29.348 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:29 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:29.118+0000 7f4672dce040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:47:29.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-15T21:47:29.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-15T21:47:29.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T21:47:29.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-15T21:47:29.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-15T21:47:29.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-15T21:47:29.582 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f6202411640 1 Processor -- start 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f6202411640 1 -- start start 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f6202411640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc10a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f6202411640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc07bed0 con 0x7f61fc1081a0 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f61fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc10a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f61fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc10a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46844/0 (socket says 172.21.15.28:46844) 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f61fbfff640 1 -- 172.21.15.28:0/514277984 learned_addr learned my addr 172.21.15.28:0/514277984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.769+0000 7f61fbfff640 1 -- 172.21.15.28:0/514277984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc10aad0 con 0x7f61fc1081a0 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.770+0000 7f61fbfff640 1 --2- 172.21.15.28:0/514277984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc10a590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009820 tx=0x7f61ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e52a8f1105ad841 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.770+0000 7f61faffd640 1 -- 172.21.15.28:0/514277984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec037070 con 0x7f61fc1081a0 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.770+0000 7f61faffd640 1 -- 172.21.15.28:0/514277984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f61ec032900 con 0x7f61fc1081a0 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.770+0000 7f61faffd640 1 -- 172.21.15.28:0/514277984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec030850 con 0x7f61fc1081a0 2024-09-15T21:47:29.583 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.771+0000 7f6202411640 1 -- 172.21.15.28:0/514277984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 msgr2=0x7f61fc10a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.771+0000 7f6202411640 1 --2- 172.21.15.28:0/514277984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc10a590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009820 tx=0x7f61ec02a270 comp rx=0 tx=0).stop 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.771+0000 7f6202411640 1 -- 172.21.15.28:0/514277984 shutdown_connections 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.771+0000 7f6202411640 1 --2- 172.21.15.28:0/514277984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc10a590 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.771+0000 7f6202411640 1 -- 172.21.15.28:0/514277984 >> 172.21.15.28:0/514277984 conn(0x7f61fc101dc0 msgr2=0x7f61fc104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.771+0000 7f6202411640 1 -- 172.21.15.28:0/514277984 shutdown_connections 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.771+0000 7f6202411640 1 -- 172.21.15.28:0/514277984 wait complete. 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.772+0000 7f6202411640 1 Processor -- start 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.772+0000 7f6202411640 1 -- start start 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.772+0000 7f6202411640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc107a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.772+0000 7f6202411640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10bb70 con 0x7f61fc1081a0 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.772+0000 7f61fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc107a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.772+0000 7f61fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc107a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46846/0 (socket says 172.21.15.28:46846) 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.772+0000 7f61fbfff640 1 -- 172.21.15.28:0/4212156227 learned_addr learned my addr 172.21.15.28:0/4212156227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.773+0000 7f61fbfff640 1 -- 172.21.15.28:0/4212156227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc107f60 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.773+0000 7f61fbfff640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc107a20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f61ec031000 tx=0x7f61ec0326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.773+0000 7f61f97fa640 1 -- 172.21.15.28:0/4212156227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec042070 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.773+0000 7f61f97fa640 1 -- 172.21.15.28:0/4212156227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f61ec032be0 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.773+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc106150 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.773+0000 7f61f97fa640 1 -- 172.21.15.28:0/4212156227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61ec037040 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.773+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc106690 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.774+0000 7f61f97fa640 1 -- 172.21.15.28:0/4212156227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f61ec045070 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.774+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61fc108600 con 0x7f61fc1081a0 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.774+0000 7f61f97fa640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 0x7f61e403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.774+0000 7f61fb7fe640 1 -- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 msgr2=0x7f61e403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3052473616 2024-09-15T21:47:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.774+0000 7f61f97fa640 1 -- 172.21.15.28:0/4212156227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f61ec06c200 con 0x7f61fc1081a0 2024-09-15T21:47:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.774+0000 7f61fb7fe640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 0x7f61e403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:47:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.779+0000 7f61f97fa640 1 -- 172.21.15.28:0/4212156227 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f61ec042210 con 0x7f61fc1081a0 2024-09-15T21:47:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.975+0000 7f61fb7fe640 1 -- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 msgr2=0x7f61e403b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3052473616 2024-09-15T21:47:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:28.975+0000 7f61fb7fe640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 0x7f61e403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T21:47:29.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.007+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f61fc106910 con 0x7f61fc1081a0 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.008+0000 7f61f97fa640 1 -- 172.21.15.28:0/4212156227 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f61ec032d90 con 0x7f61fc1081a0 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 msgr2=0x7f61e403b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 0x7f61e403b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 msgr2=0x7f61fc107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc107a20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f61ec031000 tx=0x7f61ec0326e0 comp rx=0 tx=0).stop 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 shutdown_connections 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f61e4039200 0x7f61e403b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 --2- 172.21.15.28:0/4212156227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61fc1081a0 0x7f61fc107a20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 >> 172.21.15.28:0/4212156227 conn(0x7f61fc101dc0 msgr2=0x7f61fc1029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 shutdown_connections 2024-09-15T21:47:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.010+0000 7f6202411640 1 -- 172.21.15.28:0/4212156227 wait complete. 2024-09-15T21:47:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-09-15T21:47:29.588 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-09-15T21:47:29.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:29 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4212156227' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-15T21:47:30.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:30 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:30.062+0000 7f4672dce040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:47:30.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:30 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:30.132+0000 7f4672dce040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:47:31.148 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:30 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:30.877+0000 7f4672dce040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:47:31.148 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:30 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:30.997+0000 7f4672dce040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:47:31.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.146+0000 7f4672dce040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:47:31.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.209+0000 7f4672dce040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:47:31.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.365+0000 7f4672dce040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:47:31.733 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.436+0000 7f4672dce040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:47:31.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.498+0000 7f4672dce040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:47:31.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.664+0000 7f4672dce040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:47:31.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.731+0000 7f4672dce040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:47:32.012 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.801+0000 7f4672dce040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:47:32.012 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.887+0000 7f4672dce040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:47:32.012 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:31.949+0000 7f4672dce040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:47:32.012 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:32.010+0000 7f4672dce040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:47:33.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:32.833+0000 7f4672dce040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:47:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: Active manager daemon x restarted 2024-09-15T21:47:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: Activating manager daemon x 2024-09-15T21:47:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: osdmap e2: 0 total, 0 up, 0 in 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: mgrmap e6: x(active, starting, since 0.00322119s) 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: Manager daemon x is now available 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T21:47:33.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:32 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T21:47:34.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:33 smithi028 ceph-mon[30969]: Found migration_current of "None". Setting to last migration. 2024-09-15T21:47:34.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:33 smithi028 ceph-mon[30969]: mgrmap e7: x(active, since 1.00538s) 2024-09-15T21:47:34.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:33 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:34.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:33 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:34.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-15T21:47:34.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75fdf2a640 1 Processor -- start 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75fdf2a640 1 -- start start 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75fdf2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f81069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75fdf2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75f8106f90 con 0x7f75f81065c0 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f81069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f81069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46856/0 (socket says 172.21.15.28:46856) 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75f77fe640 1 -- 172.21.15.28:0/3857325147 learned_addr learned my addr 172.21.15.28:0/3857325147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.955+0000 7f75f77fe640 1 -- 172.21.15.28:0/3857325147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f81077c0 con 0x7f75f81065c0 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.956+0000 7f75f77fe640 1 --2- 172.21.15.28:0/3857325147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f81069c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009820 tx=0x7f75e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8ad1bd9d759e60 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.956+0000 7f75f67fc640 1 -- 172.21.15.28:0/3857325147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75e8037070 con 0x7f75f81065c0 2024-09-15T21:47:34.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.956+0000 7f75f67fc640 1 -- 172.21.15.28:0/3857325147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75e8032900 con 0x7f75f81065c0 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.956+0000 7f75f67fc640 1 -- 172.21.15.28:0/3857325147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75e8030850 con 0x7f75f81065c0 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.957+0000 7f75fdf2a640 1 -- 172.21.15.28:0/3857325147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 msgr2=0x7f75f81069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.957+0000 7f75fdf2a640 1 --2- 172.21.15.28:0/3857325147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f81069c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009820 tx=0x7f75e802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.957+0000 7f75fdf2a640 1 -- 172.21.15.28:0/3857325147 shutdown_connections 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.957+0000 7f75fdf2a640 1 --2- 172.21.15.28:0/3857325147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f81069c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.957+0000 7f75fdf2a640 1 -- 172.21.15.28:0/3857325147 >> 172.21.15.28:0/3857325147 conn(0x7f75f8101d30 msgr2=0x7f75f8104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.957+0000 7f75fdf2a640 1 -- 172.21.15.28:0/3857325147 shutdown_connections 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.957+0000 7f75fdf2a640 1 -- 172.21.15.28:0/3857325147 wait complete. 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.958+0000 7f75fdf2a640 1 Processor -- start 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.958+0000 7f75fdf2a640 1 -- start start 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.958+0000 7f75fdf2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.958+0000 7f75fdf2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75f810ab70 con 0x7f75f81065c0 2024-09-15T21:47:34.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.958+0000 7f75f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.958+0000 7f75f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f819b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46872/0 (socket says 172.21.15.28:46872) 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.958+0000 7f75f77fe640 1 -- 172.21.15.28:0/2144752623 learned_addr learned my addr 172.21.15.28:0/2144752623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.959+0000 7f75f77fe640 1 -- 172.21.15.28:0/2144752623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f819bcf0 con 0x7f75f81065c0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.959+0000 7f75f77fe640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f819b7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009940 tx=0x7f75e8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.959+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75e8042070 con 0x7f75f81065c0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.959+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75e80308c0 con 0x7f75f81065c0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.959+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75e8037040 con 0x7f75f81065c0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.959+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f819bf80 con 0x7f75f81065c0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.959+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f819ec70 con 0x7f75f81065c0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.960+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f75e8045070 con 0x7f75f81065c0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.960+0000 7f75f4ff9640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 0x7f75e003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.960+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 --> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f75f8106a40 con 0x7f75e00391b0 2024-09-15T21:47:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.960+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f75e806c2b0 con 0x7f75f81065c0 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.960+0000 7f75f6ffd640 1 -- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 msgr2=0x7f75e003b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3052473616 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:29.960+0000 7f75f6ffd640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 0x7f75e003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:30.161+0000 7f75f6ffd640 1 -- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 msgr2=0x7f75e003b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3052473616 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:30.161+0000 7f75f6ffd640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 0x7f75e003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:30.562+0000 7f75f6ffd640 1 -- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 msgr2=0x7f75e003b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3052473616 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:30.562+0000 7f75f6ffd640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 0x7f75e003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:31.363+0000 7f75f6ffd640 1 -- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 msgr2=0x7f75e003b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3052473616 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:31.363+0000 7f75f6ffd640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 0x7f75e003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:32.837+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46228+0+0 (secure 0 0 0) 0x7f75e806ae30 con 0x7f75f81065c0 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:32.837+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 msgr2=0x7f75e003b670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:32.837+0000 7f75f4ff9640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 0x7f75e003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.838+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7f75e803e000 con 0x7f75f81065c0 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.838+0000 7f75f4ff9640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f75e003c8f0 0x7f75e003ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.838+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f75f8106a40 con 0x7f75e003c8f0 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.839+0000 7f75f6ffd640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f75e003c8f0 0x7f75e003ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.840+0000 7f75f6ffd640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f75e003c8f0 0x7f75e003ece0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f75ec003980 tx=0x7f75ec006e10 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.840+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f75f8106a40 con 0x7f75e003c8f0 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.847+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f75f810acf0 con 0x7f75e003c8f0 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75f4ff9640 1 -- 172.21.15.28:0/2144752623 <== mgr.14118 v2:172.21.15.28:6800/2625692755 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f75f810acf0 con 0x7f75e003c8f0 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f75e003c8f0 msgr2=0x7f75e003ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f75e003c8f0 0x7f75e003ece0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f75ec003980 tx=0x7f75ec006e10 comp rx=0 tx=0).stop 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 msgr2=0x7f75f819b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f819b7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009940 tx=0x7f75e8032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 shutdown_connections 2024-09-15T21:47:34.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f75e003c8f0 0x7f75e003ece0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:34.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:6800/3052473616,v1:172.21.15.28:6801/3052473616] conn(0x7f75e00391b0 0x7f75e003b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:34.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 --2- 172.21.15.28:0/2144752623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f81065c0 0x7f75f819b7b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:34.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 >> 172.21.15.28:0/2144752623 conn(0x7f75f8101d30 msgr2=0x7f75f8102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:34.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 shutdown_connections 2024-09-15T21:47:34.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:33.848+0000 7f75fdf2a640 1 -- 172.21.15.28:0/2144752623 wait complete. 2024-09-15T21:47:34.386 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-09-15T21:47:34.386 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-09-15T21:47:35.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:34 smithi028 ceph-mon[30969]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-15T21:47:35.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:34 smithi028 ceph-mon[30969]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-15T21:47:35.652 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.743+0000 7f8968f1f640 1 Processor -- start 2024-09-15T21:47:35.652 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.743+0000 7f8968f1f640 1 -- start start 2024-09-15T21:47:35.652 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.744+0000 7f8968f1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896410a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:35.652 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.744+0000 7f8968f1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f896407be50 con 0x7f89641081b0 2024-09-15T21:47:35.652 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.744+0000 7f896259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896410a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:35.652 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.744+0000 7f896259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896410a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51606/0 (socket says 172.21.15.28:51606) 2024-09-15T21:47:35.652 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.744+0000 7f896259c640 1 -- 172.21.15.28:0/2957668070 learned_addr learned my addr 172.21.15.28:0/2957668070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.744+0000 7f896259c640 1 -- 172.21.15.28:0/2957668070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896410aae0 con 0x7f89641081b0 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.745+0000 7f896259c640 1 --2- 172.21.15.28:0/2957668070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896410a5a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8954009820 tx=0x7f895402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156ce34469c616ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.745+0000 7f8961d9b640 1 -- 172.21.15.28:0/2957668070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8954037070 con 0x7f89641081b0 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.745+0000 7f8961d9b640 1 -- 172.21.15.28:0/2957668070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8954032900 con 0x7f89641081b0 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.745+0000 7f8961d9b640 1 -- 172.21.15.28:0/2957668070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8954030850 con 0x7f89641081b0 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.746+0000 7f8968f1f640 1 -- 172.21.15.28:0/2957668070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 msgr2=0x7f896410a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.746+0000 7f8968f1f640 1 --2- 172.21.15.28:0/2957668070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896410a5a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8954009820 tx=0x7f895402a270 comp rx=0 tx=0).stop 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.746+0000 7f8968f1f640 1 -- 172.21.15.28:0/2957668070 shutdown_connections 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.746+0000 7f8968f1f640 1 --2- 172.21.15.28:0/2957668070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896410a5a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.746+0000 7f8968f1f640 1 -- 172.21.15.28:0/2957668070 >> 172.21.15.28:0/2957668070 conn(0x7f8964101d30 msgr2=0x7f8964104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.746+0000 7f8968f1f640 1 -- 172.21.15.28:0/2957668070 shutdown_connections 2024-09-15T21:47:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.746+0000 7f8968f1f640 1 -- 172.21.15.28:0/2957668070 wait complete. 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f8968f1f640 1 Processor -- start 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f8968f1f640 1 -- start start 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f8968f1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896419b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f8968f1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f896410bac0 con 0x7f89641081b0 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f896259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896419b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f896259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896419b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51608/0 (socket says 172.21.15.28:51608) 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f896259c640 1 -- 172.21.15.28:0/3239285732 learned_addr learned my addr 172.21.15.28:0/3239285732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f896259c640 1 -- 172.21.15.28:0/3239285732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896419bc30 con 0x7f89641081b0 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.747+0000 7f896259c640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896419b6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89540097f0 tx=0x7f8954030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.748+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8954037040 con 0x7f89641081b0 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.748+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8954040070 con 0x7f89641081b0 2024-09-15T21:47:35.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.748+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896419bec0 con 0x7f89641081b0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.748+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f895403a390 con 0x7f89641081b0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.748+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896419c2e0 con 0x7f89641081b0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.749+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7f895403a640 con 0x7f89641081b0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.749+0000 7f895b7fe640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f89500390f0 0x7f895003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.749+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f895406c640 con 0x7f89641081b0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.749+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8964108610 con 0x7f89641081b0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.749+0000 7f895bfff640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f89500390f0 0x7f895003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.750+0000 7f895bfff640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f89500390f0 0x7f895003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f894c0098c0 tx=0x7f894c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:35.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.754+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8954042e30 con 0x7f89641081b0 2024-09-15T21:47:35.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.889+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f895402fd40 con 0x7f89641081b0 2024-09-15T21:47:35.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.946+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f8964103040 con 0x7f89500390f0 2024-09-15T21:47:35.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.950+0000 7f895b7fe640 1 -- 172.21.15.28:0/3239285732 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8964103040 con 0x7f89500390f0 2024-09-15T21:47:35.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f89500390f0 msgr2=0x7f895003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:35.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f89500390f0 0x7f895003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f894c0098c0 tx=0x7f894c006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:35.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 msgr2=0x7f896419b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:35.657 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896419b6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89540097f0 tx=0x7f8954030d80 comp rx=0 tx=0).stop 2024-09-15T21:47:35.657 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 shutdown_connections 2024-09-15T21:47:35.657 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f89500390f0 0x7f895003b5b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:35.657 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 --2- 172.21.15.28:0/3239285732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89641081b0 0x7f896419b6f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:35.657 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.952+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 >> 172.21.15.28:0/3239285732 conn(0x7f8964101d30 msgr2=0x7f8964102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:35.657 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.953+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 shutdown_connections 2024-09-15T21:47:35.658 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:34.953+0000 7f8968f1f640 1 -- 172.21.15.28:0/3239285732 wait complete. 2024-09-15T21:47:35.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: mgrmap e8: x(active, since 2s) 2024-09-15T21:47:35.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: [15/Sep/2024:21:47:35] ENGINE Bus STARTING 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: [15/Sep/2024:21:47:35] ENGINE Serving on https://172.21.15.28:7150 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: [15/Sep/2024:21:47:35] ENGINE Error in HTTPServer.serve 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:47:35.922 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:47:35.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:35.750+0000 7f45df8f3640 -1 log_channel(cephadm) log [ERR] : [15/Sep/2024:21:47:35] ENGINE Error in HTTPServer.serve 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: Traceback (most recent call last): 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._connections.run(self.expiration_interval) 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._run(expiration_interval) 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: s = self.context.wrap_socket( 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:47:35.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: return self.sslsocket_class._create( 2024-09-15T21:47:35.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:47:35.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self.do_handshake() 2024-09-15T21:47:35.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:47:35.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._sslobj.do_handshake() 2024-09-15T21:47:35.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:47:35.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:35 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:36.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.032+0000 7f18cb2a5640 1 Processor -- start 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.032+0000 7f18cb2a5640 1 -- start start 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.032+0000 7f18cb2a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c4107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.032+0000 7f18cb2a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41080d0 con 0x7f18c4107700 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.032+0000 7f18c9041640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c4107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.033+0000 7f18c9041640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c4107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51612/0 (socket says 172.21.15.28:51612) 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.033+0000 7f18c9041640 1 -- 172.21.15.28:0/2929663407 learned_addr learned my addr 172.21.15.28:0/2929663407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.033+0000 7f18c9041640 1 -- 172.21.15.28:0/2929663407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c4108900 con 0x7f18c4107700 2024-09-15T21:47:36.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.033+0000 7f18c9041640 1 --2- 172.21.15.28:0/2929663407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c4107b00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009820 tx=0x7f18c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5eb28c3ebf9af8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:36.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18bbfff640 1 -- 172.21.15.28:0/2929663407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18c0037070 con 0x7f18c4107700 2024-09-15T21:47:36.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18bbfff640 1 -- 172.21.15.28:0/2929663407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f18c0032900 con 0x7f18c4107700 2024-09-15T21:47:36.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18bbfff640 1 -- 172.21.15.28:0/2929663407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18c0030850 con 0x7f18c4107700 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2929663407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 msgr2=0x7f18c4107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18cb2a5640 1 --2- 172.21.15.28:0/2929663407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c4107b00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009820 tx=0x7f18c002a270 comp rx=0 tx=0).stop 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2929663407 shutdown_connections 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18cb2a5640 1 --2- 172.21.15.28:0/2929663407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c4107b00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.034+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2929663407 >> 172.21.15.28:0/2929663407 conn(0x7f18c4102ed0 msgr2=0x7f18c41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2929663407 shutdown_connections 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2929663407 wait complete. 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18cb2a5640 1 Processor -- start 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18cb2a5640 1 -- start start 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18cb2a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c419b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18cb2a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18c410bcb0 con 0x7f18c4107700 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18c9041640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c419b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:36.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18c9041640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c419b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51622/0 (socket says 172.21.15.28:51622) 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.035+0000 7f18c9041640 1 -- 172.21.15.28:0/2652472877 learned_addr learned my addr 172.21.15.28:0/2652472877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.036+0000 7f18c9041640 1 -- 172.21.15.28:0/2652472877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c419bc70 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.036+0000 7f18c9041640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c419b730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18c00097f0 tx=0x7f18c0030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.036+0000 7f18ba7fc640 1 -- 172.21.15.28:0/2652472877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18c0037040 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.036+0000 7f18ba7fc640 1 -- 172.21.15.28:0/2652472877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f18c0040070 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.036+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c419bf00 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.036+0000 7f18ba7fc640 1 -- 172.21.15.28:0/2652472877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18c003a390 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.036+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c419ebf0 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.037+0000 7f18ba7fc640 1 -- 172.21.15.28:0/2652472877 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f18c0032900 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.037+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c410be30 con 0x7f18c4107700 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.037+0000 7f18ba7fc640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f18b0039200 0x7f18b003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:36.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.037+0000 7f18ba7fc640 1 -- 172.21.15.28:0/2652472877 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f18c006c700 con 0x7f18c4107700 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.037+0000 7f18c8840640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f18b0039200 0x7f18b003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.038+0000 7f18c8840640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f18b0039200 0x7f18b003b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098c0 tx=0x7f18b4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.042+0000 7f18ba7fc640 1 -- 172.21.15.28:0/2652472877 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f18c003a5e0 con 0x7f18c4107700 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.218+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f18c4104190 con 0x7f18b0039200 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.219+0000 7f18ba7fc640 1 -- 172.21.15.28:0/2652472877 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f18c4104190 con 0x7f18b0039200 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f18b0039200 msgr2=0x7f18b003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f18b0039200 0x7f18b003b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098c0 tx=0x7f18b4006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 msgr2=0x7f18c419b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c419b730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18c00097f0 tx=0x7f18c0030d80 comp rx=0 tx=0).stop 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 shutdown_connections 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f18b0039200 0x7f18b003b6c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 --2- 172.21.15.28:0/2652472877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c4107700 0x7f18c419b730 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:36.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 >> 172.21.15.28:0/2652472877 conn(0x7f18c4102ed0 msgr2=0x7f18c4103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:36.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 shutdown_connections 2024-09-15T21:47:36.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:36.221+0000 7f18cb2a5640 1 -- 172.21.15.28:0/2652472877 wait complete. 2024-09-15T21:47:36.756 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-09-15T21:47:37.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:36 smithi028 ceph-mon[30969]: [15/Sep/2024:21:47:35] ENGINE Serving on http://172.21.15.28:8765 2024-09-15T21:47:37.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:36 smithi028 ceph-mon[30969]: [15/Sep/2024:21:47:35] ENGINE Bus STARTED 2024-09-15T21:47:37.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:36 smithi028 ceph-mon[30969]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:47:37.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: Generating public/private rsa key pair. 2024-09-15T21:47:37.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: Your identification has been saved in /tmp/tmpecdnab3h/key 2024-09-15T21:47:37.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: Your public key has been saved in /tmp/tmpecdnab3h/key.pub 2024-09-15T21:47:37.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: The key fingerprint is: 2024-09-15T21:47:37.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: SHA256:dMDwa7fTeb4hnRQzaInFhMaWbRwmRWykylCnyvo0vMc ceph-e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:37.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: The key's randomart image is: 2024-09-15T21:47:37.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: +---[RSA 3072]----+ 2024-09-15T21:47:37.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | .oo.o%O. | 2024-09-15T21:47:37.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | o.+**Bo | 2024-09-15T21:47:37.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | . +o+o+ + | 2024-09-15T21:47:37.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | . = + . + | 2024-09-15T21:47:37.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | o S . . | 2024-09-15T21:47:37.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | o . . o + . | 2024-09-15T21:47:37.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | . +. o + = | 2024-09-15T21:47:37.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | o oE . + . | 2024-09-15T21:47:37.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: | o. o. | 2024-09-15T21:47:37.924 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:37 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: +----[SHA256]-----+ 2024-09-15T21:47:38.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.161+0000 7f0d94b07640 1 Processor -- start 2024-09-15T21:47:38.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.161+0000 7f0d94b07640 1 -- start start 2024-09-15T21:47:38.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.161+0000 7f0d94b07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:38.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.161+0000 7f0d94b07640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d90106f90 con 0x7f0d901065c0 2024-09-15T21:47:38.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.161+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.161+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51636/0 (socket says 172.21.15.28:51636) 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.161+0000 7f0d8e59c640 1 -- 172.21.15.28:0/117657818 learned_addr learned my addr 172.21.15.28:0/117657818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.162+0000 7f0d8e59c640 1 -- 172.21.15.28:0/117657818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d901077c0 con 0x7f0d901065c0 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.162+0000 7f0d8e59c640 1 --2- 172.21.15.28:0/117657818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901069c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009820 tx=0x7f0d8002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42337973fa08c209 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d8d59a640 1 -- 172.21.15.28:0/117657818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d80037070 con 0x7f0d901065c0 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d8d59a640 1 -- 172.21.15.28:0/117657818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0d80032900 con 0x7f0d901065c0 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d8d59a640 1 -- 172.21.15.28:0/117657818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d80030850 con 0x7f0d901065c0 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d94b07640 1 -- 172.21.15.28:0/117657818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 msgr2=0x7f0d901069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d94b07640 1 --2- 172.21.15.28:0/117657818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901069c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009820 tx=0x7f0d8002a270 comp rx=0 tx=0).stop 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d94b07640 1 -- 172.21.15.28:0/117657818 shutdown_connections 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d94b07640 1 --2- 172.21.15.28:0/117657818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901069c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:38.197 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.163+0000 7f0d94b07640 1 -- 172.21.15.28:0/117657818 >> 172.21.15.28:0/117657818 conn(0x7f0d90101d30 msgr2=0x7f0d90104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.164+0000 7f0d94b07640 1 -- 172.21.15.28:0/117657818 shutdown_connections 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.164+0000 7f0d94b07640 1 -- 172.21.15.28:0/117657818 wait complete. 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.164+0000 7f0d94b07640 1 Processor -- start 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.164+0000 7f0d94b07640 1 -- start start 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d94b07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901917f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d94b07640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9010ab70 con 0x7f0d901065c0 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901917f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51650/0 (socket says 172.21.15.28:51650) 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d8e59c640 1 -- 172.21.15.28:0/2029420695 learned_addr learned my addr 172.21.15.28:0/2029420695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d8e59c640 1 -- 172.21.15.28:0/2029420695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d90191d30 con 0x7f0d901065c0 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d8e59c640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901917f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009940 tx=0x7f0d80032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:38.198 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d877fe640 1 -- 172.21.15.28:0/2029420695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d80042070 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d877fe640 1 -- 172.21.15.28:0/2029420695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0d800308c0 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9018ff20 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d877fe640 1 -- 172.21.15.28:0/2029420695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d80037040 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.165+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d90190460 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.166+0000 7f0d877fe640 1 -- 172.21.15.28:0/2029420695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f0d80045070 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.166+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d9010acf0 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.167+0000 7f0d877fe640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f0d7c039200 0x7f0d7c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.167+0000 7f0d877fe640 1 -- 172.21.15.28:0/2029420695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0d8006c320 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.167+0000 7f0d8dd9b640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f0d7c039200 0x7f0d7c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.167+0000 7f0d8dd9b640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f0d7c039200 0x7f0d7c03b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d780098c0 tx=0x7f0d78006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.171+0000 7f0d877fe640 1 -- 172.21.15.28:0/2029420695 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0d80045520 con 0x7f0d901065c0 2024-09-15T21:47:38.199 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.350+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f0d90195610 con 0x7f0d7c039200 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.638+0000 7f0d877fe640 1 -- 172.21.15.28:0/2029420695 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0d90195610 con 0x7f0d7c039200 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f0d7c039200 msgr2=0x7f0d7c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f0d7c039200 0x7f0d7c03b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0d780098c0 tx=0x7f0d78006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 msgr2=0x7f0d901917f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901917f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009940 tx=0x7f0d80032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 shutdown_connections 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f0d7c039200 0x7f0d7c03b6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 --2- 172.21.15.28:0/2029420695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d901065c0 0x7f0d901917f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:38.200 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 >> 172.21.15.28:0/2029420695 conn(0x7f0d90101d30 msgr2=0x7f0d90102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:38.201 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 shutdown_connections 2024-09-15T21:47:38.201 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:37.640+0000 7f0d94b07640 1 -- 172.21.15.28:0/2029420695 wait complete. 2024-09-15T21:47:38.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:38 smithi028 ceph-mon[30969]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:47:38.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:38 smithi028 ceph-mon[30969]: Generating ssh key... 2024-09-15T21:47:38.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:38 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:38.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:38 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYwI8nQrVt/fkOI+XjW2/lzFCuc/ug2eBBRZ+o9d3qSOUT+GgwIajlNK6hikv8lwmF0LtdaHChFS6nihqaQe0CAgMbvyuvcc5Ysj6TPoBDajflFzhPPCCgVz9AWHGCQo4IrH27x4inQZaprDgymAcaQvGfx05FgQsAMtKFlguZXYUUig3Ah6/4XrCQeqWaU3YJBGhVt3U6V4kHW69nkHXRwvKMKnmBlUQpth0hkMD7m/BcI+C6dC9erhqU7dvLTTN6SxRhnDjsd2bg6nwmGBm2lklE5kQdadfqqZiRrqv0SVaEGc3jJKqHU+NOYcUpJCRnRUy3SvDPTWCaoxaSMU1nlBrG7Cg7mXn6p8kOeRdEl1Meon0RrWBd8Hjb5n6oeSohrdCz0e6NtFKaAvqQ3YTqEcdcRiiOEQy4YbdsFay6sZ2QypsZGR4095A1Gs99oDf3Ixoo09jw4pSRxhDtsfYLihA5b6+Cbb7p/h2ssI5bbCnGKDvYSDB9mOT3gePLaqE= ceph-e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.552+0000 7f648cde8640 1 Processor -- start 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.552+0000 7f648cde8640 1 -- start start 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.552+0000 7f648cde8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648810a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.553+0000 7f648cde8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f648807bed0 con 0x7f6488108130 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.553+0000 7f648659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648810a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.553+0000 7f648659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648810a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51660/0 (socket says 172.21.15.28:51660) 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.553+0000 7f648659c640 1 -- 172.21.15.28:0/3757218487 learned_addr learned my addr 172.21.15.28:0/3757218487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.553+0000 7f648659c640 1 -- 172.21.15.28:0/3757218487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648810aa60 con 0x7f6488108130 2024-09-15T21:47:39.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.554+0000 7f648659c640 1 --2- 172.21.15.28:0/3757218487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648810a520 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6478009820 tx=0x7f647802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41eaa95a22e1c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.554+0000 7f648559a640 1 -- 172.21.15.28:0/3757218487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478037070 con 0x7f6488108130 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.554+0000 7f648559a640 1 -- 172.21.15.28:0/3757218487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6478032900 con 0x7f6488108130 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.554+0000 7f648559a640 1 -- 172.21.15.28:0/3757218487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478030850 con 0x7f6488108130 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.554+0000 7f648cde8640 1 -- 172.21.15.28:0/3757218487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 msgr2=0x7f648810a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.554+0000 7f648cde8640 1 --2- 172.21.15.28:0/3757218487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648810a520 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6478009820 tx=0x7f647802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.555+0000 7f648cde8640 1 -- 172.21.15.28:0/3757218487 shutdown_connections 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.555+0000 7f648cde8640 1 --2- 172.21.15.28:0/3757218487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648810a520 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.555+0000 7f648cde8640 1 -- 172.21.15.28:0/3757218487 >> 172.21.15.28:0/3757218487 conn(0x7f6488101d30 msgr2=0x7f6488104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.555+0000 7f648cde8640 1 -- 172.21.15.28:0/3757218487 shutdown_connections 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.555+0000 7f648cde8640 1 -- 172.21.15.28:0/3757218487 wait complete. 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.555+0000 7f648cde8640 1 Processor -- start 2024-09-15T21:47:39.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.555+0000 7f648cde8640 1 -- start start 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648cde8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648819b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648cde8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f648810bb00 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648819b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648819b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51664/0 (socket says 172.21.15.28:51664) 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648659c640 1 -- 172.21.15.28:0/868332811 learned_addr learned my addr 172.21.15.28:0/868332811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648659c640 1 -- 172.21.15.28:0/868332811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648819bc00 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648659c640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648819b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f64780097f0 tx=0x7f6478030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f64777fe640 1 -- 172.21.15.28:0/868332811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478037040 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648819be90 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f64777fe640 1 -- 172.21.15.28:0/868332811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6478040070 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f64777fe640 1 -- 172.21.15.28:0/868332811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f647803a390 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.556+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648819eb80 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.557+0000 7f64777fe640 1 -- 172.21.15.28:0/868332811 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f647803a5e0 con 0x7f6488108130 2024-09-15T21:47:39.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.557+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6488108590 con 0x7f6488108130 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.558+0000 7f64777fe640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f6470039200 0x7f647003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.558+0000 7f64777fe640 1 -- 172.21.15.28:0/868332811 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f647806c780 con 0x7f6488108130 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.558+0000 7f6485d9b640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f6470039200 0x7f647003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.558+0000 7f6485d9b640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f6470039200 0x7f647003b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f647c0098c0 tx=0x7f647c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.562+0000 7f64777fe640 1 -- 172.21.15.28:0/868332811 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f647803a9d0 con 0x7f6488108130 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.737+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6488103060 con 0x7f6470039200 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.738+0000 7f64777fe640 1 -- 172.21.15.28:0/868332811 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6488103060 con 0x7f6470039200 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f6470039200 msgr2=0x7f647003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f6470039200 0x7f647003b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f647c0098c0 tx=0x7f647c006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 msgr2=0x7f648819b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648819b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f64780097f0 tx=0x7f6478030d80 comp rx=0 tx=0).stop 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 shutdown_connections 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f6470039200 0x7f647003b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 --2- 172.21.15.28:0/868332811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488108130 0x7f648819b6c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 >> 172.21.15.28:0/868332811 conn(0x7f6488101d30 msgr2=0x7f6488102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 shutdown_connections 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:38.741+0000 7f648cde8640 1 -- 172.21.15.28:0/868332811 wait complete. 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-15T21:47:39.357 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-09-15T21:47:39.358 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-09-15T21:47:39.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:39 smithi028 ceph-mon[30969]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:47:41.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:40 smithi028 ceph-mon[30969]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:47:42.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:41 smithi028 ceph-mon[30969]: Deploying cephadm binary to smithi028 2024-09-15T21:47:44.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-09-15T21:47:44.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.758+0000 7f45d56e4640 1 Processor -- start 2024-09-15T21:47:44.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.758+0000 7f45d56e4640 1 -- start start 2024-09-15T21:47:44.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.759+0000 7f45d56e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d0107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:44.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.759+0000 7f45d56e4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01080d0 con 0x7f45d0107700 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.759+0000 7f45ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d0107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.759+0000 7f45ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d0107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51678/0 (socket says 172.21.15.28:51678) 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.759+0000 7f45ceffd640 1 -- 172.21.15.28:0/2758188175 learned_addr learned my addr 172.21.15.28:0/2758188175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.759+0000 7f45ceffd640 1 -- 172.21.15.28:0/2758188175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d0108900 con 0x7f45d0107700 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.760+0000 7f45ceffd640 1 --2- 172.21.15.28:0/2758188175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d0107b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009820 tx=0x7f45c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf001db64570c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.760+0000 7f45cdffb640 1 -- 172.21.15.28:0/2758188175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45c8037070 con 0x7f45d0107700 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.760+0000 7f45cdffb640 1 -- 172.21.15.28:0/2758188175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f45c8032900 con 0x7f45d0107700 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.760+0000 7f45cdffb640 1 -- 172.21.15.28:0/2758188175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45c8030850 con 0x7f45d0107700 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 -- 172.21.15.28:0/2758188175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 msgr2=0x7f45d0107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 --2- 172.21.15.28:0/2758188175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d0107b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009820 tx=0x7f45c802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 -- 172.21.15.28:0/2758188175 shutdown_connections 2024-09-15T21:47:44.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 --2- 172.21.15.28:0/2758188175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d0107b00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 -- 172.21.15.28:0/2758188175 >> 172.21.15.28:0/2758188175 conn(0x7f45d0102ed0 msgr2=0x7f45d01052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 -- 172.21.15.28:0/2758188175 shutdown_connections 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 -- 172.21.15.28:0/2758188175 wait complete. 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.761+0000 7f45d56e4640 1 Processor -- start 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45d56e4640 1 -- start start 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45d56e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d019b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45d56e4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010bcb0 con 0x7f45d0107700 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d019b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d019b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51694/0 (socket says 172.21.15.28:51694) 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45ceffd640 1 -- 172.21.15.28:0/143733620 learned_addr learned my addr 172.21.15.28:0/143733620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45ceffd640 1 -- 172.21.15.28:0/143733620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d019bc60 con 0x7f45d0107700 2024-09-15T21:47:44.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.762+0000 7f45ceffd640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d019b720 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009940 tx=0x7f45c8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.763+0000 7f45c7fff640 1 -- 172.21.15.28:0/143733620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45c8042070 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.763+0000 7f45c7fff640 1 -- 172.21.15.28:0/143733620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f45c80308c0 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.763+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d019bef0 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.763+0000 7f45c7fff640 1 -- 172.21.15.28:0/143733620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45c8037040 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.763+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d019ebe0 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.764+0000 7f45c7fff640 1 -- 172.21.15.28:0/143733620 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f45c8045070 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.764+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d010be30 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.764+0000 7f45c7fff640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f45bc039200 0x7f45bc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.764+0000 7f45c7fff640 1 -- 172.21.15.28:0/143733620 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f45c806c2e0 con 0x7f45d0107700 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.764+0000 7f45ce7fc640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f45bc039200 0x7f45bc03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.764+0000 7f45ce7fc640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f45bc039200 0x7f45bc03b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098c0 tx=0x7f45b8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:44.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.769+0000 7f45c7fff640 1 -- 172.21.15.28:0/143733620 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f45c804b170 con 0x7f45d0107700 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:39.951+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7f45d0104170 con 0x7f45bc039200 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.883+0000 7f45c7fff640 1 -- 172.21.15.28:0/143733620 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f45d0104170 con 0x7f45bc039200 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.886+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f45bc039200 msgr2=0x7f45bc03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f45bc039200 0x7f45bc03b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f45b80098c0 tx=0x7f45b8006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 msgr2=0x7f45d019b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d019b720 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009940 tx=0x7f45c8032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 shutdown_connections 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f45bc039200 0x7f45bc03b6c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 --2- 172.21.15.28:0/143733620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0107700 0x7f45d019b720 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 >> 172.21.15.28:0/143733620 conn(0x7f45d0102ed0 msgr2=0x7f45d0103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 shutdown_connections 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:43.887+0000 7f45d56e4640 1 -- 172.21.15.28:0/143733620 wait complete. 2024-09-15T21:47:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-09-15T21:47:45.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:44 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:45.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:44 smithi028 ceph-mon[30969]: Added host smithi028 2024-09-15T21:47:45.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:44 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:47:45.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.905+0000 7f57a0868640 1 Processor -- start 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.905+0000 7f57a0868640 1 -- start start 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.905+0000 7f57a0868640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.905+0000 7f57a0868640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f579c1080d0 con 0x7f579c107700 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.905+0000 7f579a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.905+0000 7f579a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58396/0 (socket says 172.21.15.28:58396) 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.905+0000 7f579a59c640 1 -- 172.21.15.28:0/3556498545 learned_addr learned my addr 172.21.15.28:0/3556498545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.906+0000 7f579a59c640 1 -- 172.21.15.28:0/3556498545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579c108900 con 0x7f579c107700 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.906+0000 7f579a59c640 1 --2- 172.21.15.28:0/3556498545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c107b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f578c009820 tx=0x7f578c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48c8d0d7cd26af9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:45.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f579959a640 1 -- 172.21.15.28:0/3556498545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f578c037070 con 0x7f579c107700 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f579959a640 1 -- 172.21.15.28:0/3556498545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f578c032900 con 0x7f579c107700 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f579959a640 1 -- 172.21.15.28:0/3556498545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f578c030850 con 0x7f579c107700 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f57a0868640 1 -- 172.21.15.28:0/3556498545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 msgr2=0x7f579c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f57a0868640 1 --2- 172.21.15.28:0/3556498545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c107b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f578c009820 tx=0x7f578c02a270 comp rx=0 tx=0).stop 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f57a0868640 1 -- 172.21.15.28:0/3556498545 shutdown_connections 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f57a0868640 1 --2- 172.21.15.28:0/3556498545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c107b00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f57a0868640 1 -- 172.21.15.28:0/3556498545 >> 172.21.15.28:0/3556498545 conn(0x7f579c102ed0 msgr2=0x7f579c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f57a0868640 1 -- 172.21.15.28:0/3556498545 shutdown_connections 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.907+0000 7f57a0868640 1 -- 172.21.15.28:0/3556498545 wait complete. 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.908+0000 7f57a0868640 1 Processor -- start 2024-09-15T21:47:45.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.908+0000 7f57a0868640 1 -- start start 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.908+0000 7f57a0868640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c19b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.908+0000 7f57a0868640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f579c10bcb0 con 0x7f579c107700 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.908+0000 7f579a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c19b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.908+0000 7f579a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c19b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58406/0 (socket says 172.21.15.28:58406) 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.908+0000 7f579a59c640 1 -- 172.21.15.28:0/1485317000 learned_addr learned my addr 172.21.15.28:0/1485317000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.909+0000 7f579a59c640 1 -- 172.21.15.28:0/1485317000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579c19bc60 con 0x7f579c107700 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.909+0000 7f579a59c640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c19b720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f578c009940 tx=0x7f578c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.909+0000 7f578b7fe640 1 -- 172.21.15.28:0/1485317000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f578c042070 con 0x7f579c107700 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.909+0000 7f578b7fe640 1 -- 172.21.15.28:0/1485317000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f578c0308c0 con 0x7f579c107700 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.909+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579c19bef0 con 0x7f579c107700 2024-09-15T21:47:45.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.909+0000 7f578b7fe640 1 -- 172.21.15.28:0/1485317000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f578c037040 con 0x7f579c107700 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.909+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f579c19ebe0 con 0x7f579c107700 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.910+0000 7f578b7fe640 1 -- 172.21.15.28:0/1485317000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f578c045070 con 0x7f579c107700 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.910+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f579c10be30 con 0x7f579c107700 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.910+0000 7f578b7fe640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f5784039200 0x7f578403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.910+0000 7f578b7fe640 1 -- 172.21.15.28:0/1485317000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f578c06c320 con 0x7f579c107700 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.910+0000 7f5799d9b640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f5784039200 0x7f578403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.911+0000 7f5799d9b640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f5784039200 0x7f578403b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57900098c0 tx=0x7f5790006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:44.915+0000 7f578b7fe640 1 -- 172.21.15.28:0/1485317000 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f578c045360 con 0x7f579c107700 2024-09-15T21:47:45.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.115+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f579c104170 con 0x7f5784039200 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.119+0000 7f578b7fe640 1 -- 172.21.15.28:0/1485317000 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f579c104170 con 0x7f5784039200 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f5784039200 msgr2=0x7f578403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f5784039200 0x7f578403b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57900098c0 tx=0x7f5790006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 msgr2=0x7f579c19b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c19b720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f578c009940 tx=0x7f578c032ce0 comp rx=0 tx=0).stop 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 shutdown_connections 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f5784039200 0x7f578403b6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 --2- 172.21.15.28:0/1485317000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c107700 0x7f579c19b720 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.121+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 >> 172.21.15.28:0/1485317000 conn(0x7f579c102ed0 msgr2=0x7f579c103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.122+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 shutdown_connections 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:45.122+0000 7f57a0868640 1 -- 172.21.15.28:0/1485317000 wait complete. 2024-09-15T21:47:45.890 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-09-15T21:47:46.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:46 smithi028 ceph-mon[30969]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:47:46.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:46 smithi028 ceph-mon[30969]: Saving service mon spec with placement count:5 2024-09-15T21:47:46.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:46 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:47.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-15T21:47:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.310+0000 7f4b6de87640 1 Processor -- start 2024-09-15T21:47:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.310+0000 7f4b6de87640 1 -- start start 2024-09-15T21:47:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.310+0000 7f4b6de87640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b6810a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.310+0000 7f4b6de87640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6807be90 con 0x7f4b68108160 2024-09-15T21:47:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.311+0000 7f4b677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b6810a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.311+0000 7f4b677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b6810a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58418/0 (socket says 172.21.15.28:58418) 2024-09-15T21:47:47.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.311+0000 7f4b677fe640 1 -- 172.21.15.28:0/490217883 learned_addr learned my addr 172.21.15.28:0/490217883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.311+0000 7f4b677fe640 1 -- 172.21.15.28:0/490217883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6810aa90 con 0x7f4b68108160 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.312+0000 7f4b677fe640 1 --2- 172.21.15.28:0/490217883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b6810a550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009820 tx=0x7f4b5002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45452772e92e4c3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.312+0000 7f4b667fc640 1 -- 172.21.15.28:0/490217883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b50037070 con 0x7f4b68108160 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.312+0000 7f4b667fc640 1 -- 172.21.15.28:0/490217883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4b50032900 con 0x7f4b68108160 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.312+0000 7f4b667fc640 1 -- 172.21.15.28:0/490217883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b50030850 con 0x7f4b68108160 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 -- 172.21.15.28:0/490217883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 msgr2=0x7f4b6810a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 --2- 172.21.15.28:0/490217883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b6810a550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009820 tx=0x7f4b5002a270 comp rx=0 tx=0).stop 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 -- 172.21.15.28:0/490217883 shutdown_connections 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 --2- 172.21.15.28:0/490217883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b6810a550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:47.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 -- 172.21.15.28:0/490217883 >> 172.21.15.28:0/490217883 conn(0x7f4b68101d00 msgr2=0x7f4b68104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 -- 172.21.15.28:0/490217883 shutdown_connections 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 -- 172.21.15.28:0/490217883 wait complete. 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.313+0000 7f4b6de87640 1 Processor -- start 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b6de87640 1 -- start start 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b6de87640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b681079e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b6de87640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6810bad0 con 0x7f4b68108160 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b681079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b681079e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58430/0 (socket says 172.21.15.28:58430) 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b677fe640 1 -- 172.21.15.28:0/1448213115 learned_addr learned my addr 172.21.15.28:0/1448213115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b677fe640 1 -- 172.21.15.28:0/1448213115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b68107f20 con 0x7f4b68108160 2024-09-15T21:47:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.314+0000 7f4b677fe640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b681079e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b500097f0 tx=0x7f4b50030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.315+0000 7f4b64ff9640 1 -- 172.21.15.28:0/1448213115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b50037040 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.315+0000 7f4b64ff9640 1 -- 172.21.15.28:0/1448213115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4b50040070 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.315+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b68106110 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.315+0000 7f4b64ff9640 1 -- 172.21.15.28:0/1448213115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b5003a390 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.315+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b681065f0 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.316+0000 7f4b64ff9640 1 -- 172.21.15.28:0/1448213115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f4b5003a630 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.316+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b681085c0 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.316+0000 7f4b64ff9640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f4b540391b0 0x7f4b5403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.316+0000 7f4b64ff9640 1 -- 172.21.15.28:0/1448213115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4b5006c800 con 0x7f4b68108160 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.316+0000 7f4b66ffd640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f4b540391b0 0x7f4b5403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.316+0000 7f4b66ffd640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f4b540391b0 0x7f4b5403b670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b580098c0 tx=0x7f4b58006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.325+0000 7f4b64ff9640 1 -- 172.21.15.28:0/1448213115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4b5004a200 con 0x7f4b68108160 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.522+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4b68106ea0 con 0x7f4b540391b0 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.526+0000 7f4b64ff9640 1 -- 172.21.15.28:0/1448213115 <== mgr.14118 v2:172.21.15.28:6800/2625692755 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4b68106ea0 con 0x7f4b540391b0 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f4b540391b0 msgr2=0x7f4b5403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f4b540391b0 0x7f4b5403b670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b580098c0 tx=0x7f4b58006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 msgr2=0x7f4b681079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b681079e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b500097f0 tx=0x7f4b50030d80 comp rx=0 tx=0).stop 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 shutdown_connections 2024-09-15T21:47:47.106 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f4b540391b0 0x7f4b5403b670 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:47.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 --2- 172.21.15.28:0/1448213115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b68108160 0x7f4b681079e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:47.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 >> 172.21.15.28:0/1448213115 conn(0x7f4b68101d00 msgr2=0x7f4b68102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:47.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 shutdown_connections 2024-09-15T21:47:47.107 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:46.528+0000 7f4b6de87640 1 -- 172.21.15.28:0/1448213115 wait complete. 2024-09-15T21:47:47.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:47 smithi028 ceph-mon[30969]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:47:47.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:47 smithi028 ceph-mon[30969]: Saving service mgr spec with placement count:2 2024-09-15T21:47:47.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:47 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:48.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:48 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:48.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:48 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:48.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:48 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2085892678' entity='client.admin' 2024-09-15T21:47:49.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.466+0000 7f8fcdaaa640 1 Processor -- start 2024-09-15T21:47:49.906 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.466+0000 7f8fcdaaa640 1 -- start start 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.466+0000 7f8fcdaaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc810a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.466+0000 7f8fcdaaa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc807be50 con 0x7f8fc8108200 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.467+0000 7f8fc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc810a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.467+0000 7f8fc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc810a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58442/0 (socket says 172.21.15.28:58442) 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.467+0000 7f8fc77fe640 1 -- 172.21.15.28:0/3647151689 learned_addr learned my addr 172.21.15.28:0/3647151689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.467+0000 7f8fc77fe640 1 -- 172.21.15.28:0/3647151689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc810ab30 con 0x7f8fc8108200 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fc77fe640 1 --2- 172.21.15.28:0/3647151689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc810a5f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009820 tx=0x7f8fb802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f287a4f304a1d227 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fc67fc640 1 -- 172.21.15.28:0/3647151689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fb8037070 con 0x7f8fc8108200 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fc67fc640 1 -- 172.21.15.28:0/3647151689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fb8032920 con 0x7f8fc8108200 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fc67fc640 1 -- 172.21.15.28:0/3647151689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fb8030810 con 0x7f8fc8108200 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/3647151689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 msgr2=0x7f8fc810a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:49.907 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fcdaaa640 1 --2- 172.21.15.28:0/3647151689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc810a5f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009820 tx=0x7f8fb802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/3647151689 shutdown_connections 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fcdaaa640 1 --2- 172.21.15.28:0/3647151689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc810a5f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.468+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/3647151689 >> 172.21.15.28:0/3647151689 conn(0x7f8fc8101da0 msgr2=0x7f8fc81041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.469+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/3647151689 shutdown_connections 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.469+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/3647151689 wait complete. 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.469+0000 7f8fcdaaa640 1 Processor -- start 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.469+0000 7f8fcdaaa640 1 -- start start 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fcdaaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc819b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fcdaaa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc810bb50 con 0x7f8fc8108200 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc819b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc819b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58444/0 (socket says 172.21.15.28:58444) 2024-09-15T21:47:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc77fe640 1 -- 172.21.15.28:0/2085892678 learned_addr learned my addr 172.21.15.28:0/2085892678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc77fe640 1 -- 172.21.15.28:0/2085892678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc819bc40 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc77fe640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc819b700 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80097f0 tx=0x7f8fb80327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc4ff9640 1 -- 172.21.15.28:0/2085892678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fb8037040 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc819bed0 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc4ff9640 1 -- 172.21.15.28:0/2085892678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fb8030cc0 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fc4ff9640 1 -- 172.21.15.28:0/2085892678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fb803a6b0 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.470+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc819ebc0 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.471+0000 7f8fc4ff9640 1 -- 172.21.15.28:0/2085892678 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f8fb803a8a0 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.471+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc8108660 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.472+0000 7f8fc4ff9640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f8fb00391b0 0x7f8fb003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.472+0000 7f8fc4ff9640 1 -- 172.21.15.28:0/2085892678 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8fb806c7f0 con 0x7f8fc8108200 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.472+0000 7f8fc6ffd640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f8fb00391b0 0x7f8fb003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.472+0000 7f8fc6ffd640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f8fb00391b0 0x7f8fb003b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0098c0 tx=0x7f8fbc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.476+0000 7f8fc4ff9640 1 -- 172.21.15.28:0/2085892678 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8fb803b8d0 con 0x7f8fc8108200 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.666+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f8fc819f1c0 con 0x7f8fc8108200 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.668+0000 7f8fc4ff9640 1 -- 172.21.15.28:0/2085892678 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f8fb803b3f0 con 0x7f8fc8108200 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.671+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f8fb00391b0 msgr2=0x7f8fb003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f8fb00391b0 0x7f8fb003b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0098c0 tx=0x7f8fbc006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 msgr2=0x7f8fc819b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc819b700 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80097f0 tx=0x7f8fb80327b0 comp rx=0 tx=0).stop 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 shutdown_connections 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f8fb00391b0 0x7f8fb003b670 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 --2- 172.21.15.28:0/2085892678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc8108200 0x7f8fc819b700 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 >> 172.21.15.28:0/2085892678 conn(0x7f8fc8101da0 msgr2=0x7f8fc8102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 shutdown_connections 2024-09-15T21:47:49.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:48.672+0000 7f8fcdaaa640 1 -- 172.21.15.28:0/2085892678 wait complete. 2024-09-15T21:47:51.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38dd25d640 1 Processor -- start 2024-09-15T21:47:51.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38dd25d640 1 -- start start 2024-09-15T21:47:51.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38dd25d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d8107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:51.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38dd25d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38d81080d0 con 0x7f38d8107700 2024-09-15T21:47:51.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d8107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:51.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d8107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58446/0 (socket says 172.21.15.28:58446) 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38d6d9d640 1 -- 172.21.15.28:0/2844190136 learned_addr learned my addr 172.21.15.28:0/2844190136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.259+0000 7f38d6d9d640 1 -- 172.21.15.28:0/2844190136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d8108900 con 0x7f38d8107700 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.260+0000 7f38d6d9d640 1 --2- 172.21.15.28:0/2844190136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d8107b00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009820 tx=0x7f38c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a15ccbf5229253 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.260+0000 7f38d659c640 1 -- 172.21.15.28:0/2844190136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38c8037070 con 0x7f38d8107700 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.260+0000 7f38d659c640 1 -- 172.21.15.28:0/2844190136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c8032920 con 0x7f38d8107700 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.260+0000 7f38d659c640 1 -- 172.21.15.28:0/2844190136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38c8030810 con 0x7f38d8107700 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.261+0000 7f38dd25d640 1 -- 172.21.15.28:0/2844190136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 msgr2=0x7f38d8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.261+0000 7f38dd25d640 1 --2- 172.21.15.28:0/2844190136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d8107b00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009820 tx=0x7f38c802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.261+0000 7f38dd25d640 1 -- 172.21.15.28:0/2844190136 shutdown_connections 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.261+0000 7f38dd25d640 1 --2- 172.21.15.28:0/2844190136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d8107b00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.261+0000 7f38dd25d640 1 -- 172.21.15.28:0/2844190136 >> 172.21.15.28:0/2844190136 conn(0x7f38d8102ed0 msgr2=0x7f38d81052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:51.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.261+0000 7f38dd25d640 1 -- 172.21.15.28:0/2844190136 shutdown_connections 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.261+0000 7f38dd25d640 1 -- 172.21.15.28:0/2844190136 wait complete. 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38dd25d640 1 Processor -- start 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38dd25d640 1 -- start start 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38dd25d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38dd25d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38d810bcb0 con 0x7f38d8107700 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58452/0 (socket says 172.21.15.28:58452) 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38d6d9d640 1 -- 172.21.15.28:0/3092705681 learned_addr learned my addr 172.21.15.28:0/3092705681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38d6d9d640 1 -- 172.21.15.28:0/3092705681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d807e0d0 con 0x7f38d8107700 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.262+0000 7f38d6d9d640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d807db90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f38c8030560 tx=0x7f38c8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:51.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.263+0000 7f38d4d99640 1 -- 172.21.15.28:0/3092705681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38c8037040 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.263+0000 7f38d4d99640 1 -- 172.21.15.28:0/3092705681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c8040070 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.263+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d807c2c0 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.263+0000 7f38d4d99640 1 -- 172.21.15.28:0/3092705681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38c803a740 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.263+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d807c800 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.264+0000 7f38d4d99640 1 -- 172.21.15.28:0/3092705681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f38c803a8e0 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.264+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d8107b80 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.264+0000 7f38d4d99640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f38c4039200 0x7f38c403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.264+0000 7f38d4d99640 1 -- 172.21.15.28:0/3092705681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f38c806c830 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.264+0000 7f38cffff640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f38c4039200 0x7f38c403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.264+0000 7f38cffff640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f38c4039200 0x7f38c403b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f38c00098c0 tx=0x7f38c0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.269+0000 7f38d4d99640 1 -- 172.21.15.28:0/3092705681 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f38c8044c70 con 0x7f38d8107700 2024-09-15T21:47:51.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.449+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f38d807ca80 con 0x7f38d8107700 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.451+0000 7f38d4d99640 1 -- 172.21.15.28:0/3092705681 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f38c8041270 con 0x7f38d8107700 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f38c4039200 msgr2=0x7f38c403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f38c4039200 0x7f38c403b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f38c00098c0 tx=0x7f38c0006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 msgr2=0x7f38d807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d807db90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f38c8030560 tx=0x7f38c8032670 comp rx=0 tx=0).stop 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 shutdown_connections 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f38c4039200 0x7f38c403b6c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 --2- 172.21.15.28:0/3092705681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d8107700 0x7f38d807db90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 >> 172.21.15.28:0/3092705681 conn(0x7f38d8102ed0 msgr2=0x7f38d8103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 shutdown_connections 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:50.454+0000 7f38dd25d640 1 -- 172.21.15.28:0/3092705681 wait complete. 2024-09-15T21:47:51.086 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-09-15T21:47:51.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:51 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:51.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:51 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3092705681' entity='client.admin' 2024-09-15T21:47:51.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:51 smithi028 ceph-mon[30969]: from='mgr.14118 172.21.15.28:0/939190639' entity='mgr.x' 2024-09-15T21:47:52.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4268532120' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-15T21:47:52.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ignoring --setuser ceph since I am not root 2024-09-15T21:47:52.672 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ignoring --setgroup ceph since I am not root 2024-09-15T21:47:52.672 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:52.530+0000 7fa750af5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:47:52.672 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:52.589+0000 7fa750af5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:47:52.969 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:52.744+0000 7fa750af5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:47:52.970 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:52.813+0000 7fa750af5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:47:52.970 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:52.967+0000 7fa750af5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:47:53.345 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:53 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:53.114+0000 7fa750af5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:47:53.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.492+0000 7fec4752a640 1 Processor -- start 2024-09-15T21:47:53.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.493+0000 7fec4752a640 1 -- start start 2024-09-15T21:47:53.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.493+0000 7fec4752a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4010a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:53.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.493+0000 7fec4752a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec4007bed0 con 0x7fec40108180 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.493+0000 7fec452c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4010a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.493+0000 7fec452c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4010a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41846/0 (socket says 172.21.15.28:41846) 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.493+0000 7fec452c6640 1 -- 172.21.15.28:0/1401865673 learned_addr learned my addr 172.21.15.28:0/1401865673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.493+0000 7fec452c6640 1 -- 172.21.15.28:0/1401865673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4010aab0 con 0x7fec40108180 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.494+0000 7fec452c6640 1 --2- 172.21.15.28:0/1401865673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4010a570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009820 tx=0x7fec3c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5e65136d05a29e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.494+0000 7fec37fff640 1 -- 172.21.15.28:0/1401865673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec3c037070 con 0x7fec40108180 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.494+0000 7fec37fff640 1 -- 172.21.15.28:0/1401865673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec3c032920 con 0x7fec40108180 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.494+0000 7fec37fff640 1 -- 172.21.15.28:0/1401865673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec3c030810 con 0x7fec40108180 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.495+0000 7fec4752a640 1 -- 172.21.15.28:0/1401865673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 msgr2=0x7fec4010a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.495+0000 7fec4752a640 1 --2- 172.21.15.28:0/1401865673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4010a570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009820 tx=0x7fec3c02a270 comp rx=0 tx=0).stop 2024-09-15T21:47:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.495+0000 7fec4752a640 1 -- 172.21.15.28:0/1401865673 shutdown_connections 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.495+0000 7fec4752a640 1 --2- 172.21.15.28:0/1401865673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4010a570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.495+0000 7fec4752a640 1 -- 172.21.15.28:0/1401865673 >> 172.21.15.28:0/1401865673 conn(0x7fec40101da0 msgr2=0x7fec401041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.495+0000 7fec4752a640 1 -- 172.21.15.28:0/1401865673 shutdown_connections 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.495+0000 7fec4752a640 1 -- 172.21.15.28:0/1401865673 wait complete. 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.496+0000 7fec4752a640 1 Processor -- start 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.496+0000 7fec4752a640 1 -- start start 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec4752a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4019b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec4752a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec4010bb90 con 0x7fec40108180 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec452c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4019b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec452c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4019b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41848/0 (socket says 172.21.15.28:41848) 2024-09-15T21:47:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec452c6640 1 -- 172.21.15.28:0/4268532120 learned_addr learned my addr 172.21.15.28:0/4268532120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec452c6640 1 -- 172.21.15.28:0/4268532120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4019bc70 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec452c6640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4019b730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec3c030380 tx=0x7fec3c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec3c037040 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec3c040070 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec3c03a7b0 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4019bf00 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.497+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4019ebf0 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.498+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fec3c03aa30 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.498+0000 7fec367fc640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7fec2c0391b0 0x7fec2c03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.499+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fec3c06c900 con 0x7fec40108180 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.499+0000 7fec44ac5640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7fec2c0391b0 0x7fec2c03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.499+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec401085e0 con 0x7fec40108180 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.499+0000 7fec44ac5640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7fec2c0391b0 0x7fec2c03b670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fec300098c0 tx=0x7fec30006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.504+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fec3c031560 con 0x7fec40108180 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:51.762+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fec4019c3e0 con 0x7fec40108180 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.375+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fec4019c3e0 con 0x7fec40108180 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.377+0000 7fec367fc640 1 -- 172.21.15.28:0/4268532120 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fec3c03ad70 con 0x7fec40108180 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7fec2c0391b0 msgr2=0x7fec2c03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7fec2c0391b0 0x7fec2c03b670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fec300098c0 tx=0x7fec30006db0 comp rx=0 tx=0).stop 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 msgr2=0x7fec4019b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4019b730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec3c030380 tx=0x7fec3c032670 comp rx=0 tx=0).stop 2024-09-15T21:47:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 shutdown_connections 2024-09-15T21:47:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7fec2c0391b0 0x7fec2c03b670 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 --2- 172.21.15.28:0/4268532120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec40108180 0x7fec4019b730 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.378+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 >> 172.21.15.28:0/4268532120 conn(0x7fec40101da0 msgr2=0x7fec40102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.379+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 shutdown_connections 2024-09-15T21:47:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:52.379+0000 7fec4752a640 1 -- 172.21.15.28:0/4268532120 wait complete. 2024-09-15T21:47:53.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:53 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4268532120' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-15T21:47:53.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:53 smithi028 ceph-mon[30969]: mgrmap e9: x(active, since 19s) 2024-09-15T21:47:53.652 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:53 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:53.445+0000 7fa750af5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:47:53.652 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:53 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:53.508+0000 7fa750af5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:47:53.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:53 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:53.706+0000 7fa750af5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:47:54.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:54 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3878707893' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-15T21:47:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-15T21:47:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-15T21:47:54.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.726+0000 7f7feca7c640 1 Processor -- start 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.726+0000 7f7feca7c640 1 -- start start 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.727+0000 7f7feca7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe810a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.727+0000 7f7feca7c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe807be50 con 0x7f7fe81081b0 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.727+0000 7f7fe659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe810a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:54.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.727+0000 7f7fe659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe810a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41876/0 (socket says 172.21.15.28:41876) 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.727+0000 7f7fe659c640 1 -- 172.21.15.28:0/1762040387 learned_addr learned my addr 172.21.15.28:0/1762040387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.727+0000 7f7fe659c640 1 -- 172.21.15.28:0/1762040387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe810aae0 con 0x7f7fe81081b0 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.728+0000 7f7fe659c640 1 --2- 172.21.15.28:0/1762040387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe810a5a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009820 tx=0x7f7fd802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fca171d5a57237 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.728+0000 7f7fe5d9b640 1 -- 172.21.15.28:0/1762040387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd8037070 con 0x7f7fe81081b0 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.728+0000 7f7fe5d9b640 1 -- 172.21.15.28:0/1762040387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd8032920 con 0x7f7fe81081b0 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.728+0000 7f7fe5d9b640 1 -- 172.21.15.28:0/1762040387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd8030810 con 0x7f7fe81081b0 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.729+0000 7f7feca7c640 1 -- 172.21.15.28:0/1762040387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 msgr2=0x7f7fe810a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.729+0000 7f7feca7c640 1 --2- 172.21.15.28:0/1762040387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe810a5a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009820 tx=0x7f7fd802a270 comp rx=0 tx=0).stop 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.729+0000 7f7feca7c640 1 -- 172.21.15.28:0/1762040387 shutdown_connections 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.729+0000 7f7feca7c640 1 --2- 172.21.15.28:0/1762040387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe810a5a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.729+0000 7f7feca7c640 1 -- 172.21.15.28:0/1762040387 >> 172.21.15.28:0/1762040387 conn(0x7f7fe8101d30 msgr2=0x7f7fe8104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.729+0000 7f7feca7c640 1 -- 172.21.15.28:0/1762040387 shutdown_connections 2024-09-15T21:47:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.729+0000 7f7feca7c640 1 -- 172.21.15.28:0/1762040387 wait complete. 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7feca7c640 1 Processor -- start 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7feca7c640 1 -- start start 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7feca7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe819b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7feca7c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe810bb00 con 0x7f7fe81081b0 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7fe659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe819b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7fe659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe819b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41882/0 (socket says 172.21.15.28:41882) 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7fe659c640 1 -- 172.21.15.28:0/3878707893 learned_addr learned my addr 172.21.15.28:0/3878707893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.730+0000 7f7fe659c640 1 -- 172.21.15.28:0/3878707893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe819bc30 con 0x7f7fe81081b0 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.731+0000 7f7fe659c640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe819b6f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8030380 tx=0x7f7fd8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.731+0000 7f7fdf7fe640 1 -- 172.21.15.28:0/3878707893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd8037040 con 0x7f7fe81081b0 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.731+0000 7f7fdf7fe640 1 -- 172.21.15.28:0/3878707893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd8040070 con 0x7f7fe81081b0 2024-09-15T21:47:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.731+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe819bec0 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.731+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe819ebb0 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.731+0000 7f7fdf7fe640 1 -- 172.21.15.28:0/3878707893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fd803a7b0 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.732+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fe8108610 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.732+0000 7f7fdf7fe640 1 -- 172.21.15.28:0/3878707893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f7fd803a9e0 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.732+0000 7f7fdf7fe640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 0x7f7fd403b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.732+0000 7f7fdf7fe640 1 -- 172.21.15.28:0/3878707893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7fd806ca80 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.732+0000 7f7fdffff640 1 -- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 msgr2=0x7f7fd403b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2625692755 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.732+0000 7f7fdffff640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 0x7f7fd403b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.741+0000 7f7fdf7fe640 1 -- 172.21.15.28:0/3878707893 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7fd8044c70 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.933+0000 7f7fdffff640 1 -- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 msgr2=0x7f7fd403b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2625692755 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.933+0000 7f7fdffff640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 0x7f7fd403b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.967+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f7fe819f1a0 con 0x7f7fe81081b0 2024-09-15T21:47:54.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.967+0000 7f7fdf7fe640 1 -- 172.21.15.28:0/3878707893 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f7fd803ae20 con 0x7f7fe81081b0 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 msgr2=0x7f7fd403b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 0x7f7fd403b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 msgr2=0x7f7fe819b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe819b6f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8030380 tx=0x7f7fd8032670 comp rx=0 tx=0).stop 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 shutdown_connections 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f7fd4039260 0x7f7fd403b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 --2- 172.21.15.28:0/3878707893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe81081b0 0x7f7fe819b6f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 >> 172.21.15.28:0/3878707893 conn(0x7f7fe8101d30 msgr2=0x7f7fe8102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 shutdown_connections 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:53.970+0000 7f7feca7c640 1 -- 172.21.15.28:0/3878707893 wait complete. 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-09-15T21:47:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-09-15T21:47:54.911 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:54 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:54.659+0000 7fa750af5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:47:54.911 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:54 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:54.730+0000 7fa750af5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:47:55.606 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:55.483+0000 7fa750af5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:47:55.607 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:55.604+0000 7fa750af5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:47:55.906 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:55.753+0000 7fa750af5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:47:55.906 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:55.816+0000 7fa750af5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:47:56.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:55.976+0000 7fa750af5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:47:56.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.047+0000 7fa750af5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:47:56.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.109+0000 7fa750af5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:47:56.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.275+0000 7fa750af5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:47:56.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.342+0000 7fa750af5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:47:56.567 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.411+0000 7fa750af5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:47:56.567 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.502+0000 7fa750af5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:47:56.567 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.564+0000 7fa750af5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:47:56.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:56.626+0000 7fa750af5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:47:57.706 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:47:57.451+0000 7fa750af5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: Active manager daemon x restarted 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: Activating manager daemon x 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: osdmap e3: 0 total, 0 up, 0 in 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: mgrmap e10: x(active, starting, since 0.00329563s) 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T21:47:57.707 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T21:47:57.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: Manager daemon x is now available 2024-09-15T21:47:57.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:47:57.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T21:47:57.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T21:47:59.446 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-09-15T21:47:59.446 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-15T21:47:59.446 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-15T21:47:59.446 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-09-15T21:47:59.446 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.091+0000 7f4308d4a640 1 Processor -- start 2024-09-15T21:47:59.446 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.091+0000 7f4308d4a640 1 -- start start 2024-09-15T21:47:59.446 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.091+0000 7f4308d4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.091+0000 7f4308d4a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4304108560 con 0x7f4304107b90 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.092+0000 7f430259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.092+0000 7f430259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41896/0 (socket says 172.21.15.28:41896) 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.092+0000 7f430259c640 1 -- 172.21.15.28:0/2235972491 learned_addr learned my addr 172.21.15.28:0/2235972491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.092+0000 7f430259c640 1 -- 172.21.15.28:0/2235972491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43041067a0 con 0x7f4304107b90 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.093+0000 7f430259c640 1 --2- 172.21.15.28:0/2235972491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304107f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009820 tx=0x7f42f402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11698e9e0cb068b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.093+0000 7f4301d9b640 1 -- 172.21.15.28:0/2235972491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f4037070 con 0x7f4304107b90 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.093+0000 7f4301d9b640 1 -- 172.21.15.28:0/2235972491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42f4032920 con 0x7f4304107b90 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.093+0000 7f4301d9b640 1 -- 172.21.15.28:0/2235972491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f4030810 con 0x7f4304107b90 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.094+0000 7f4308d4a640 1 -- 172.21.15.28:0/2235972491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 msgr2=0x7f4304107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.094+0000 7f4308d4a640 1 --2- 172.21.15.28:0/2235972491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304107f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009820 tx=0x7f42f402a270 comp rx=0 tx=0).stop 2024-09-15T21:47:59.447 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.094+0000 7f4308d4a640 1 -- 172.21.15.28:0/2235972491 shutdown_connections 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.094+0000 7f4308d4a640 1 --2- 172.21.15.28:0/2235972491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304107f90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.094+0000 7f4308d4a640 1 -- 172.21.15.28:0/2235972491 >> 172.21.15.28:0/2235972491 conn(0x7f4304101500 msgr2=0x7f4304103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.094+0000 7f4308d4a640 1 -- 172.21.15.28:0/2235972491 shutdown_connections 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.094+0000 7f4308d4a640 1 -- 172.21.15.28:0/2235972491 wait complete. 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f4308d4a640 1 Processor -- start 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f4308d4a640 1 -- start start 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f4308d4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304106150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f4308d4a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f430410ab60 con 0x7f4304107b90 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f430259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304106150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f430259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304106150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41902/0 (socket says 172.21.15.28:41902) 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f430259c640 1 -- 172.21.15.28:0/4182555337 learned_addr learned my addr 172.21.15.28:0/4182555337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:47:59.448 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f430259c640 1 -- 172.21.15.28:0/4182555337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430419e450 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f430259c640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304106150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f42f40097f0 tx=0x7f42f40327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.095+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f4037040 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.096+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42f4030cc0 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.096+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f430419e620 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.096+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f403a640 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.096+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430419ea40 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.097+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f42f403a940 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.097+0000 7f42fb7fe640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 0x7f42f003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.097+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 --> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f42f003bdd0 con 0x7f42f0039200 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.097+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f42f406c9a0 con 0x7f4304107b90 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.097+0000 7f42fbfff640 1 -- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 msgr2=0x7f42f003b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2625692755 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.097+0000 7f42fbfff640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 0x7f42f003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:47:59.449 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.298+0000 7f42fbfff640 1 -- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 msgr2=0x7f42f003b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2625692755 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.298+0000 7f42fbfff640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 0x7f42f003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.699+0000 7f42fbfff640 1 -- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 msgr2=0x7f42f003b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2625692755 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:55.699+0000 7f42fbfff640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 0x7f42f003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:56.501+0000 7f42fbfff640 1 -- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 msgr2=0x7f42f003b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2625692755 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:56.501+0000 7f42fbfff640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 0x7f42f003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:57.455+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46241+0+0 (secure 0 0 0) 0x7f42f406b450 con 0x7f4304107b90 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:57.455+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 msgr2=0x7f42f003b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:57.455+0000 7f42fb7fe640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 0x7f42f003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.456+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7f42f4049cf0 con 0x7f4304107b90 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.456+0000 7f42fb7fe640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42f003ccc0 0x7f42f003f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.456+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f42f003bdd0 con 0x7f42f003ccc0 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.457+0000 7f42fbfff640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42f003ccc0 0x7f42f003f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:47:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.458+0000 7f42fbfff640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42f003ccc0 0x7f42f003f0b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f42ec003980 tx=0x7f42ec006e10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.458+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f42f003bdd0 con 0x7f42f003ccc0 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.465+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f430410ace0 con 0x7f42f003ccc0 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f42fb7fe640 1 -- 172.21.15.28:0/4182555337 <== mgr.14150 v2:172.21.15.28:6800/1670367809 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f430410ace0 con 0x7f42f003ccc0 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42f003ccc0 msgr2=0x7f42f003f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42f003ccc0 0x7f42f003f0b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f42ec003980 tx=0x7f42ec006e10 comp rx=0 tx=0).stop 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 msgr2=0x7f4304106150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304106150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f42f40097f0 tx=0x7f42f40327b0 comp rx=0 tx=0).stop 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 shutdown_connections 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42f003ccc0 0x7f42f003f0b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:6800/2625692755,v1:172.21.15.28:6801/2625692755] conn(0x7f42f0039200 0x7f42f003b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 --2- 172.21.15.28:0/4182555337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4304107b90 0x7f4304106150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:47:59.451 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 >> 172.21.15.28:0/4182555337 conn(0x7f4304101500 msgr2=0x7f43041020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:47:59.452 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 shutdown_connections 2024-09-15T21:47:59.452 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:58.466+0000 7f4308d4a640 1 -- 172.21.15.28:0/4182555337 wait complete. 2024-09-15T21:47:59.452 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-09-15T21:47:59.452 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-09-15T21:47:59.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:59 smithi028 ceph-mon[30969]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-15T21:47:59.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:59 smithi028 ceph-mon[30969]: mgrmap e11: x(active, since 1.00636s) 2024-09-15T21:47:59.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:59 smithi028 ceph-mon[30969]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-15T21:47:59.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:47:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:00.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-15T21:48:00.809 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.862+0000 7fdce9972640 1 Processor -- start 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.862+0000 7fdce9972640 1 -- start start 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.862+0000 7fdce9972640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce410a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.862+0000 7fdce9972640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdce407be10 con 0x7fdce4108240 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.863+0000 7fdce2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce410a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.863+0000 7fdce2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce410a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41972/0 (socket says 172.21.15.28:41972) 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.863+0000 7fdce2ffd640 1 -- 172.21.15.28:0/139193900 learned_addr learned my addr 172.21.15.28:0/139193900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.863+0000 7fdce2ffd640 1 -- 172.21.15.28:0/139193900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce410ab70 con 0x7fdce4108240 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.864+0000 7fdce2ffd640 1 --2- 172.21.15.28:0/139193900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce410a630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009820 tx=0x7fdcd402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8388ab52f56c9b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.864+0000 7fdce27fc640 1 -- 172.21.15.28:0/139193900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcd4037070 con 0x7fdce4108240 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce27fc640 1 -- 172.21.15.28:0/139193900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd4032920 con 0x7fdce4108240 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce27fc640 1 -- 172.21.15.28:0/139193900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcd4030810 con 0x7fdce4108240 2024-09-15T21:48:00.810 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce9972640 1 -- 172.21.15.28:0/139193900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 msgr2=0x7fdce410a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce9972640 1 --2- 172.21.15.28:0/139193900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce410a630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009820 tx=0x7fdcd402a270 comp rx=0 tx=0).stop 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce9972640 1 -- 172.21.15.28:0/139193900 shutdown_connections 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce9972640 1 --2- 172.21.15.28:0/139193900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce410a630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce9972640 1 -- 172.21.15.28:0/139193900 >> 172.21.15.28:0/139193900 conn(0x7fdce4101da0 msgr2=0x7fdce41041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce9972640 1 -- 172.21.15.28:0/139193900 shutdown_connections 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.865+0000 7fdce9972640 1 -- 172.21.15.28:0/139193900 wait complete. 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.866+0000 7fdce9972640 1 Processor -- start 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.866+0000 7fdce9972640 1 -- start start 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.866+0000 7fdce9972640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.866+0000 7fdce9972640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdce410bb90 con 0x7fdce4108240 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.866+0000 7fdce2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:00.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.866+0000 7fdce2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce419d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41982/0 (socket says 172.21.15.28:41982) 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.866+0000 7fdce2ffd640 1 -- 172.21.15.28:0/1791461031 learned_addr learned my addr 172.21.15.28:0/1791461031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.867+0000 7fdce2ffd640 1 -- 172.21.15.28:0/1791461031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce419de90 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.867+0000 7fdce2ffd640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce419d950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4030380 tx=0x7fdcd4032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.867+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcd4037040 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.867+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd4040070 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.867+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce419e120 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.867+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcd403a7b0 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.867+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce41a0e10 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.868+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fdcd403aa30 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.868+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce41086a0 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.869+0000 7fdce8970640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdcd00390f0 0x7fdcd003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.869+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdcd406c990 con 0x7fdce4108240 2024-09-15T21:48:00.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.869+0000 7fdcdbfff640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdcd00390f0 0x7fdcd003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.870+0000 7fdcdbfff640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdcd00390f0 0x7fdcd003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098c0 tx=0x7fdccc006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:47:59.874+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdcd4045980 con 0x7fdce4108240 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.026+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fdcd4045460 con 0x7fdce4108240 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.065+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fdce4103000 con 0x7fdcd00390f0 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.206+0000 7fdce8970640 1 -- 172.21.15.28:0/1791461031 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fdce4103000 con 0x7fdcd00390f0 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdcd00390f0 msgr2=0x7fdcd003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdcd00390f0 0x7fdcd003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098c0 tx=0x7fdccc006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 msgr2=0x7fdce419d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce419d950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4030380 tx=0x7fdcd4032670 comp rx=0 tx=0).stop 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 shutdown_connections 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdcd00390f0 0x7fdcd003b5b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:00.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 --2- 172.21.15.28:0/1791461031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce4108240 0x7fdce419d950 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:00.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 >> 172.21.15.28:0/1791461031 conn(0x7fdce4101da0 msgr2=0x7fdce41028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:00.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 shutdown_connections 2024-09-15T21:48:00.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:00.208+0000 7fdce9972640 1 -- 172.21.15.28:0/1791461031 wait complete. 2024-09-15T21:48:00.814 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:48:00.633+0000 7fa6bbc49640 -1 log_channel(cephadm) log [ERR] : [15/Sep/2024:21:48:00] ENGINE Error in HTTPServer.serve 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: Traceback (most recent call last): 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._connections.run(self.expiration_interval) 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._run(expiration_interval) 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:48:00.896 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: s = self.context.wrap_socket( 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: return self.sslsocket_class._create( 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self.do_handshake() 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._sslobj.do_handshake() 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:48:00.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:48:01.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:01.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-mon[30969]: mgrmap e12: x(active, since 2s) 2024-09-15T21:48:01.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-mon[30969]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:01.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:01.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:02.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:01 smithi028 ceph-mon[30969]: [15/Sep/2024:21:48:00] ENGINE Bus STARTING 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:01 smithi028 ceph-mon[30969]: [15/Sep/2024:21:48:00] ENGINE Serving on https://172.21.15.28:7150 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:01 smithi028 ceph-mon[30969]: [15/Sep/2024:21:48:00] ENGINE Error in HTTPServer.serve 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:48:02.172 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:01 smithi028 ceph-mon[30969]: [15/Sep/2024:21:48:00] ENGINE Serving on http://172.21.15.28:8765 2024-09-15T21:48:02.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:01 smithi028 ceph-mon[30969]: [15/Sep/2024:21:48:00] ENGINE Bus STARTED 2024-09-15T21:48:02.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:01 smithi028 ceph-mon[30969]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:02.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:02.874 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$TljfkfsSV5Cv5Aa.QvqpQ.y66n1TjM9IF57wn21N0I6bFVLn2WhTG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726436881, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-15T21:48:02.874 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.203+0000 7f9bf8de3640 1 Processor -- start 2024-09-15T21:48:02.874 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.203+0000 7f9bf8de3640 1 -- start start 2024-09-15T21:48:02.874 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.204+0000 7f9bf8de3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf41069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.204+0000 7f9bf8de3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4106f90 con 0x7f9bf41065c0 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.204+0000 7f9bf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf41069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.204+0000 7f9bf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf41069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37748/0 (socket says 172.21.15.28:37748) 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.204+0000 7f9bf259c640 1 -- 172.21.15.28:0/1164453793 learned_addr learned my addr 172.21.15.28:0/1164453793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.204+0000 7f9bf259c640 1 -- 172.21.15.28:0/1164453793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf41077c0 con 0x7f9bf41065c0 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.205+0000 7f9bf259c640 1 --2- 172.21.15.28:0/1164453793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf41069c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009820 tx=0x7f9be402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe97588e8a47641 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.205+0000 7f9bf159a640 1 -- 172.21.15.28:0/1164453793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9be4037070 con 0x7f9bf41065c0 2024-09-15T21:48:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.205+0000 7f9bf159a640 1 -- 172.21.15.28:0/1164453793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be4032920 con 0x7f9bf41065c0 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.205+0000 7f9bf159a640 1 -- 172.21.15.28:0/1164453793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9be4030810 con 0x7f9bf41065c0 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 -- 172.21.15.28:0/1164453793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 msgr2=0x7f9bf41069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 --2- 172.21.15.28:0/1164453793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf41069c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009820 tx=0x7f9be402a270 comp rx=0 tx=0).stop 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 -- 172.21.15.28:0/1164453793 shutdown_connections 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 --2- 172.21.15.28:0/1164453793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf41069c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 -- 172.21.15.28:0/1164453793 >> 172.21.15.28:0/1164453793 conn(0x7f9bf4101d30 msgr2=0x7f9bf4104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 -- 172.21.15.28:0/1164453793 shutdown_connections 2024-09-15T21:48:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 -- 172.21.15.28:0/1164453793 wait complete. 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.206+0000 7f9bf8de3640 1 Processor -- start 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf8de3640 1 -- start start 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf8de3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf8de3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf410ab70 con 0x7f9bf41065c0 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf419b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37756/0 (socket says 172.21.15.28:37756) 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf259c640 1 -- 172.21.15.28:0/229595419 learned_addr learned my addr 172.21.15.28:0/229595419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf259c640 1 -- 172.21.15.28:0/229595419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf419bd40 con 0x7f9bf41065c0 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9bf259c640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf419b800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9be4030560 tx=0x7f9be4032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9be37fe640 1 -- 172.21.15.28:0/229595419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9be4037040 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.207+0000 7f9be37fe640 1 -- 172.21.15.28:0/229595419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be4040070 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.208+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf419bfd0 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.208+0000 7f9be37fe640 1 -- 172.21.15.28:0/229595419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9be403a740 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.208+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf419ec70 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.208+0000 7f9be37fe640 1 -- 172.21.15.28:0/229595419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f9be403a8e0 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.209+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf410acf0 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.209+0000 7f9be37fe640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f9bdc039260 0x7f9bdc03b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.209+0000 7f9be37fe640 1 -- 172.21.15.28:0/229595419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9be406cbe0 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.209+0000 7f9bf1d9b640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f9bdc039260 0x7f9bdc03b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.209+0000 7f9bf1d9b640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f9bdc039260 0x7f9bdc03b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9be80098c0 tx=0x7f9be8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.213+0000 7f9be37fe640 1 -- 172.21.15.28:0/229595419 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9be4044c70 con 0x7f9bf41065c0 2024-09-15T21:48:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.404+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f9bf4106a40 con 0x7f9bdc039260 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.654+0000 7f9be37fe640 1 -- 172.21.15.28:0/229595419 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f9bf4106a40 con 0x7f9bdc039260 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f9bdc039260 msgr2=0x7f9bdc03b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f9bdc039260 0x7f9bdc03b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9be80098c0 tx=0x7f9be8006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 msgr2=0x7f9bf419b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf419b800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9be4030560 tx=0x7f9be4032670 comp rx=0 tx=0).stop 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 shutdown_connections 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f9bdc039260 0x7f9bdc03b720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 --2- 172.21.15.28:0/229595419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41065c0 0x7f9bf419b800 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 >> 172.21.15.28:0/229595419 conn(0x7f9bf4101d30 msgr2=0x7f9bf4102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 shutdown_connections 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:01.656+0000 7f9bf8de3640 1 -- 172.21.15.28:0/229595419 wait complete. 2024-09-15T21:48:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-09-15T21:48:03.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:03 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/665854063' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-15T21:48:03.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:03 smithi028 ceph-mon[30969]: mgrmap e13: x(active, since 6s) 2024-09-15T21:48:04.065 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-09-15T21:48:04.066 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4adbf77640 1 Processor -- start 2024-09-15T21:48:04.066 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4adbf77640 1 -- start start 2024-09-15T21:48:04.066 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4adbf77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad4107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:04.066 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4adbf77640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41080d0 con 0x7f4ad4107700 2024-09-15T21:48:04.066 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4ad9d13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad4107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:04.066 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4ad9d13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad4107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37762/0 (socket says 172.21.15.28:37762) 2024-09-15T21:48:04.066 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4ad9d13640 1 -- 172.21.15.28:0/3565476369 learned_addr learned my addr 172.21.15.28:0/3565476369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.287+0000 7f4ad9d13640 1 -- 172.21.15.28:0/3565476369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad4108900 con 0x7f4ad4107700 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.288+0000 7f4ad9d13640 1 --2- 172.21.15.28:0/3565476369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad4107b00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0009820 tx=0x7f4ad002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb1f226185ac954 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.288+0000 7f4ad8d11640 1 -- 172.21.15.28:0/3565476369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad0037070 con 0x7f4ad4107700 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.288+0000 7f4ad8d11640 1 -- 172.21.15.28:0/3565476369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad0032920 con 0x7f4ad4107700 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.288+0000 7f4ad8d11640 1 -- 172.21.15.28:0/3565476369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad0030810 con 0x7f4ad4107700 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.289+0000 7f4adbf77640 1 -- 172.21.15.28:0/3565476369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 msgr2=0x7f4ad4107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.289+0000 7f4adbf77640 1 --2- 172.21.15.28:0/3565476369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad4107b00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0009820 tx=0x7f4ad002a270 comp rx=0 tx=0).stop 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.289+0000 7f4adbf77640 1 -- 172.21.15.28:0/3565476369 shutdown_connections 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.289+0000 7f4adbf77640 1 --2- 172.21.15.28:0/3565476369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad4107b00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.289+0000 7f4adbf77640 1 -- 172.21.15.28:0/3565476369 >> 172.21.15.28:0/3565476369 conn(0x7f4ad4102ed0 msgr2=0x7f4ad41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.289+0000 7f4adbf77640 1 -- 172.21.15.28:0/3565476369 shutdown_connections 2024-09-15T21:48:04.067 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.289+0000 7f4adbf77640 1 -- 172.21.15.28:0/3565476369 wait complete. 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4adbf77640 1 Processor -- start 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4adbf77640 1 -- start start 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4adbf77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad407db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4adbf77640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad410bcb0 con 0x7f4ad4107700 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4ad9d13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad407db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4ad9d13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad407db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37766/0 (socket says 172.21.15.28:37766) 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4ad9d13640 1 -- 172.21.15.28:0/665854063 learned_addr learned my addr 172.21.15.28:0/665854063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4ad9d13640 1 -- 172.21.15.28:0/665854063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad407e0d0 con 0x7f4ad4107700 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4ad9d13640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad407db90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0030560 tx=0x7f4ad0032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.290+0000 7f4acaffd640 1 -- 172.21.15.28:0/665854063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad0037040 con 0x7f4ad4107700 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.291+0000 7f4acaffd640 1 -- 172.21.15.28:0/665854063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad0040070 con 0x7f4ad4107700 2024-09-15T21:48:04.068 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.291+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad407c2c0 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.291+0000 7f4acaffd640 1 -- 172.21.15.28:0/665854063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ad003a740 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.291+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad407c800 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.291+0000 7f4acaffd640 1 -- 172.21.15.28:0/665854063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f4ad003b420 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.292+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ad4107b80 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.292+0000 7f4acaffd640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4ac0039260 0x7f4ac003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.292+0000 7f4acaffd640 1 -- 172.21.15.28:0/665854063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4ad006c960 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.292+0000 7f4ad9512640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4ac0039260 0x7f4ac003b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.292+0000 7f4ad9512640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4ac0039260 0x7f4ac003b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40098c0 tx=0x7f4ac4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.296+0000 7f4acaffd640 1 -- 172.21.15.28:0/665854063 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4ad004a200 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.470+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f4ad407ca80 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.471+0000 7f4acaffd640 1 -- 172.21.15.28:0/665854063 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f4ad0031e80 con 0x7f4ad4107700 2024-09-15T21:48:04.069 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4ac0039260 msgr2=0x7f4ac003b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4ac0039260 0x7f4ac003b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40098c0 tx=0x7f4ac4006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 msgr2=0x7f4ad407db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad407db90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0030560 tx=0x7f4ad0032670 comp rx=0 tx=0).stop 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 shutdown_connections 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4ac0039260 0x7f4ac003b720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 --2- 172.21.15.28:0/665854063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad4107700 0x7f4ad407db90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 >> 172.21.15.28:0/665854063 conn(0x7f4ad4102ed0 msgr2=0x7f4ad4103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 shutdown_connections 2024-09-15T21:48:04.070 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:03.473+0000 7f4adbf77640 1 -- 172.21.15.28:0/665854063 wait complete. 2024-09-15T21:48:04.071 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T21:48:04.071 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-09-15T21:48:04.079 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T21:48:04.079 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-09-15T21:48:04.079 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-09-15T21:48:04.079 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-15T21:48:04.081 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-09-15T21:48:04.081 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:04.081 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-09-15T21:48:04.081 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-09-15T21:48:04.081 INFO:teuthology.orchestra.run.smithi028.stdout: Password: d35p1am7sx 2024-09-15T21:48:04.081 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:04.081 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config directory 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.485+0000 7fc330beb640 1 Processor -- start 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.485+0000 7fc330beb640 1 -- start start 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.486+0000 7fc330beb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c107b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.486+0000 7fc330beb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1080d0 con 0x7fc32c107700 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.486+0000 7fc32a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.486+0000 7fc32a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37778/0 (socket says 172.21.15.28:37778) 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.486+0000 7fc32a59c640 1 -- 172.21.15.28:0/2131740593 learned_addr learned my addr 172.21.15.28:0/2131740593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.486+0000 7fc32a59c640 1 -- 172.21.15.28:0/2131740593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c108900 con 0x7fc32c107700 2024-09-15T21:48:05.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.487+0000 7fc32a59c640 1 --2- 172.21.15.28:0/2131740593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c107b00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc324009820 tx=0x7fc32402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3c8fc67b16f334 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:05.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.487+0000 7fc32959a640 1 -- 172.21.15.28:0/2131740593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc324037070 con 0x7fc32c107700 2024-09-15T21:48:05.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.487+0000 7fc32959a640 1 -- 172.21.15.28:0/2131740593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc324032920 con 0x7fc32c107700 2024-09-15T21:48:05.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.487+0000 7fc32959a640 1 -- 172.21.15.28:0/2131740593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc324030810 con 0x7fc32c107700 2024-09-15T21:48:05.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.488+0000 7fc330beb640 1 -- 172.21.15.28:0/2131740593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 msgr2=0x7fc32c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:05.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.488+0000 7fc330beb640 1 --2- 172.21.15.28:0/2131740593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c107b00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc324009820 tx=0x7fc32402a270 comp rx=0 tx=0).stop 2024-09-15T21:48:05.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.488+0000 7fc330beb640 1 -- 172.21.15.28:0/2131740593 shutdown_connections 2024-09-15T21:48:05.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.488+0000 7fc330beb640 1 --2- 172.21.15.28:0/2131740593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c107b00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.488+0000 7fc330beb640 1 -- 172.21.15.28:0/2131740593 >> 172.21.15.28:0/2131740593 conn(0x7fc32c102ed0 msgr2=0x7fc32c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.488+0000 7fc330beb640 1 -- 172.21.15.28:0/2131740593 shutdown_connections 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.488+0000 7fc330beb640 1 -- 172.21.15.28:0/2131740593 wait complete. 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc330beb640 1 Processor -- start 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc330beb640 1 -- start start 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc330beb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c19b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc330beb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10bcb0 con 0x7fc32c107700 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc32a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c19b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc32a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c19b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37794/0 (socket says 172.21.15.28:37794) 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc32a59c640 1 -- 172.21.15.28:0/163991005 learned_addr learned my addr 172.21.15.28:0/163991005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc32a59c640 1 -- 172.21.15.28:0/163991005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c19bc70 con 0x7fc32c107700 2024-09-15T21:48:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.489+0000 7fc32a59c640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c19b730 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc324030560 tx=0x7fc324032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.490+0000 7fc3237fe640 1 -- 172.21.15.28:0/163991005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc324037040 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.490+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c19bf00 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.490+0000 7fc3237fe640 1 -- 172.21.15.28:0/163991005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc324040070 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.490+0000 7fc3237fe640 1 -- 172.21.15.28:0/163991005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc32403a740 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.490+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c19ebf0 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.491+0000 7fc3237fe640 1 -- 172.21.15.28:0/163991005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fc32403a9b0 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.491+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc32c10be30 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.491+0000 7fc3237fe640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc3180392a0 0x7fc31803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.491+0000 7fc3237fe640 1 -- 172.21.15.28:0/163991005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc32406cb90 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.491+0000 7fc329d9b640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc3180392a0 0x7fc31803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.492+0000 7fc329d9b640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc3180392a0 0x7fc31803b760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc3140098c0 tx=0x7fc314006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.496+0000 7fc3237fe640 1 -- 172.21.15.28:0/163991005 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc324039950 con 0x7fc32c107700 2024-09-15T21:48:05.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.764+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fc32c19edc0 con 0x7fc32c107700 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.767+0000 7fc3237fe640 1 -- 172.21.15.28:0/163991005 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fc32403aca0 con 0x7fc32c107700 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc3180392a0 msgr2=0x7fc31803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc3180392a0 0x7fc31803b760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc3140098c0 tx=0x7fc314006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 msgr2=0x7fc32c19b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c19b730 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc324030560 tx=0x7fc324032670 comp rx=0 tx=0).stop 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 shutdown_connections 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc3180392a0 0x7fc31803b760 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 --2- 172.21.15.28:0/163991005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c107700 0x7fc32c19b730 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 >> 172.21.15.28:0/163991005 conn(0x7fc32c102ed0 msgr2=0x7fc32c103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 shutdown_connections 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-09-15T21:48:04.769+0000 7fc330beb640 1 -- 172.21.15.28:0/163991005 wait complete. 2024-09-15T21:48:05.360 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-09-15T21:48:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-15T21:48:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:05.362 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-09-15T21:48:05.402 INFO:tasks.cephadm:Fetching config... 2024-09-15T21:48:05.402 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:48:05.402 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-15T21:48:05.421 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-15T21:48:05.421 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:48:05.421 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-15T21:48:05.478 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-15T21:48:05.478 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:48:05.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-15T21:48:05.544 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-15T21:48:05.544 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:48:05.544 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-15T21:48:05.602 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-15T21:48:05.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYwI8nQrVt/fkOI+XjW2/lzFCuc/ug2eBBRZ+o9d3qSOUT+GgwIajlNK6hikv8lwmF0LtdaHChFS6nihqaQe0CAgMbvyuvcc5Ysj6TPoBDajflFzhPPCCgVz9AWHGCQo4IrH27x4inQZaprDgymAcaQvGfx05FgQsAMtKFlguZXYUUig3Ah6/4XrCQeqWaU3YJBGhVt3U6V4kHW69nkHXRwvKMKnmBlUQpth0hkMD7m/BcI+C6dC9erhqU7dvLTTN6SxRhnDjsd2bg6nwmGBm2lklE5kQdadfqqZiRrqv0SVaEGc3jJKqHU+NOYcUpJCRnRUy3SvDPTWCaoxaSMU1nlBrG7Cg7mXn6p8kOeRdEl1Meon0RrWBd8Hjb5n6oeSohrdCz0e6NtFKaAvqQ3YTqEcdcRiiOEQy4YbdsFay6sZ2QypsZGR4095A1Gs99oDf3Ixoo09jw4pSRxhDtsfYLihA5b6+Cbb7p/h2ssI5bbCnGKDvYSDB9mOT3gePLaqE= ceph-e78b7764-73ab-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T21:48:05.679 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYwI8nQrVt/fkOI+XjW2/lzFCuc/ug2eBBRZ+o9d3qSOUT+GgwIajlNK6hikv8lwmF0LtdaHChFS6nihqaQe0CAgMbvyuvcc5Ysj6TPoBDajflFzhPPCCgVz9AWHGCQo4IrH27x4inQZaprDgymAcaQvGfx05FgQsAMtKFlguZXYUUig3Ah6/4XrCQeqWaU3YJBGhVt3U6V4kHW69nkHXRwvKMKnmBlUQpth0hkMD7m/BcI+C6dC9erhqU7dvLTTN6SxRhnDjsd2bg6nwmGBm2lklE5kQdadfqqZiRrqv0SVaEGc3jJKqHU+NOYcUpJCRnRUy3SvDPTWCaoxaSMU1nlBrG7Cg7mXn6p8kOeRdEl1Meon0RrWBd8Hjb5n6oeSohrdCz0e6NtFKaAvqQ3YTqEcdcRiiOEQy4YbdsFay6sZ2QypsZGR4095A1Gs99oDf3Ixoo09jw4pSRxhDtsfYLihA5b6+Cbb7p/h2ssI5bbCnGKDvYSDB9mOT3gePLaqE= ceph-e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:05.694 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYwI8nQrVt/fkOI+XjW2/lzFCuc/ug2eBBRZ+o9d3qSOUT+GgwIajlNK6hikv8lwmF0LtdaHChFS6nihqaQe0CAgMbvyuvcc5Ysj6TPoBDajflFzhPPCCgVz9AWHGCQo4IrH27x4inQZaprDgymAcaQvGfx05FgQsAMtKFlguZXYUUig3Ah6/4XrCQeqWaU3YJBGhVt3U6V4kHW69nkHXRwvKMKnmBlUQpth0hkMD7m/BcI+C6dC9erhqU7dvLTTN6SxRhnDjsd2bg6nwmGBm2lklE5kQdadfqqZiRrqv0SVaEGc3jJKqHU+NOYcUpJCRnRUy3SvDPTWCaoxaSMU1nlBrG7Cg7mXn6p8kOeRdEl1Meon0RrWBd8Hjb5n6oeSohrdCz0e6NtFKaAvqQ3YTqEcdcRiiOEQy4YbdsFay6sZ2QypsZGR4095A1Gs99oDf3Ixoo09jw4pSRxhDtsfYLihA5b6+Cbb7p/h2ssI5bbCnGKDvYSDB9mOT3gePLaqE= ceph-e78b7764-73ab-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T21:48:05.736 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYwI8nQrVt/fkOI+XjW2/lzFCuc/ug2eBBRZ+o9d3qSOUT+GgwIajlNK6hikv8lwmF0LtdaHChFS6nihqaQe0CAgMbvyuvcc5Ysj6TPoBDajflFzhPPCCgVz9AWHGCQo4IrH27x4inQZaprDgymAcaQvGfx05FgQsAMtKFlguZXYUUig3Ah6/4XrCQeqWaU3YJBGhVt3U6V4kHW69nkHXRwvKMKnmBlUQpth0hkMD7m/BcI+C6dC9erhqU7dvLTTN6SxRhnDjsd2bg6nwmGBm2lklE5kQdadfqqZiRrqv0SVaEGc3jJKqHU+NOYcUpJCRnRUy3SvDPTWCaoxaSMU1nlBrG7Cg7mXn6p8kOeRdEl1Meon0RrWBd8Hjb5n6oeSohrdCz0e6NtFKaAvqQ3YTqEcdcRiiOEQy4YbdsFay6sZ2QypsZGR4095A1Gs99oDf3Ixoo09jw4pSRxhDtsfYLihA5b6+Cbb7p/h2ssI5bbCnGKDvYSDB9mOT3gePLaqE= ceph-e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:05.749 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYwI8nQrVt/fkOI+XjW2/lzFCuc/ug2eBBRZ+o9d3qSOUT+GgwIajlNK6hikv8lwmF0LtdaHChFS6nihqaQe0CAgMbvyuvcc5Ysj6TPoBDajflFzhPPCCgVz9AWHGCQo4IrH27x4inQZaprDgymAcaQvGfx05FgQsAMtKFlguZXYUUig3Ah6/4XrCQeqWaU3YJBGhVt3U6V4kHW69nkHXRwvKMKnmBlUQpth0hkMD7m/BcI+C6dC9erhqU7dvLTTN6SxRhnDjsd2bg6nwmGBm2lklE5kQdadfqqZiRrqv0SVaEGc3jJKqHU+NOYcUpJCRnRUy3SvDPTWCaoxaSMU1nlBrG7Cg7mXn6p8kOeRdEl1Meon0RrWBd8Hjb5n6oeSohrdCz0e6NtFKaAvqQ3YTqEcdcRiiOEQy4YbdsFay6sZ2QypsZGR4095A1Gs99oDf3Ixoo09jw4pSRxhDtsfYLihA5b6+Cbb7p/h2ssI5bbCnGKDvYSDB9mOT3gePLaqE= ceph-e78b7764-73ab-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T21:48:05.791 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYwI8nQrVt/fkOI+XjW2/lzFCuc/ug2eBBRZ+o9d3qSOUT+GgwIajlNK6hikv8lwmF0LtdaHChFS6nihqaQe0CAgMbvyuvcc5Ysj6TPoBDajflFzhPPCCgVz9AWHGCQo4IrH27x4inQZaprDgymAcaQvGfx05FgQsAMtKFlguZXYUUig3Ah6/4XrCQeqWaU3YJBGhVt3U6V4kHW69nkHXRwvKMKnmBlUQpth0hkMD7m/BcI+C6dC9erhqU7dvLTTN6SxRhnDjsd2bg6nwmGBm2lklE5kQdadfqqZiRrqv0SVaEGc3jJKqHU+NOYcUpJCRnRUy3SvDPTWCaoxaSMU1nlBrG7Cg7mXn6p8kOeRdEl1Meon0RrWBd8Hjb5n6oeSohrdCz0e6NtFKaAvqQ3YTqEcdcRiiOEQy4YbdsFay6sZ2QypsZGR4095A1Gs99oDf3Ixoo09jw4pSRxhDtsfYLihA5b6+Cbb7p/h2ssI5bbCnGKDvYSDB9mOT3gePLaqE= ceph-e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:05.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-15T21:48:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:05 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/163991005' entity='client.admin' 2024-09-15T21:48:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.159+0000 7fdf4e46d640 1 -- 172.21.15.28:0/1656276730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 msgr2=0x7fdf481019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.159+0000 7fdf4e46d640 1 --2- 172.21.15.28:0/1656276730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf481019e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009880 tx=0x7fdf3802a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.159+0000 7fdf4e46d640 1 -- 172.21.15.28:0/1656276730 shutdown_connections 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.159+0000 7fdf4e46d640 1 --2- 172.21.15.28:0/1656276730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf481019e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.159+0000 7fdf4e46d640 1 -- 172.21.15.28:0/1656276730 >> 172.21.15.28:0/1656276730 conn(0x7fdf480fcd70 msgr2=0x7fdf480ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.159+0000 7fdf4e46d640 1 -- 172.21.15.28:0/1656276730 shutdown_connections 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.159+0000 7fdf4e46d640 1 -- 172.21.15.28:0/1656276730 wait complete. 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.160+0000 7fdf4e46d640 1 Processor -- start 2024-09-15T21:48:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.160+0000 7fdf4e46d640 1 -- start start 2024-09-15T21:48:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.160+0000 7fdf4e46d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf48196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.160+0000 7fdf4e46d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481076f0 con 0x7fdf481015e0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.160+0000 7fdf47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf48196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.160+0000 7fdf47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf48196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37810/0 (socket says 172.21.15.28:37810) 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.160+0000 7fdf47fff640 1 -- 172.21.15.28:0/2368224875 learned_addr learned my addr 172.21.15.28:0/2368224875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.161+0000 7fdf47fff640 1 -- 172.21.15.28:0/2368224875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf481972a0 con 0x7fdf481015e0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.161+0000 7fdf47fff640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf48196d60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf38004f60 tx=0x7fdf380048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.161+0000 7fdf457fa640 1 -- 172.21.15.28:0/2368224875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf3803f070 con 0x7fdf481015e0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.161+0000 7fdf457fa640 1 -- 172.21.15.28:0/2368224875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf38004d40 con 0x7fdf481015e0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.161+0000 7fdf457fa640 1 -- 172.21.15.28:0/2368224875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf38035030 con 0x7fdf481015e0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.161+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf48197530 con 0x7fdf481015e0 2024-09-15T21:48:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.162+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf48197950 con 0x7fdf481015e0 2024-09-15T21:48:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.162+0000 7fdf457fa640 1 -- 172.21.15.28:0/2368224875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fdf38042070 con 0x7fdf481015e0 2024-09-15T21:48:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.163+0000 7fdf457fa640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdf3003d710 0x7fdf3003fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.163+0000 7fdf457fa640 1 -- 172.21.15.28:0/2368224875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdf3806cfe0 con 0x7fdf481015e0 2024-09-15T21:48:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.163+0000 7fdf477fe640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdf3003d710 0x7fdf3003fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.163+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf48107e70 con 0x7fdf481015e0 2024-09-15T21:48:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.163+0000 7fdf477fe640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdf3003d710 0x7fdf3003fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0098c0 tx=0x7fdf3c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.168+0000 7fdf457fa640 1 -- 172.21.15.28:0/2368224875 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdf38042360 con 0x7fdf481015e0 2024-09-15T21:48:07.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.351+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fdf4802d8d0 con 0x7fdf481015e0 2024-09-15T21:48:07.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.354+0000 7fdf457fa640 1 -- 172.21.15.28:0/2368224875 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fdf3802f300 con 0x7fdf481015e0 2024-09-15T21:48:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdf3003d710 msgr2=0x7fdf3003fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdf3003d710 0x7fdf3003fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0098c0 tx=0x7fdf3c006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 msgr2=0x7fdf48196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf48196d60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf38004f60 tx=0x7fdf380048f0 comp rx=0 tx=0).stop 2024-09-15T21:48:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 shutdown_connections 2024-09-15T21:48:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdf3003d710 0x7fdf3003fbd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 --2- 172.21.15.28:0/2368224875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf481015e0 0x7fdf48196d60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 >> 172.21.15.28:0/2368224875 conn(0x7fdf480fcd70 msgr2=0x7fdf480fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 shutdown_connections 2024-09-15T21:48:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:07.356+0000 7fdf4e46d640 1 -- 172.21.15.28:0/2368224875 wait complete. 2024-09-15T21:48:08.211 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-15T21:48:08.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-15T21:48:08.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:08 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2368224875' entity='client.admin' 2024-09-15T21:48:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.353+0000 7f13efc32640 1 -- 172.21.15.28:0/3625015733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 msgr2=0x7f13e8100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.353+0000 7f13efc32640 1 --2- 172.21.15.28:0/3625015733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8100070 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f13d800a990 tx=0x7f13d802b910 comp rx=0 tx=0).stop 2024-09-15T21:48:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.354+0000 7f13efc32640 1 -- 172.21.15.28:0/3625015733 shutdown_connections 2024-09-15T21:48:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.354+0000 7f13efc32640 1 --2- 172.21.15.28:0/3625015733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8100070 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.354+0000 7f13efc32640 1 -- 172.21.15.28:0/3625015733 >> 172.21.15.28:0/3625015733 conn(0x7f13e80fb400 msgr2=0x7f13e80fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.354+0000 7f13efc32640 1 -- 172.21.15.28:0/3625015733 shutdown_connections 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.354+0000 7f13efc32640 1 -- 172.21.15.28:0/3625015733 wait complete. 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.354+0000 7f13efc32640 1 Processor -- start 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.354+0000 7f13efc32640 1 -- start start 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13efc32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13efc32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13e8105500 con 0x7f13e80ffc70 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13ed9ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13ed9ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37820/0 (socket says 172.21.15.28:37820) 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13ed9ce640 1 -- 172.21.15.28:0/3401337084 learned_addr learned my addr 172.21.15.28:0/3401337084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13ed9ce640 1 -- 172.21.15.28:0/3401337084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e8199430 con 0x7f13e80ffc70 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13ed9ce640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8198ef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f13d8031600 tx=0x7f13d8004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13deffd640 1 -- 172.21.15.28:0/3401337084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d8004040 con 0x7f13e80ffc70 2024-09-15T21:48:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13deffd640 1 -- 172.21.15.28:0/3401337084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d80054e0 con 0x7f13e80ffc70 2024-09-15T21:48:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e81996c0 con 0x7f13e80ffc70 2024-09-15T21:48:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.355+0000 7f13deffd640 1 -- 172.21.15.28:0/3401337084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d8003a60 con 0x7f13e80ffc70 2024-09-15T21:48:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.356+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e819c3b0 con 0x7f13e80ffc70 2024-09-15T21:48:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.356+0000 7f13deffd640 1 -- 172.21.15.28:0/3401337084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f13d8039040 con 0x7f13e80ffc70 2024-09-15T21:48:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.356+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e8105c80 con 0x7f13e80ffc70 2024-09-15T21:48:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.357+0000 7f13deffd640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f13d40392a0 0x7f13d403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.357+0000 7f13deffd640 1 -- 172.21.15.28:0/3401337084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f13d8037020 con 0x7f13e80ffc70 2024-09-15T21:48:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.357+0000 7f13ed1cd640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f13d40392a0 0x7f13d403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.357+0000 7f13ed1cd640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f13d40392a0 0x7f13d403b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f13e40098c0 tx=0x7f13e4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.361+0000 7f13deffd640 1 -- 172.21.15.28:0/3401337084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f13d8040030 con 0x7f13e80ffc70 2024-09-15T21:48:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.544+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f13e802d940 con 0x7f13d40392a0 2024-09-15T21:48:09.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.547+0000 7f13deffd640 1 -- 172.21.15.28:0/3401337084 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f13e802d940 con 0x7f13d40392a0 2024-09-15T21:48:09.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f13d40392a0 msgr2=0x7f13d403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f13d40392a0 0x7f13d403b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f13e40098c0 tx=0x7f13e4006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 msgr2=0x7f13e8198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8198ef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f13d8031600 tx=0x7f13d8004950 comp rx=0 tx=0).stop 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 shutdown_connections 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f13d40392a0 0x7f13d403b760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 --2- 172.21.15.28:0/3401337084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e80ffc70 0x7f13e8198ef0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 >> 172.21.15.28:0/3401337084 conn(0x7f13e80fb400 msgr2=0x7f13e80fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 shutdown_connections 2024-09-15T21:48:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:09.549+0000 7f13efc32640 1 -- 172.21.15.28:0/3401337084 wait complete. 2024-09-15T21:48:10.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:10.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:10.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:48:10.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:10.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:10.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:10.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:10.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:10.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:10.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:48:10.297 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-09-15T21:48:10.298 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:48:10.298 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-09-15T21:48:10.318 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:48:10.318 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:10.377 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-09-15T21:48:10.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch host add smithi174 2024-09-15T21:48:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:11.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:11.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-15T21:48:11.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:11.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:11.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:48:11.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:11.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:11.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:10 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 -- 172.21.15.28:0/3254211471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 msgr2=0x7fd0e8106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 --2- 172.21.15.28:0/3254211471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8106940 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009880 tx=0x7fd0d802a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 -- 172.21.15.28:0/3254211471 shutdown_connections 2024-09-15T21:48:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 --2- 172.21.15.28:0/3254211471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8106940 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 -- 172.21.15.28:0/3254211471 >> 172.21.15.28:0/3254211471 conn(0x7fd0e80fc1b0 msgr2=0x7fd0e80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 -- 172.21.15.28:0/3254211471 shutdown_connections 2024-09-15T21:48:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 -- 172.21.15.28:0/3254211471 wait complete. 2024-09-15T21:48:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.113+0000 7fd0ed95c640 1 Processor -- start 2024-09-15T21:48:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0ed95c640 1 -- start start 2024-09-15T21:48:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0ed95c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0ed95c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e810ab90 con 0x7fd0e8106560 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54358/0 (socket says 172.21.15.28:54358) 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0e6ffd640 1 -- 172.21.15.28:0/4179095259 learned_addr learned my addr 172.21.15.28:0/4179095259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0e6ffd640 1 -- 172.21.15.28:0/4179095259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e8197640 con 0x7fd0e8106560 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0e6ffd640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8197100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009230 tx=0x7fd0d8003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.114+0000 7fd0ec95a640 1 -- 172.21.15.28:0/4179095259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0d803f070 con 0x7fd0e8106560 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.115+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e81978d0 con 0x7fd0e8106560 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.115+0000 7fd0ec95a640 1 -- 172.21.15.28:0/4179095259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0d803bc80 con 0x7fd0e8106560 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.115+0000 7fd0ec95a640 1 -- 172.21.15.28:0/4179095259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0d8035030 con 0x7fd0e8106560 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.115+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e819b9d0 con 0x7fd0e8106560 2024-09-15T21:48:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.115+0000 7fd0ec95a640 1 -- 172.21.15.28:0/4179095259 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fd0d8042070 con 0x7fd0e8106560 2024-09-15T21:48:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.116+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e8101bd0 con 0x7fd0e8106560 2024-09-15T21:48:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.116+0000 7fd0ec95a640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd0d40392a0 0x7fd0d403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.116+0000 7fd0ec95a640 1 -- 172.21.15.28:0/4179095259 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd0d806ccd0 con 0x7fd0e8106560 2024-09-15T21:48:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.116+0000 7fd0dffff640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd0d40392a0 0x7fd0d403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.116+0000 7fd0dffff640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd0d40392a0 0x7fd0d403b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00098c0 tx=0x7fd0d0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.120+0000 7fd0ec95a640 1 -- 172.21.15.28:0/4179095259 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd0d8031570 con 0x7fd0e8106560 2024-09-15T21:48:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:11.299+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) -- 0x7fd0e81069c0 con 0x7fd0d40392a0 2024-09-15T21:48:12.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:11 smithi028 ceph-mon[30969]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:13.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:12 smithi028 ceph-mon[30969]: Deploying cephadm binary to smithi174 2024-09-15T21:48:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.103+0000 7fd0ec95a640 1 -- 172.21.15.28:0/4179095259 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd0e81069c0 con 0x7fd0d40392a0 2024-09-15T21:48:15.108 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-09-15T21:48:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd0d40392a0 msgr2=0x7fd0d403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd0d40392a0 0x7fd0d403b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00098c0 tx=0x7fd0d0006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 msgr2=0x7fd0e8197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8197100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009230 tx=0x7fd0d8003d40 comp rx=0 tx=0).stop 2024-09-15T21:48:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 shutdown_connections 2024-09-15T21:48:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd0d40392a0 0x7fd0d403b760 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 --2- 172.21.15.28:0/4179095259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0e8106560 0x7fd0e8197100 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 >> 172.21.15.28:0/4179095259 conn(0x7fd0e80fc1b0 msgr2=0x7fd0e8069ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 shutdown_connections 2024-09-15T21:48:15.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:15.106+0000 7fd0ed95c640 1 -- 172.21.15.28:0/4179095259 wait complete. 2024-09-15T21:48:15.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-15T21:48:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:16.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:16.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:16 smithi028 ceph-mon[30969]: Added host smithi174 2024-09-15T21:48:16.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:16.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:16.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.440+0000 7f841dbec640 1 -- 172.21.15.28:0/4028695511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 msgr2=0x7f8418102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.440+0000 7f841dbec640 1 --2- 172.21.15.28:0/4028695511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f8418102b30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8408009880 tx=0x7f840802a670 comp rx=0 tx=0).stop 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.440+0000 7f841dbec640 1 -- 172.21.15.28:0/4028695511 shutdown_connections 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.440+0000 7f841dbec640 1 --2- 172.21.15.28:0/4028695511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f8418102b30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.440+0000 7f841dbec640 1 -- 172.21.15.28:0/4028695511 >> 172.21.15.28:0/4028695511 conn(0x7f84180fc1b0 msgr2=0x7f84180fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.440+0000 7f841dbec640 1 -- 172.21.15.28:0/4028695511 shutdown_connections 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.440+0000 7f841dbec640 1 -- 172.21.15.28:0/4028695511 wait complete. 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.441+0000 7f841dbec640 1 Processor -- start 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.441+0000 7f841dbec640 1 -- start start 2024-09-15T21:48:16.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f841dbec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f84181003b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f841dbec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f841810bcc0 con 0x7f8418102750 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f84177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f84181003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f84177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f84181003b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54372/0 (socket says 172.21.15.28:54372) 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f84177fe640 1 -- 172.21.15.28:0/3275243009 learned_addr learned my addr 172.21.15.28:0/3275243009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f84177fe640 1 -- 172.21.15.28:0/3275243009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84181022a0 con 0x7f8418102750 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f84177fe640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f84181003b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8408009230 tx=0x7f8408003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f8414ff9640 1 -- 172.21.15.28:0/3275243009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f840803f070 con 0x7f8418102750 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f8414ff9640 1 -- 172.21.15.28:0/3275243009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f840803bc80 con 0x7f8418102750 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f8414ff9640 1 -- 172.21.15.28:0/3275243009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8408035030 con 0x7f8418102750 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84181008f0 con 0x7f8418102750 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.442+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8418100db0 con 0x7f8418102750 2024-09-15T21:48:16.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.443+0000 7f8414ff9640 1 -- 172.21.15.28:0/3275243009 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f8408042070 con 0x7f8418102750 2024-09-15T21:48:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.444+0000 7f8414ff9640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8400039300 0x7f840003b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.443+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f841806a7b0 con 0x7f8418102750 2024-09-15T21:48:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.444+0000 7f8414ff9640 1 -- 172.21.15.28:0/3275243009 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f840806cfb0 con 0x7f8418102750 2024-09-15T21:48:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.444+0000 7f8416ffd640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8400039300 0x7f840003b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.444+0000 7f8416ffd640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8400039300 0x7f840003b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f840c0098c0 tx=0x7f840c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.449+0000 7f8414ff9640 1 -- 172.21.15.28:0/3275243009 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f84080372d0 con 0x7f8418102750 2024-09-15T21:48:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.628+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8418101950 con 0x7f8400039300 2024-09-15T21:48:16.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.629+0000 7f8414ff9640 1 -- 172.21.15.28:0/3275243009 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f8418101950 con 0x7f8400039300 2024-09-15T21:48:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:16.631 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-09-15T21:48:16.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.631+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8400039300 msgr2=0x7f840003b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:16.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.631+0000 7f841dbec640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8400039300 0x7f840003b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f840c0098c0 tx=0x7f840c006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:16.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.631+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 msgr2=0x7f84181003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.631+0000 7f841dbec640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f84181003b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8408009230 tx=0x7f8408003d40 comp rx=0 tx=0).stop 2024-09-15T21:48:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.632+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 shutdown_connections 2024-09-15T21:48:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.632+0000 7f841dbec640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8400039300 0x7f840003b7c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.632+0000 7f841dbec640 1 --2- 172.21.15.28:0/3275243009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8418102750 0x7f84181003b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.632+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 >> 172.21.15.28:0/3275243009 conn(0x7f84180fc1b0 msgr2=0x7f8418103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.632+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 shutdown_connections 2024-09-15T21:48:16.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:16.632+0000 7f841dbec640 1 -- 172.21.15.28:0/3275243009 wait complete. 2024-09-15T21:48:17.198 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-09-15T21:48:17.198 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:48:17.198 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-09-15T21:48:17.216 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:48:17.216 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:17.272 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-09-15T21:48:17.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch host add smithi203 2024-09-15T21:48:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:17.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:17.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:17 smithi028 ceph-mon[30969]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:48:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.926+0000 7f87a9a02640 1 -- 172.21.15.28:0/3237147427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4100520 msgr2=0x7f87a4100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.926+0000 7f87a9a02640 1 --2- 172.21.15.28:0/3237147427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4100520 0x7f87a4100900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8794009880 tx=0x7f879402a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.926+0000 7f87a9a02640 1 -- 172.21.15.28:0/3237147427 shutdown_connections 2024-09-15T21:48:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.926+0000 7f87a9a02640 1 --2- 172.21.15.28:0/3237147427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4100520 0x7f87a4100900 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.926+0000 7f87a9a02640 1 -- 172.21.15.28:0/3237147427 >> 172.21.15.28:0/3237147427 conn(0x7f87a40771d0 msgr2=0x7f87a40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.926+0000 7f87a9a02640 1 -- 172.21.15.28:0/3237147427 shutdown_connections 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.926+0000 7f87a9a02640 1 -- 172.21.15.28:0/3237147427 wait complete. 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.927+0000 7f87a9a02640 1 Processor -- start 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.927+0000 7f87a9a02640 1 -- start start 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.927+0000 7f87a9a02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4196fb0 0x7f87a4197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.927+0000 7f87a9a02640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87a410a970 con 0x7f87a4196fb0 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.927+0000 7f87a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4196fb0 0x7f87a4197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.927+0000 7f87a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4196fb0 0x7f87a4197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54402/0 (socket says 172.21.15.28:54402) 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a2ffd640 1 -- 172.21.15.28:0/80586198 learned_addr learned my addr 172.21.15.28:0/80586198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a2ffd640 1 -- 172.21.15.28:0/80586198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a41978d0 con 0x7f87a4196fb0 2024-09-15T21:48:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a2ffd640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4196fb0 0x7f87a4197390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8794009230 tx=0x7f8794003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a8a00640 1 -- 172.21.15.28:0/80586198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f879403f070 con 0x7f87a4196fb0 2024-09-15T21:48:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a8a00640 1 -- 172.21.15.28:0/80586198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f879403bc80 con 0x7f87a4196fb0 2024-09-15T21:48:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a4197bc0 con 0x7f87a4196fb0 2024-09-15T21:48:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a8a00640 1 -- 172.21.15.28:0/80586198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8794035030 con 0x7f87a4196fb0 2024-09-15T21:48:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.928+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a419ba50 con 0x7f87a4196fb0 2024-09-15T21:48:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.929+0000 7f87a8a00640 1 -- 172.21.15.28:0/80586198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f8794042070 con 0x7f87a4196fb0 2024-09-15T21:48:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.929+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a4101a00 con 0x7f87a4196fb0 2024-09-15T21:48:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.929+0000 7f87a8a00640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f87900392a0 0x7f879003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.929+0000 7f87a8a00640 1 -- 172.21.15.28:0/80586198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f879406cf80 con 0x7f87a4196fb0 2024-09-15T21:48:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.930+0000 7f879bfff640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f87900392a0 0x7f879003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.930+0000 7f879bfff640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f87900392a0 0x7f879003b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f878c0098c0 tx=0x7f878c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:17.934+0000 7f87a8a00640 1 -- 172.21.15.28:0/80586198 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f879404c350 con 0x7f87a4196fb0 2024-09-15T21:48:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:18.120+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) -- 0x7f87a4198310 con 0x7f87900392a0 2024-09-15T21:48:18.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:18 smithi028 ceph-mon[30969]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:18.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:18 smithi028 ceph-mon[30969]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:20.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:20.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:20.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: Deploying cephadm binary to smithi203 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:20.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.177+0000 7f87a8a00640 1 -- 172.21.15.28:0/80586198 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f87a4198310 con 0x7f87900392a0 2024-09-15T21:48:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-09-15T21:48:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f87900392a0 msgr2=0x7f879003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f87900392a0 0x7f879003b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f878c0098c0 tx=0x7f878c006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4196fb0 msgr2=0x7f87a4197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4196fb0 0x7f87a4197390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8794009230 tx=0x7f8794003d40 comp rx=0 tx=0).stop 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 shutdown_connections 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f87900392a0 0x7f879003b760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 --2- 172.21.15.28:0/80586198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87a4196fb0 0x7f87a4197390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.179+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 >> 172.21.15.28:0/80586198 conn(0x7f87a4100520 msgr2=0x7f87a40696b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.180+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 shutdown_connections 2024-09-15T21:48:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:22.180+0000 7f87a9a02640 1 -- 172.21.15.28:0/80586198 wait complete. 2024-09-15T21:48:22.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-15T21:48:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:23.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:23 smithi028 ceph-mon[30969]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:23.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:23.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:23 smithi028 ceph-mon[30969]: Added host smithi203 2024-09-15T21:48:23.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:23.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:23.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 -- 172.21.15.28:0/2525056097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 msgr2=0x7f227c102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 --2- 172.21.15.28:0/2525056097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c102b10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2278009810 tx=0x7f227802a540 comp rx=0 tx=0).stop 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 -- 172.21.15.28:0/2525056097 shutdown_connections 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 --2- 172.21.15.28:0/2525056097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c102b10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 -- 172.21.15.28:0/2525056097 >> 172.21.15.28:0/2525056097 conn(0x7f227c077310 msgr2=0x7f227c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 -- 172.21.15.28:0/2525056097 shutdown_connections 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 -- 172.21.15.28:0/2525056097 wait complete. 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.479+0000 7f22838a6640 1 Processor -- start 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f22838a6640 1 -- start start 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f22838a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c1971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f22838a6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f227c10bc80 con 0x7f227c102730 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f2281642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c1971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f2281642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c1971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52768/0 (socket says 172.21.15.28:52768) 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f2281642640 1 -- 172.21.15.28:0/3103072684 learned_addr learned my addr 172.21.15.28:0/3103072684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f2281642640 1 -- 172.21.15.28:0/3103072684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c1976f0 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.480+0000 7f2281642640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c1971b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2278009930 tx=0x7f2278004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.481+0000 7f22727fc640 1 -- 172.21.15.28:0/3103072684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2278031400 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.481+0000 7f22727fc640 1 -- 172.21.15.28:0/3103072684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22780035f0 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.481+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c197980 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.481+0000 7f22727fc640 1 -- 172.21.15.28:0/3103072684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f227803c440 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.481+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c197de0 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.481+0000 7f22727fc640 1 -- 172.21.15.28:0/3103072684 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f2278039040 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.482+0000 7f22727fc640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f22680392a0 0x7f226803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.482+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227c069f90 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.482+0000 7f22727fc640 1 -- 172.21.15.28:0/3103072684 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f227806d540 con 0x7f227c102730 2024-09-15T21:48:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.482+0000 7f2280e41640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f22680392a0 0x7f226803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.482+0000 7f2280e41640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f22680392a0 0x7f226803b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f226c0098c0 tx=0x7f226c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.487+0000 7f22727fc640 1 -- 172.21.15.28:0/3103072684 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2278040030 con 0x7f227c102730 2024-09-15T21:48:23.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.671+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f227c198540 con 0x7f22680392a0 2024-09-15T21:48:23.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.672+0000 7f22727fc640 1 -- 172.21.15.28:0/3103072684 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f227c198540 con 0x7f22680392a0 2024-09-15T21:48:23.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:23.674 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-09-15T21:48:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f22680392a0 msgr2=0x7f226803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f22680392a0 0x7f226803b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f226c0098c0 tx=0x7f226c006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 msgr2=0x7f227c1971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c1971b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2278009930 tx=0x7f2278004570 comp rx=0 tx=0).stop 2024-09-15T21:48:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 shutdown_connections 2024-09-15T21:48:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f22680392a0 0x7f226803b760 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 --2- 172.21.15.28:0/3103072684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f227c102730 0x7f227c1971b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 >> 172.21.15.28:0/3103072684 conn(0x7f227c077310 msgr2=0x7f227c1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 shutdown_connections 2024-09-15T21:48:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:23.674+0000 7f22838a6640 1 -- 172.21.15.28:0/3103072684 wait complete. 2024-09-15T21:48:24.300 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-15T21:48:24.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-15T21:48:24.553 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:24.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:24 smithi028 ceph-mon[30969]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:24.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:24 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:24.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:24 smithi028 ceph-mon[30969]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:48:24.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.918+0000 7f231ca92640 1 -- 172.21.15.28:0/175022719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 msgr2=0x7f23181069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:24.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.918+0000 7f231ca92640 1 --2- 172.21.15.28:0/175022719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f23181069d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2308009880 tx=0x7f230802a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:24.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.918+0000 7f231ca92640 1 -- 172.21.15.28:0/175022719 shutdown_connections 2024-09-15T21:48:24.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.918+0000 7f231ca92640 1 --2- 172.21.15.28:0/175022719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f23181069d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:24.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.918+0000 7f231ca92640 1 -- 172.21.15.28:0/175022719 >> 172.21.15.28:0/175022719 conn(0x7f23180fb930 msgr2=0x7f23180fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:24.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.918+0000 7f231ca92640 1 -- 172.21.15.28:0/175022719 shutdown_connections 2024-09-15T21:48:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.918+0000 7f231ca92640 1 -- 172.21.15.28:0/175022719 wait complete. 2024-09-15T21:48:24.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.919+0000 7f231ca92640 1 Processor -- start 2024-09-15T21:48:24.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.919+0000 7f231ca92640 1 -- start start 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.919+0000 7f231ca92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f2318197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.919+0000 7f231ca92640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f231810ac00 con 0x7f23181065f0 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f231659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f2318197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f231659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f2318197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52776/0 (socket says 172.21.15.28:52776) 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f231659c640 1 -- 172.21.15.28:0/4263928045 learned_addr learned my addr 172.21.15.28:0/4263928045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f231659c640 1 -- 172.21.15.28:0/4263928045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2318197670 con 0x7f23181065f0 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f231659c640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f2318197130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2308009230 tx=0x7f2308003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f230f7fe640 1 -- 172.21.15.28:0/4263928045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f230803f070 con 0x7f23181065f0 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f230f7fe640 1 -- 172.21.15.28:0/4263928045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f230803bc80 con 0x7f23181065f0 2024-09-15T21:48:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2318197900 con 0x7f23181065f0 2024-09-15T21:48:24.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f230f7fe640 1 -- 172.21.15.28:0/4263928045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2308035030 con 0x7f23181065f0 2024-09-15T21:48:24.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.920+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f231819ba00 con 0x7f23181065f0 2024-09-15T21:48:24.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.921+0000 7f230f7fe640 1 -- 172.21.15.28:0/4263928045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f2308042070 con 0x7f23181065f0 2024-09-15T21:48:24.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.921+0000 7f230f7fe640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f23040392a0 0x7f230403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:24.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.921+0000 7f230f7fe640 1 -- 172.21.15.28:0/4263928045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f230806d530 con 0x7f23181065f0 2024-09-15T21:48:24.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.922+0000 7f230ffff640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f23040392a0 0x7f230403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:24.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.921+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2318101c40 con 0x7f23181065f0 2024-09-15T21:48:24.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.922+0000 7f230ffff640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f23040392a0 0x7f230403b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f23000098c0 tx=0x7f2300006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:24.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:24.926+0000 7f230f7fe640 1 -- 172.21.15.28:0/4263928045 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f230804c350 con 0x7f23181065f0 2024-09-15T21:48:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.105+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f23181069d0 con 0x7f23181065f0 2024-09-15T21:48:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.521+0000 7f230f7fe640 1 -- 172.21.15.28:0/4263928045 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f2308031e20 con 0x7f23181065f0 2024-09-15T21:48:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-09-15T21:48:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.524+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f23040392a0 msgr2=0x7f230403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.524+0000 7f231ca92640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f23040392a0 0x7f230403b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f23000098c0 tx=0x7f2300006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.524+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 msgr2=0x7f2318197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.524+0000 7f231ca92640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f2318197130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2308009230 tx=0x7f2308003d40 comp rx=0 tx=0).stop 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.525+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 shutdown_connections 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.525+0000 7f231ca92640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f23040392a0 0x7f230403b760 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.525+0000 7f231ca92640 1 --2- 172.21.15.28:0/4263928045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23181065f0 0x7f2318197130 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.525+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 >> 172.21.15.28:0/4263928045 conn(0x7f23180fb930 msgr2=0x7f23180ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.525+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 shutdown_connections 2024-09-15T21:48:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:25.525+0000 7f231ca92640 1 -- 172.21.15.28:0/4263928045 wait complete. 2024-09-15T21:48:25.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:25 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4263928045' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-15T21:48:26.191 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-09-15T21:48:26.191 INFO:tasks.cephadm:Adding mon.b on smithi174 2024-09-15T21:48:26.192 INFO:tasks.cephadm:Adding mon.c on smithi203 2024-09-15T21:48:26.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi174:172.21.15.174=b;smithi203:172.21.15.203=c' 2024-09-15T21:48:26.447 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-15T21:48:26.492 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-15T21:48:26.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:26.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4263928045' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-15T21:48:26.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:48:26.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:26 smithi028 ceph-mon[30969]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:48:26.982 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.973+0000 7f250bec8640 1 -- 172.21.15.203:0/2059776999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 msgr2=0x7f2504101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:26.982 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.973+0000 7f250bec8640 1 --2- 172.21.15.203:0/2059776999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504101a00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2500009880 tx=0x7f250002a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:26.982 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.973+0000 7f250bec8640 1 -- 172.21.15.203:0/2059776999 shutdown_connections 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.973+0000 7f250bec8640 1 --2- 172.21.15.203:0/2059776999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504101a00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.973+0000 7f250bec8640 1 -- 172.21.15.203:0/2059776999 >> 172.21.15.203:0/2059776999 conn(0x7f25040fcd70 msgr2=0x7f25040ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.973+0000 7f250bec8640 1 -- 172.21.15.203:0/2059776999 shutdown_connections 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.973+0000 7f250bec8640 1 -- 172.21.15.203:0/2059776999 wait complete. 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.974+0000 7f250bec8640 1 Processor -- start 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.974+0000 7f250bec8640 1 -- start start 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.974+0000 7f250bec8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.974+0000 7f250bec8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2504107710 con 0x7f2504101600 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.974+0000 7f2509c64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.974+0000 7f2509c64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504075eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.203:37598/0 (socket says 172.21.15.203:37598) 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.974+0000 7f2509c64640 1 -- 172.21.15.203:0/3115676402 learned_addr learned my addr 172.21.15.203:0/3115676402 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:48:26.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.975+0000 7f2509c64640 1 -- 172.21.15.203:0/3115676402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25040763f0 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.975+0000 7f2509c64640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504075eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2500004f60 tx=0x7f25000048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.975+0000 7f24faffd640 1 -- 172.21.15.203:0/3115676402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f250003f070 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.975+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25040745e0 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.975+0000 7f24faffd640 1 -- 172.21.15.203:0/3115676402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2500004d40 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.975+0000 7f24faffd640 1 -- 172.21.15.203:0/3115676402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2500035030 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.975+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2504074b20 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.976+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2504107e90 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.976+0000 7f24faffd640 1 -- 172.21.15.203:0/3115676402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f2500042070 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.976+0000 7f24faffd640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f24f00392a0 0x7f24f003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.976+0000 7f24faffd640 1 -- 172.21.15.203:0/3115676402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f250006d040 con 0x7f2504101600 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.977+0000 7f2509463640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f24f00392a0 0x7f24f003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.977+0000 7f2509463640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f24f00392a0 0x7f24f003b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f24f40098c0 tx=0x7f24f4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:26.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:26.982+0000 7f24faffd640 1 -- 172.21.15.203:0/3115676402 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f250004a260 con 0x7f2504101600 2024-09-15T21:48:27.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.173+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi174:172.21.15.174=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}) -- 0x7f25041901d0 con 0x7f24f00392a0 2024-09-15T21:48:27.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.178+0000 7f24faffd640 1 -- 172.21.15.203:0/3115676402 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f25041901d0 con 0x7f24f00392a0 2024-09-15T21:48:27.180 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-09-15T21:48:27.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f24f00392a0 msgr2=0x7f24f003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:27.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f24f00392a0 0x7f24f003b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f24f40098c0 tx=0x7f24f4006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 msgr2=0x7f2504075eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504075eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2500004f60 tx=0x7f25000048f0 comp rx=0 tx=0).stop 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 shutdown_connections 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f24f00392a0 0x7f24f003b760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 --2- 172.21.15.203:0/3115676402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2504101600 0x7f2504075eb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 >> 172.21.15.203:0/3115676402 conn(0x7f25040fcd70 msgr2=0x7f25040fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 shutdown_connections 2024-09-15T21:48:27.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:27.180+0000 7f250bec8640 1 -- 172.21.15.203:0/3115676402 wait complete. 2024-09-15T21:48:27.772 DEBUG:teuthology.orchestra.run.smithi174:mon.b> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.b.service 2024-09-15T21:48:27.774 DEBUG:teuthology.orchestra.run.smithi203:mon.c> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.c.service 2024-09-15T21:48:27.777 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T21:48:27.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi174:172.21.15.174=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi174:172.21.15.174=b;smithi203:172.21.15.203=c;count:3 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:28.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:28.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:28.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:28 smithi028 ceph-mon[30969]: Deploying daemon mon.c on smithi203 2024-09-15T21:48:28.526 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:28.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.937+0000 7f01771be640 1 -- 172.21.15.203:0/1066415048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 msgr2=0x7f0170106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:28.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.937+0000 7f01771be640 1 --2- 172.21.15.203:0/1066415048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170106920 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0164009810 tx=0x7f016402a540 comp rx=0 tx=0).stop 2024-09-15T21:48:28.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.938+0000 7f01771be640 1 -- 172.21.15.203:0/1066415048 shutdown_connections 2024-09-15T21:48:28.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.938+0000 7f01771be640 1 --2- 172.21.15.203:0/1066415048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170106920 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:28.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.938+0000 7f01771be640 1 -- 172.21.15.203:0/1066415048 >> 172.21.15.203:0/1066415048 conn(0x7f0170077310 msgr2=0x7f01700fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:28.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.938+0000 7f01771be640 1 -- 172.21.15.203:0/1066415048 shutdown_connections 2024-09-15T21:48:28.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.938+0000 7f01771be640 1 -- 172.21.15.203:0/1066415048 wait complete. 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.938+0000 7f01771be640 1 Processor -- start 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.938+0000 7f01771be640 1 -- start start 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.939+0000 7f01771be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.939+0000 7f01771be640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f017010ab70 con 0x7f0170106540 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.939+0000 7f0174f5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.939+0000 7f0174f5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.203:47380/0 (socket says 172.21.15.203:47380) 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.939+0000 7f0174f5a640 1 -- 172.21.15.203:0/3063302068 learned_addr learned my addr 172.21.15.203:0/3063302068 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.939+0000 7f0174f5a640 1 -- 172.21.15.203:0/3063302068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01701965a0 con 0x7f0170106540 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.940+0000 7f0174f5a640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170196060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f01640045e0 tx=0x7f0164004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.940+0000 7f016dffb640 1 -- 172.21.15.203:0/3063302068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0164030e70 con 0x7f0170106540 2024-09-15T21:48:28.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.940+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0170196830 con 0x7f0170106540 2024-09-15T21:48:28.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.940+0000 7f016dffb640 1 -- 172.21.15.203:0/3063302068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01640318e0 con 0x7f0170106540 2024-09-15T21:48:28.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.940+0000 7f016dffb640 1 -- 172.21.15.203:0/3063302068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01640049c0 con 0x7f0170106540 2024-09-15T21:48:28.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.940+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0170196c90 con 0x7f0170106540 2024-09-15T21:48:28.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.941+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0170101bb0 con 0x7f0170106540 2024-09-15T21:48:28.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.941+0000 7f016dffb640 1 -- 172.21.15.203:0/3063302068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f0164039040 con 0x7f0170106540 2024-09-15T21:48:28.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.941+0000 7f016dffb640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f01600392a0 0x7f016003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:28.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.941+0000 7f016dffb640 1 -- 172.21.15.203:0/3063302068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f016406d5b0 con 0x7f0170106540 2024-09-15T21:48:28.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.941+0000 7f016ffff640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f01600392a0 0x7f016003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:28.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.942+0000 7f016ffff640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f01600392a0 0x7f016003b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f015c0098c0 tx=0x7f015c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:28.949 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:28.946+0000 7f016dffb640 1 -- 172.21.15.203:0/3063302068 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0164040030 con 0x7f0170106540 2024-09-15T21:48:29.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.180+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0170197420 con 0x7f0170106540 2024-09-15T21:48:29.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.181+0000 7f016dffb640 1 -- 172.21.15.203:0/3063302068 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f016403b3b0 con 0x7f0170106540 2024-09-15T21:48:29.183 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:48:29.183 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","modified":"2024-09-15T21:47:06.538730Z","created":"2024-09-15T21:47:06.538730Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-15T21:48:29.184 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-09-15T21:48:29.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f01600392a0 msgr2=0x7f016003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f01600392a0 0x7f016003b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f015c0098c0 tx=0x7f015c006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 msgr2=0x7f0170196060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170196060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f01640045e0 tx=0x7f0164004610 comp rx=0 tx=0).stop 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 shutdown_connections 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f01600392a0 0x7f016003b760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 --2- 172.21.15.203:0/3063302068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0170106540 0x7f0170196060 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 >> 172.21.15.203:0/3063302068 conn(0x7f0170077310 msgr2=0x7f01700fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 shutdown_connections 2024-09-15T21:48:29.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:29.184+0000 7f01771be640 1 -- 172.21.15.203:0/3063302068 wait complete. 2024-09-15T21:48:29.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:29 smithi028 ceph-mon[30969]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:29.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:29 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/3063302068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:30.904 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T21:48:30.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T21:48:31.182 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:48:31.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:31 smithi028 ceph-mon[30969]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:31.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.808+0000 7f4dc40f0640 1 -- 172.21.15.203:0/3650676228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 msgr2=0x7f4dbc106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:31.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.808+0000 7f4dc40f0640 1 --2- 172.21.15.203:0/3650676228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc106a10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009880 tx=0x7f4db802a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:31.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.808+0000 7f4dc40f0640 1 -- 172.21.15.203:0/3650676228 shutdown_connections 2024-09-15T21:48:31.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.808+0000 7f4dc40f0640 1 --2- 172.21.15.203:0/3650676228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc106a10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:31.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.808+0000 7f4dc40f0640 1 -- 172.21.15.203:0/3650676228 >> 172.21.15.203:0/3650676228 conn(0x7f4dbc0fc220 msgr2=0x7f4dbc0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.808+0000 7f4dc40f0640 1 -- 172.21.15.203:0/3650676228 shutdown_connections 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.808+0000 7f4dc40f0640 1 -- 172.21.15.203:0/3650676228 wait complete. 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc40f0640 1 Processor -- start 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc40f0640 1 -- start start 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc40f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc40f0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc10ac60 con 0x7f4dbc106630 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc1e8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc1e8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.203:47404/0 (socket says 172.21.15.203:47404) 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc1e8c640 1 -- 172.21.15.203:0/474320331 learned_addr learned my addr 172.21.15.203:0/474320331 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.809+0000 7f4dc1e8c640 1 -- 172.21.15.203:0/474320331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dbc198560 con 0x7f4dbc106630 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.810+0000 7f4dc1e8c640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc198020 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009230 tx=0x7f4db8003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:31.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.810+0000 7f4db2ffd640 1 -- 172.21.15.203:0/474320331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4db803f070 con 0x7f4dbc106630 2024-09-15T21:48:31.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.810+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dbc1987f0 con 0x7f4dbc106630 2024-09-15T21:48:31.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.810+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dbc198c50 con 0x7f4dbc106630 2024-09-15T21:48:31.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.810+0000 7f4db2ffd640 1 -- 172.21.15.203:0/474320331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4db803bc80 con 0x7f4dbc106630 2024-09-15T21:48:31.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.810+0000 7f4db2ffd640 1 -- 172.21.15.203:0/474320331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4db8049060 con 0x7f4dbc106630 2024-09-15T21:48:31.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.811+0000 7f4db2ffd640 1 -- 172.21.15.203:0/474320331 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f4db8042070 con 0x7f4dbc106630 2024-09-15T21:48:31.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.811+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da00052c0 con 0x7f4dbc106630 2024-09-15T21:48:31.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.811+0000 7f4db2ffd640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4da8039300 0x7f4da803b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:31.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.811+0000 7f4db2ffd640 1 -- 172.21.15.203:0/474320331 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4db806ccc0 con 0x7f4dbc106630 2024-09-15T21:48:31.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.812+0000 7f4dc168b640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4da8039300 0x7f4da803b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:31.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.812+0000 7f4dc168b640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4da8039300 0x7f4da803b7c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0098c0 tx=0x7f4dac006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:31.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:31.816+0000 7f4db2ffd640 1 -- 172.21.15.203:0/474320331 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4db8043ce0 con 0x7f4dbc106630 2024-09-15T21:48:32.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.071+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4da00055b0 con 0x7f4dbc106630 2024-09-15T21:48:32.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.072+0000 7f4db2ffd640 1 -- 172.21.15.203:0/474320331 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f4db8042360 con 0x7f4dbc106630 2024-09-15T21:48:32.075 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:48:32.076 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","modified":"2024-09-15T21:47:06.538730Z","created":"2024-09-15T21:47:06.538730Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-15T21:48:32.076 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-09-15T21:48:32.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4da8039300 msgr2=0x7f4da803b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:32.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4da8039300 0x7f4da803b7c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0098c0 tx=0x7f4dac006db0 comp rx=0 tx=0).stop 2024-09-15T21:48:32.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 msgr2=0x7f4dbc198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:32.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc198020 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009230 tx=0x7f4db8003d40 comp rx=0 tx=0).stop 2024-09-15T21:48:32.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 shutdown_connections 2024-09-15T21:48:32.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4da8039300 0x7f4da803b7c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:32.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 --2- 172.21.15.203:0/474320331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dbc106630 0x7f4dbc198020 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:32.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 >> 172.21.15.203:0/474320331 conn(0x7f4dbc0fc220 msgr2=0x7f4dbc0fc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:32.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 shutdown_connections 2024-09-15T21:48:32.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:32.074+0000 7f4dc40f0640 1 -- 172.21.15.203:0/474320331 wait complete. 2024-09-15T21:48:32.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:32 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/474320331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:33 smithi203 systemd[1]: Started Ceph mon.c for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:48:33.715 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T21:48:33.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T21:48:33.973 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:48:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:48:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: mon.a calling monitor election 2024-09-15T21:48:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: mon.c calling monitor election 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: monmap epoch 2 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: last_changed 2024-09-15T21:48:33.115883+0000 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: min_mon_release 19 (squid) 2024-09-15T21:48:38.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: election_strategy: 1 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: fsmap 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: mgrmap e13: x(active, since 40s) 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: overall HEALTH_OK 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:38.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: mon.a calling monitor election 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: mon.c calling monitor election 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: monmap epoch 2 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: last_changed 2024-09-15T21:48:33.115883+0000 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: min_mon_release 19 (squid) 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: election_strategy: 1 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:38.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: fsmap 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: mgrmap e13: x(active, since 40s) 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: overall HEALTH_OK 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:38.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:38.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.815+0000 7fdf113af640 1 -- 172.21.15.203:0/920391057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 msgr2=0x7fdef80059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:38.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.815+0000 7fdf113af640 1 --2- 172.21.15.203:0/920391057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 0x7fdef80059c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c102af0 tx=0x7fdefc02c490 comp rx=0 tx=0).stop 2024-09-15T21:48:38.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.816+0000 7fdf113af640 1 -- 172.21.15.203:0/920391057 shutdown_connections 2024-09-15T21:48:38.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.816+0000 7fdf113af640 1 --2- 172.21.15.203:0/920391057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 0x7fdef80059c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:38.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.816+0000 7fdf113af640 1 -- 172.21.15.203:0/920391057 >> 172.21.15.203:0/920391057 conn(0x7fdf0c0fd320 msgr2=0x7fdf0c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:38.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.816+0000 7fdf113af640 1 -- 172.21.15.203:0/920391057 shutdown_connections 2024-09-15T21:48:38.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.816+0000 7fdf113af640 1 -- 172.21.15.203:0/920391057 wait complete. 2024-09-15T21:48:38.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.817+0000 7fdf113af640 1 Processor -- start 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.817+0000 7fdf113af640 1 -- start start 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.817+0000 7fdf113af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 0x7fdf0c1972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.817+0000 7fdf113af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 0x7fdf0c19bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.817+0000 7fdf113af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c109780 con 0x7fdef8003530 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.817+0000 7fdf113af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0c109600 con 0x7fdf0c1977e0 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf027fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 0x7fdf0c19bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf027fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 0x7fdf0c19bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57738/0 (socket says 172.21.15.203:57738) 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf027fc640 1 -- 172.21.15.203:0/1245868982 learned_addr learned my addr 172.21.15.203:0/1245868982 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf0affd640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 0x7fdf0c1972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:38.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf027fc640 1 -- 172.21.15.203:0/1245868982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 msgr2=0x7fdf0c19bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf027fc640 1 -- 172.21.15.203:0/1245868982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 msgr2=0x7fdf0c19bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf027fc640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 0x7fdf0c19bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf027fc640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 0x7fdf0c19bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf0affd640 1 -- 172.21.15.203:0/1245868982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 msgr2=0x7fdf0c19bb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf0affd640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 0x7fdf0c19bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf0affd640 1 -- 172.21.15.203:0/1245868982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0c197e00 con 0x7fdef8003530 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.818+0000 7fdf0affd640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 0x7fdf0c1972a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdefc005f30 tx=0x7fdefc033680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.819+0000 7fdf08ff9640 1 -- 172.21.15.203:0/1245868982 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fdefc038040 con 0x7fdef8003530 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.819+0000 7fdf08ff9640 1 -- 172.21.15.203:0/1245868982 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdefc032e00 con 0x7fdef8003530 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.819+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0c19c0b0 con 0x7fdef8003530 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.819+0000 7fdf08ff9640 1 -- 172.21.15.203:0/1245868982 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fdefc004a00 con 0x7fdef8003530 2024-09-15T21:48:38.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.819+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf0c19c5f0 con 0x7fdef8003530 2024-09-15T21:48:38.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.820+0000 7fdf08ff9640 1 -- 172.21.15.203:0/1245868982 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fdefc043020 con 0x7fdef8003530 2024-09-15T21:48:38.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.820+0000 7fdf08ff9640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdeec039090 0x7fdeec03b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:38.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.820+0000 7fdf08ff9640 1 -- 172.21.15.203:0/1245868982 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdefc06e9a0 con 0x7fdef8003530 2024-09-15T21:48:38.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.820+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf0c102da0 con 0x7fdef8003530 2024-09-15T21:48:38.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.820+0000 7fdf027fc640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdeec039090 0x7fdeec03b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:38.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.821+0000 7fdf027fc640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdeec039090 0x7fdeec03b550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009100 tx=0x7fdef4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:38.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:38.826+0000 7fdf08ff9640 1 -- 172.21.15.203:0/1245868982 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdefc04a020 con 0x7fdef8003530 2024-09-15T21:48:39.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.061+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fdf0c190f60 con 0x7fdef8003530 2024-09-15T21:48:39.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.062+0000 7fdf08ff9640 1 -- 172.21.15.203:0/1245868982 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fdefc04cbc0 con 0x7fdef8003530 2024-09-15T21:48:39.064 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:48:39.064 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","modified":"2024-09-15T21:48:33.115883Z","created":"2024-09-15T21:47:06.538730Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T21:48:39.064 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-09-15T21:48:39.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdeec039090 msgr2=0x7fdeec03b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:39.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdeec039090 0x7fdeec03b550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009100 tx=0x7fdef4008040 comp rx=0 tx=0).stop 2024-09-15T21:48:39.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 msgr2=0x7fdf0c1972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:39.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 0x7fdf0c1972a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdefc005f30 tx=0x7fdefc033680 comp rx=0 tx=0).stop 2024-09-15T21:48:39.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 shutdown_connections 2024-09-15T21:48:39.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdeec039090 0x7fdeec03b550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:39.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf0c1977e0 0x7fdf0c19bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:39.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 --2- 172.21.15.203:0/1245868982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef8003530 0x7fdf0c1972a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:39.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 >> 172.21.15.203:0/1245868982 conn(0x7fdf0c0fd320 msgr2=0x7fdf0c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:39.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 shutdown_connections 2024-09-15T21:48:39.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:39.064+0000 7fdf113af640 1 -- 172.21.15.203:0/1245868982 wait complete. 2024-09-15T21:48:39.367 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:39 smithi203 ceph-mon[35546]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:39.367 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:39 smithi203 ceph-mon[35546]: Deploying daemon mon.b on smithi174 2024-09-15T21:48:39.367 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:39 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/1245868982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:39.367 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:39 smithi028 ceph-mon[30969]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:39 smithi028 ceph-mon[30969]: Deploying daemon mon.b on smithi174 2024-09-15T21:48:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:39 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/1245868982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:40.662 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T21:48:40.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T21:48:40.916 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:48:41.328 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:41 smithi203 ceph-mon[35546]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:41.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 -- 172.21.15.203:0/3465357249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c107660 msgr2=0x7f7f8c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:41.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 --2- 172.21.15.203:0/3465357249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c107660 0x7f7f8c107a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009880 tx=0x7f7f7c02a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 -- 172.21.15.203:0/3465357249 shutdown_connections 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 --2- 172.21.15.203:0/3465357249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c107660 0x7f7f8c107a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 -- 172.21.15.203:0/3465357249 >> 172.21.15.203:0/3465357249 conn(0x7f7f8c0fd320 msgr2=0x7f7f8c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 -- 172.21.15.203:0/3465357249 shutdown_connections 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 -- 172.21.15.203:0/3465357249 wait complete. 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.326+0000 7f7f915b4640 1 Processor -- start 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f915b4640 1 -- start start 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f915b4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 0x7f7f8c1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f915b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c197900 0x7f7f8c19bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:41.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f915b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c10bd20 con 0x7f7f8c197900 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f915b4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8c10bba0 con 0x7f7f8c107660 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f8affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 0x7f7f8c1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f8affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 0x7f7f8c1973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57746/0 (socket says 172.21.15.203:57746) 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f8affd640 1 -- 172.21.15.203:0/3402204918 learned_addr learned my addr 172.21.15.203:0/3402204918 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f8affd640 1 -- 172.21.15.203:0/3402204918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 msgr2=0x7f7f8c1973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f8affd640 1 -- 172.21.15.203:0/3402204918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 msgr2=0x7f7f8c1973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f8affd640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 0x7f7f8c1973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f8affd640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 0x7f7f8c1973c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.327+0000 7f7f827fc640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c197900 0x7f7f8c19bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:41.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f827fc640 1 -- 172.21.15.203:0/3402204918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 msgr2=0x7f7f8c1973c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:48:41.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f827fc640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 0x7f7f8c1973c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:41.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f827fc640 1 -- 172.21.15.203:0/3402204918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8c197f20 con 0x7f7f8c197900 2024-09-15T21:48:41.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f827fc640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c197900 0x7f7f8c19bc90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400b3d0 tx=0x7f7f7400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:41.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f88ff9640 1 -- 172.21.15.203:0/3402204918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7f74011630 con 0x7f7f8c197900 2024-09-15T21:48:41.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f88ff9640 1 -- 172.21.15.203:0/3402204918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f74011cb0 con 0x7f7f8c197900 2024-09-15T21:48:41.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8c19c1d0 con 0x7f7f8c197900 2024-09-15T21:48:41.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f88ff9640 1 -- 172.21.15.203:0/3402204918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7f74010df0 con 0x7f7f8c197900 2024-09-15T21:48:41.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.328+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f8c19c710 con 0x7f7f8c197900 2024-09-15T21:48:41.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.329+0000 7f7f88ff9640 1 -- 172.21.15.203:0/3402204918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f7f74018020 con 0x7f7f8c197900 2024-09-15T21:48:41.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.329+0000 7f7f88ff9640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7f78039040 0x7f7f7803b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:41.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.329+0000 7f7f88ff9640 1 -- 172.21.15.203:0/3402204918 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7f7404bfb0 con 0x7f7f8c197900 2024-09-15T21:48:41.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.329+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f8c102da0 con 0x7f7f8c197900 2024-09-15T21:48:41.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.330+0000 7f7f8affd640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7f78039040 0x7f7f7803b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:41.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.330+0000 7f7f8affd640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7f78039040 0x7f7f7803b500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0096c0 tx=0x7f7f7c005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:41.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.335+0000 7f7f88ff9640 1 -- 172.21.15.203:0/3402204918 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7f7401f020 con 0x7f7f8c197900 2024-09-15T21:48:41.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.566+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7f8c19ca20 con 0x7f7f8c197900 2024-09-15T21:48:41.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.567+0000 7f7f88ff9640 1 -- 172.21.15.203:0/3402204918 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f7f74017500 con 0x7f7f8c197900 2024-09-15T21:48:41.569 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:48:41.569 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","modified":"2024-09-15T21:48:33.115883Z","created":"2024-09-15T21:47:06.538730Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T21:48:41.569 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-09-15T21:48:41.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7f78039040 msgr2=0x7f7f7803b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:41.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7f78039040 0x7f7f7803b500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0096c0 tx=0x7f7f7c005a40 comp rx=0 tx=0).stop 2024-09-15T21:48:41.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c197900 msgr2=0x7f7f8c19bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:41.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c197900 0x7f7f8c19bc90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400b3d0 tx=0x7f7f7400b890 comp rx=0 tx=0).stop 2024-09-15T21:48:41.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 shutdown_connections 2024-09-15T21:48:41.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7f78039040 0x7f7f7803b500 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:41.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f8c197900 0x7f7f8c19bc90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:41.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 --2- 172.21.15.203:0/3402204918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f8c107660 0x7f7f8c1973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:41.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 >> 172.21.15.203:0/3402204918 conn(0x7f7f8c0fd320 msgr2=0x7f7f8c0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:41.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 shutdown_connections 2024-09-15T21:48:41.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:41.569+0000 7f7f915b4640 1 -- 172.21.15.203:0/3402204918 wait complete. 2024-09-15T21:48:41.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:41 smithi028 ceph-mon[30969]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:41.953 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-15T21:48:41.953 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4263928045' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:41.954 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi174:172.21.15.174=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi174:172.21.15.174=b;smithi203:172.21.15.203=c;count:3 2024-09-15T21:48:41.955 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: Deploying daemon mon.c on smithi203 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/3063302068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:41.956 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/474320331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.a calling monitor election 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.c calling monitor election 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:41.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: monmap epoch 2 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: last_changed 2024-09-15T21:48:33.115883+0000 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: min_mon_release 19 (squid) 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: election_strategy: 1 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: fsmap 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mgrmap e13: x(active, since 40s) 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: overall HEALTH_OK 2024-09-15T21:48:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: Deploying daemon mon.b on smithi174 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/1245868982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 ceph-mon[35244]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 bash[35226]: 210357fd4b1e686dab048d3df132f815b418165ecc734f961df06859e3a6425d 2024-09-15T21:48:42.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:41 smithi174 systemd[1]: Started Ceph mon.b for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:48:43.095 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T21:48:43.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T21:48:43.340 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:48:47.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.016+0000 7f5ed69c3640 1 -- 172.21.15.203:0/2119169708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8003530 msgr2=0x7f5eb80059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:47.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.016+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/2119169708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8003530 0x7f5eb80059c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0102af0 tx=0x7f5ec002c150 comp rx=0 tx=0).stop 2024-09-15T21:48:47.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.016+0000 7f5ed69c3640 1 -- 172.21.15.203:0/2119169708 shutdown_connections 2024-09-15T21:48:47.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.016+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/2119169708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8003530 0x7f5eb80059c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:47.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.016+0000 7f5ed69c3640 1 -- 172.21.15.203:0/2119169708 >> 172.21.15.203:0/2119169708 conn(0x7f5ed00fd320 msgr2=0x7f5ed00ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:47.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.016+0000 7f5ed69c3640 1 -- 172.21.15.203:0/2119169708 shutdown_connections 2024-09-15T21:48:47.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.016+0000 7f5ed69c3640 1 -- 172.21.15.203:0/2119169708 wait complete. 2024-09-15T21:48:47.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.017+0000 7f5ed69c3640 1 Processor -- start 2024-09-15T21:48:47.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.017+0000 7f5ed69c3640 1 -- start start 2024-09-15T21:48:47.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.017+0000 7f5ed69c3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 0x7f5ed0197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:47.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.017+0000 7f5ed69c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 0x7f5ed019bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:47.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed69c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed019c110 0x7f5ed019c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed69c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed0109780 con 0x7f5ed019c110 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed69c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed0109600 con 0x7f5ed0197840 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed69c3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed0109900 con 0x7f5eb8003530 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 0x7f5ed0197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 0x7f5ed0197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:57760/0 (socket says 172.21.15.203:57760) 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecffff640 1 -- 172.21.15.203:0/3147486361 learned_addr learned my addr 172.21.15.203:0/3147486361 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed4f60640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed019c110 0x7f5ed019c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecf7fe640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 0x7f5ed019bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecffff640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 msgr2=0x7f5ed0197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T21:48:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecffff640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 msgr2=0x7f5ed0197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecffff640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 0x7f5ed0197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecffff640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 0x7f5ed0197300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecf7fe640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 msgr2=0x7f5ed019bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecf7fe640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 msgr2=0x7f5ed019bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecf7fe640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 0x7f5ed019bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ecf7fe640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 0x7f5ed019bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed4f60640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 msgr2=0x7f5ed0197300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed4f60640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 0x7f5ed0197300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed4f60640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 msgr2=0x7f5ed019bbd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed4f60640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 0x7f5ed019bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:47.023 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.018+0000 7f5ed4f60640 1 -- 172.21.15.203:0/3147486361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed01a3e80 con 0x7f5ed019c110 2024-09-15T21:48:47.024 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.019+0000 7f5ed4f60640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed019c110 0x7f5ed019c5c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800bd50 tx=0x7f5ec800be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:47.024 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.020+0000 7f5ecd7fa640 1 -- 172.21.15.203:0/3147486361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8019070 con 0x7f5ed019c110 2024-09-15T21:48:47.024 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.021+0000 7f5ecd7fa640 1 -- 172.21.15.203:0/3147486361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ec8014920 con 0x7f5ed019c110 2024-09-15T21:48:47.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.021+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed01a40b0 con 0x7f5ed019c110 2024-09-15T21:48:47.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.021+0000 7f5ecd7fa640 1 -- 172.21.15.203:0/3147486361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8012990 con 0x7f5ed019c110 2024-09-15T21:48:47.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.021+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed01a45c0 con 0x7f5ed019c110 2024-09-15T21:48:47.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.022+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed010a8b0 con 0x7f5ed019c110 2024-09-15T21:48:47.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.023+0000 7f5ecd7fa640 1 -- 172.21.15.203:0/3147486361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f5ec801c070 con 0x7f5ed019c110 2024-09-15T21:48:47.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.024+0000 7f5ecd7fa640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5ebc039730 0x7f5ebc03bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:47.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.024+0000 7f5ecd7fa640 1 -- 172.21.15.203:0/3147486361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5ec804c5d0 con 0x7f5ed019c110 2024-09-15T21:48:47.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.024+0000 7f5ecffff640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5ebc039730 0x7f5ebc03bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:47.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.024+0000 7f5ecffff640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5ebc039730 0x7f5ebc03bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0005fd0 tx=0x7f5ec0005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:47.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.028+0000 7f5ecd7fa640 1 -- 172.21.15.203:0/3147486361 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5ec801c360 con 0x7f5ed019c110 2024-09-15T21:48:47.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.269+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5ed002d7c0 con 0x7f5ed019c110 2024-09-15T21:48:47.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.269+0000 7f5ecd7fa640 1 -- 172.21.15.203:0/3147486361 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f5ec801de40 con 0x7f5ed019c110 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: mon.c calling monitor election 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: mon.a calling monitor election 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: monmap epoch 3 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: last_changed 2024-09-15T21:48:41.956241+0000 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: min_mon_release 19 (squid) 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: election_strategy: 1 2024-09-15T21:48:47.273 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: 2: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: fsmap 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: mgrmap e13: x(active, since 49s) 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: overall HEALTH_OK 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.275 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:48:47.275 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","modified":"2024-09-15T21:48:41.956241Z","created":"2024-09-15T21:47:06.538730Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T21:48:47.275 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-09-15T21:48:47.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5ebc039730 msgr2=0x7f5ebc03bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5ebc039730 0x7f5ebc03bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0005fd0 tx=0x7f5ec0005d30 comp rx=0 tx=0).stop 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed019c110 msgr2=0x7f5ed019c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed019c110 0x7f5ed019c5c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800bd50 tx=0x7f5ec800be40 comp rx=0 tx=0).stop 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 shutdown_connections 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5ebc039730 0x7f5ebc03bbf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed019c110 0x7f5ed019c5c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ed0197840 0x7f5ed019bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 --2- 172.21.15.203:0/3147486361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5eb8003530 0x7f5ed0197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 >> 172.21.15.203:0/3147486361 conn(0x7f5ed00fd320 msgr2=0x7f5ed00ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 shutdown_connections 2024-09-15T21:48:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:47.273+0000 7f5ed69c3640 1 -- 172.21.15.203:0/3147486361 wait complete. 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: mon.c calling monitor election 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: mon.a calling monitor election 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: monmap epoch 3 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: last_changed 2024-09-15T21:48:41.956241+0000 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: min_mon_release 19 (squid) 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: election_strategy: 1 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: 2: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: fsmap 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: mgrmap e13: x(active, since 49s) 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: overall HEALTH_OK 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:47.891 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-15T21:48:47.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-15T21:48:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:48 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/3147486361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:48 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/3147486361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.533+0000 7f76000de640 1 -- 172.21.15.28:0/4072946778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f8107660 msgr2=0x7f75f8107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.533+0000 7f76000de640 1 --2- 172.21.15.28:0/4072946778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f8107660 0x7f75f8107a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009880 tx=0x7f75f402a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.533+0000 7f76000de640 1 -- 172.21.15.28:0/4072946778 shutdown_connections 2024-09-15T21:48:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.533+0000 7f76000de640 1 --2- 172.21.15.28:0/4072946778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f8107660 0x7f75f8107a40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.533+0000 7f76000de640 1 -- 172.21.15.28:0/4072946778 >> 172.21.15.28:0/4072946778 conn(0x7f75f80fd290 msgr2=0x7f75f80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.534+0000 7f76000de640 1 -- 172.21.15.28:0/4072946778 shutdown_connections 2024-09-15T21:48:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.534+0000 7f76000de640 1 -- 172.21.15.28:0/4072946778 wait complete. 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.534+0000 7f76000de640 1 Processor -- start 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.534+0000 7f76000de640 1 -- start start 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.534+0000 7f76000de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8107660 0x7f75f8197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f76000de640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75f8197860 0x7f75f819bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f76000de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f819c130 0x7f75f819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f76000de640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75f810bc90 con 0x7f75f819c130 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f76000de640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f75f810bb10 con 0x7f75f8197860 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f76000de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75f810be10 con 0x7f75f8107660 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fde7a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8107660 0x7f75f8197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fe67b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f819c130 0x7f75f819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fde7a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8107660 0x7f75f8197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:34180/0 (socket says 172.21.15.28:34180) 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fde7a640 1 -- 172.21.15.28:0/3020754881 learned_addr learned my addr 172.21.15.28:0/3020754881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fd679640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75f8197860 0x7f75f819bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fe67b640 1 -- 172.21.15.28:0/3020754881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8107660 msgr2=0x7f75f8197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fe67b640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8107660 0x7f75f8197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fe67b640 1 -- 172.21.15.28:0/3020754881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75f8197860 msgr2=0x7f75f819bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fe67b640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75f8197860 0x7f75f819bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fe67b640 1 -- 172.21.15.28:0/3020754881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f81a3ea0 con 0x7f75f819c130 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fde7a640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8107660 0x7f75f8197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.535+0000 7f75fe67b640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f819c130 0x7f75f819c5e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75f000ff40 tx=0x7f75f000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.536+0000 7f75eeffd640 1 -- 172.21.15.28:0/3020754881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f000d0d0 con 0x7f75f819c130 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.536+0000 7f75eeffd640 1 -- 172.21.15.28:0/3020754881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f000bb80 con 0x7f75f819c130 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.536+0000 7f75eeffd640 1 -- 172.21.15.28:0/3020754881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0018870 con 0x7f75f819c130 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.536+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f8074420 con 0x7f75f819c130 2024-09-15T21:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.536+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f8074960 con 0x7f75f819c130 2024-09-15T21:48:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.537+0000 7f75eeffd640 1 -- 172.21.15.28:0/3020754881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f75f000b6a0 con 0x7f75f819c130 2024-09-15T21:48:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.538+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f8107ac0 con 0x7f75f819c130 2024-09-15T21:48:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.538+0000 7f75eeffd640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75e4039730 0x7f75e403bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.538+0000 7f75eeffd640 1 -- 172.21.15.28:0/3020754881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f75f004fee0 con 0x7f75f819c130 2024-09-15T21:48:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.538+0000 7f75fde7a640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75e4039730 0x7f75e403bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.538+0000 7f75fde7a640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75e4039730 0x7f75e403bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f75f40096c0 tx=0x7f75f402aad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.542+0000 7f75eeffd640 1 -- 172.21.15.28:0/3020754881 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f75f000bde0 con 0x7f75f819c130 2024-09-15T21:48:48.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.720+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f75f8074d60 con 0x7f75f819c130 2024-09-15T21:48:48.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.721+0000 7f75eeffd640 1 -- 172.21.15.28:0/3020754881 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f75f001a080 con 0x7f75f819c130 2024-09-15T21:48:48.723 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:48.723 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-15T21:48:48.723 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:48.723 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-15T21:48:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75e4039730 msgr2=0x7f75e403bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75e4039730 0x7f75e403bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f75f40096c0 tx=0x7f75f402aad0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f819c130 msgr2=0x7f75f819c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f819c130 0x7f75f819c5e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75f000ff40 tx=0x7f75f000ff70 comp rx=0 tx=0).stop 2024-09-15T21:48:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 shutdown_connections 2024-09-15T21:48:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75e4039730 0x7f75e403bbf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f819c130 0x7f75f819c5e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75f8197860 0x7f75f819bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 --2- 172.21.15.28:0/3020754881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8107660 0x7f75f8197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 >> 172.21.15.28:0/3020754881 conn(0x7f75f80fd290 msgr2=0x7f75f80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 shutdown_connections 2024-09-15T21:48:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:48.723+0000 7f76000de640 1 -- 172.21.15.28:0/3020754881 wait complete. 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: mon.c calling monitor election 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: mon.a calling monitor election 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: monmap epoch 3 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: last_changed 2024-09-15T21:48:41.956241+0000 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: min_mon_release 19 (squid) 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: election_strategy: 1 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:49.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: 2: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: fsmap 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: mgrmap e13: x(active, since 49s) 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: overall HEALTH_OK 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/3147486361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.428 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-15T21:48:49.428 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:48:49.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T21:48:49.457 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:48:49.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:49.524 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:48:49.524 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T21:48:49.552 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:48:49.552 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:49.616 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:48:49.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T21:48:49.643 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:48:49.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:48:49.709 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-09-15T21:48:49.709 INFO:tasks.cephadm:Adding mgr.y on smithi174 2024-09-15T21:48:49.709 INFO:tasks.cephadm:Adding mgr.z on smithi203 2024-09-15T21:48:49.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi174=y;smithi203=z' 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: mon.b calling monitor election 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: mon.b calling monitor election 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: mon.c calling monitor election 2024-09-15T21:48:49.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: mon.a calling monitor election 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: monmap epoch 3 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: last_changed 2024-09-15T21:48:41.956241+0000 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: min_mon_release 19 (squid) 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: election_strategy: 1 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: 2: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: fsmap 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:49.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: mgrmap e13: x(active, since 51s) 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: overall HEALTH_OK 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T21:48:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:49.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:49 smithi028 ceph-mon[30969]: Reconfiguring daemon mon.a on smithi028 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: mon.b calling monitor election 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:49.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: mon.b calling monitor election 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: mon.c calling monitor election 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: mon.a calling monitor election 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: monmap epoch 3 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: last_changed 2024-09-15T21:48:41.956241+0000 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: min_mon_release 19 (squid) 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: election_strategy: 1 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: 2: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-15T21:48:49.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: fsmap 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: mgrmap e13: x(active, since 51s) 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: overall HEALTH_OK 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:49.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T21:48:49.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:49.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:49.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:49.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:49 smithi203 ceph-mon[35546]: Reconfiguring daemon mon.a on smithi028 2024-09-15T21:48:49.984 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:48:50.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: mon.b calling monitor election 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: mon.b calling monitor election 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: mon.c calling monitor election 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: mon.a calling monitor election 2024-09-15T21:48:50.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: monmap epoch 3 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: last_changed 2024-09-15T21:48:41.956241+0000 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: created 2024-09-15T21:47:06.538730+0000 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: min_mon_release 19 (squid) 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: election_strategy: 1 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: 2: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: fsmap 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: mgrmap e13: x(active, since 51s) 2024-09-15T21:48:50.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: overall HEALTH_OK 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T21:48:50.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:50.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:50.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:50.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:49 smithi174 ceph-mon[35244]: Reconfiguring daemon mon.a on smithi028 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 -- 172.21.15.203:0/2102392073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37d01005f0 msgr2=0x7f37d01009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 --2- 172.21.15.203:0/2102392073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37d01005f0 0x7f37d01009f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009880 tx=0x7f37c402a5d0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 -- 172.21.15.203:0/2102392073 shutdown_connections 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 --2- 172.21.15.203:0/2102392073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37d01005f0 0x7f37d01009f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 -- 172.21.15.203:0/2102392073 >> 172.21.15.203:0/2102392073 conn(0x7f37d00fbda0 msgr2=0x7f37d00fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 -- 172.21.15.203:0/2102392073 shutdown_connections 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 -- 172.21.15.203:0/2102392073 wait complete. 2024-09-15T21:48:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 Processor -- start 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.454+0000 7f37d6d99640 1 -- start start 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d6d99640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d01005f0 0x7f37d0195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d6d99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d0196420 0x7f37d019b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d6d99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37d01968a0 0x7f37d019da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d6d99640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01066e0 con 0x7f37d01968a0 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d6d99640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0106560 con 0x7f37d0196420 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d6d99640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0106860 con 0x7f37d01005f0 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d01005f0 0x7f37d0195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d01005f0 0x7f37d0195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:36924/0 (socket says 172.21.15.203:36924) 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 -- 172.21.15.203:0/784623932 learned_addr learned my addr 172.21.15.203:0/784623932 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37cffff640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d0196420 0x7f37d019b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:50.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 -- 172.21.15.203:0/784623932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d0196420 msgr2=0x7f37d019b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d0196420 0x7f37d019b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 -- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37d01968a0 msgr2=0x7f37d019da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37d01968a0 0x7f37d019da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 -- 172.21.15.203:0/784623932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d019e2e0 con 0x7f37d01005f0 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37cffff640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d0196420 0x7f37d019b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d4b35640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d01005f0 0x7f37d0195ee0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f37c4031610 tx=0x7f37c4004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37cdffb640 1 -- 172.21.15.203:0/784623932 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4046070 con 0x7f37d01005f0 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37cdffb640 1 -- 172.21.15.203:0/784623932 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37c4003ce0 con 0x7f37d01005f0 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d006a430 con 0x7f37d01005f0 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.455+0000 7f37cdffb640 1 -- 172.21.15.203:0/784623932 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4035050 con 0x7f37d01005f0 2024-09-15T21:48:50.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.456+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d006a990 con 0x7f37d01005f0 2024-09-15T21:48:50.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.456+0000 7f37cdffb640 1 -- 172.21.15.203:0/784623932 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f37c4049070 con 0x7f37d01005f0 2024-09-15T21:48:50.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.457+0000 7f37cdffb640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f37c0039730 0x7f37c003bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:50.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.457+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d0100a70 con 0x7f37d01005f0 2024-09-15T21:48:50.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.457+0000 7f37cdffb640 1 -- 172.21.15.203:0/784623932 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f37c406e690 con 0x7f37d01005f0 2024-09-15T21:48:50.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.457+0000 7f37cffff640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f37c0039730 0x7f37c003bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:50.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.457+0000 7f37cffff640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f37c0039730 0x7f37c003bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37d0197400 tx=0x7f37bc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:50.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.462+0000 7f37cdffb640 1 -- 172.21.15.203:0/784623932 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f37c404bce0 con 0x7f37d01005f0 2024-09-15T21:48:50.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.641+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi174=y;smithi203=z", "target": ["mon-mgr", ""]}) -- 0x7f37d006abf0 con 0x7f37c0039730 2024-09-15T21:48:50.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.646+0000 7f37cdffb640 1 -- 172.21.15.203:0/784623932 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f37d006abf0 con 0x7f37c0039730 2024-09-15T21:48:50.648 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f37c0039730 msgr2=0x7f37c003bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f37c0039730 0x7f37c003bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f37d0197400 tx=0x7f37bc008040 comp rx=0 tx=0).stop 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d01005f0 msgr2=0x7f37d0195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d01005f0 0x7f37d0195ee0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f37c4031610 tx=0x7f37c4004340 comp rx=0 tx=0).stop 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 shutdown_connections 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f37c0039730 0x7f37c003bbf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37d01968a0 0x7f37d019da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37d0196420 0x7f37d019b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 --2- 172.21.15.203:0/784623932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d01005f0 0x7f37d0195ee0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:50.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 >> 172.21.15.203:0/784623932 conn(0x7f37d00fbda0 msgr2=0x7f37d00fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:50.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 shutdown_connections 2024-09-15T21:48:50.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:48:50.648+0000 7f37d6d99640 1 -- 172.21.15.203:0/784623932 wait complete. 2024-09-15T21:48:50.933 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:50.933 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:50.933 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.933 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.933 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:50.934 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:50.934 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:50.934 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:50.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:50.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:50.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:50.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:50.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:50.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:51.156 DEBUG:teuthology.orchestra.run.smithi174:mgr.y> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.y.service 2024-09-15T21:48:51.158 DEBUG:teuthology.orchestra.run.smithi203:mgr.z> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.z.service 2024-09-15T21:48:51.161 INFO:tasks.cephadm:Deploying OSDs... 2024-09-15T21:48:51.161 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:48:51.161 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:51.180 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T21:48:51.180 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-09-15T21:48:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T21:48:51.237 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:51.237 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T21:48:51.238 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:51.238 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:51.238 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-15 21:48:08.039455288 +0000 2024-09-15T21:48:51.238 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-15 21:43:41.691313288 +0000 2024-09-15T21:48:51.238 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-15 21:43:41.691313288 +0000 2024-09-15T21:48:51.238 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-15 21:43:41.691313288 +0000 2024-09-15T21:48:51.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T21:48:51.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-15T21:48:51.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-15T21:48:51.308 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000123755 s, 4.1 MB/s 2024-09-15T21:48:51.309 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T21:48:51.367 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-09-15T21:48:51.425 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T21:48:51.426 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-15T21:48:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:51.426 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-15 21:48:08.075454293 +0000 2024-09-15T21:48:51.427 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-15 21:43:42.033303820 +0000 2024-09-15T21:48:51.427 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-15 21:43:42.033303820 +0000 2024-09-15T21:48:51.427 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-15 21:43:42.033303820 +0000 2024-09-15T21:48:51.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T21:48:51.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-15T21:48:51.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-15T21:48:51.493 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000106021 s, 4.8 MB/s 2024-09-15T21:48:51.494 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T21:48:51.553 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-09-15T21:48:51.611 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T21:48:51.611 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-15T21:48:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-15 21:48:08.111453297 +0000 2024-09-15T21:48:51.611 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-15 21:43:42.342295266 +0000 2024-09-15T21:48:51.612 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-15 21:43:42.342295266 +0000 2024-09-15T21:48:51.612 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-15 21:43:42.342295266 +0000 2024-09-15T21:48:51.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T21:48:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-15T21:48:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-15T21:48:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000116638 s, 4.4 MB/s 2024-09-15T21:48:51.681 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T21:48:51.740 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-09-15T21:48:51.798 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T21:48:51.798 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:51.798 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-15T21:48:51.798 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:51.798 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:51.799 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-15 21:48:08.147452302 +0000 2024-09-15T21:48:51.799 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-15 21:43:42.673286103 +0000 2024-09-15T21:48:51.799 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-15 21:43:42.673286103 +0000 2024-09-15T21:48:51.799 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-15 21:43:42.673286103 +0000 2024-09-15T21:48:51.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T21:48:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-15T21:48:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-15T21:48:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000115946 s, 4.4 MB/s 2024-09-15T21:48:51.867 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T21:48:51.925 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T21:48:51.925 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T21:48:51.935 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:51 smithi174 ceph-mon[35244]: Reconfiguring mon.b (monmap changed)... 2024-09-15T21:48:51.935 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:51 smithi174 ceph-mon[35244]: Reconfiguring daemon mon.b on smithi174 2024-09-15T21:48:51.935 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:51 smithi174 ceph-mon[35244]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi174=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:51.935 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:51 smithi174 ceph-mon[35244]: Saving service mgr spec with placement smithi028=x;smithi174=y;smithi203=z;count:3 2024-09-15T21:48:51.942 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T21:48:51.942 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-09-15T21:48:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:51 smithi203 ceph-mon[35546]: Reconfiguring mon.b (monmap changed)... 2024-09-15T21:48:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:51 smithi203 ceph-mon[35546]: Reconfiguring daemon mon.b on smithi174 2024-09-15T21:48:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:51 smithi203 ceph-mon[35546]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi174=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:51 smithi203 ceph-mon[35546]: Saving service mgr spec with placement smithi028=x;smithi174=y;smithi203=z;count:3 2024-09-15T21:48:51.998 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T21:48:51.998 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:51.999 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-15T21:48:51.999 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:51.999 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:51.999 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 21:48:41.411861843 +0000 2024-09-15T21:48:51.999 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 21:43:56.479500225 +0000 2024-09-15T21:48:51.999 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 21:43:56.479500225 +0000 2024-09-15T21:48:51.999 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 21:43:56.479500225 +0000 2024-09-15T21:48:51.999 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T21:48:52.065 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T21:48:52.065 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T21:48:52.065 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000249722 s, 2.1 MB/s 2024-09-15T21:48:52.065 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T21:48:52.122 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-09-15T21:48:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:51 smithi028 ceph-mon[30969]: Reconfiguring mon.b (monmap changed)... 2024-09-15T21:48:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:51 smithi028 ceph-mon[30969]: Reconfiguring daemon mon.b on smithi174 2024-09-15T21:48:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:51 smithi028 ceph-mon[30969]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi174=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:51 smithi028 ceph-mon[30969]: Saving service mgr spec with placement smithi028=x;smithi174=y;smithi203=z;count:3 2024-09-15T21:48:52.177 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 21:48:41.412861828 +0000 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 21:43:56.803495735 +0000 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 21:43:56.803495735 +0000 2024-09-15T21:48:52.178 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 21:43:56.803495735 +0000 2024-09-15T21:48:52.178 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T21:48:52.241 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T21:48:52.241 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T21:48:52.241 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000231909 s, 2.2 MB/s 2024-09-15T21:48:52.242 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T21:48:52.299 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-09-15T21:48:52.354 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T21:48:52.354 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:52.355 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-15T21:48:52.355 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:52.355 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:52.355 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 21:48:41.412861828 +0000 2024-09-15T21:48:52.355 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 21:43:57.134491148 +0000 2024-09-15T21:48:52.355 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 21:43:57.134491148 +0000 2024-09-15T21:48:52.355 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 21:43:57.134491148 +0000 2024-09-15T21:48:52.355 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T21:48:52.417 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T21:48:52.447 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T21:48:52.447 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000188952 s, 2.7 MB/s 2024-09-15T21:48:52.448 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T21:48:52.505 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 21:48:41.413861814 +0000 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 21:43:57.453486728 +0000 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 21:43:57.453486728 +0000 2024-09-15T21:48:52.561 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 21:43:57.453486728 +0000 2024-09-15T21:48:52.562 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T21:48:52.623 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T21:48:52.623 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T21:48:52.624 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000184135 s, 2.8 MB/s 2024-09-15T21:48:52.625 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T21:48:52.681 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T21:48:52.681 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T21:48:52.698 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T21:48:52.698 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-09-15T21:48:52.755 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T21:48:52.755 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:52.755 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-15T21:48:52.755 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:52.755 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:52.755 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 21:48:31.981986647 +0000 2024-09-15T21:48:52.756 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 21:43:56.902223156 +0000 2024-09-15T21:48:52.756 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 21:43:56.902223156 +0000 2024-09-15T21:48:52.756 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-15 21:43:56.902223156 +0000 2024-09-15T21:48:52.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T21:48:52.820 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T21:48:52.820 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T21:48:52.820 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000267185 s, 1.9 MB/s 2024-09-15T21:48:52.821 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T21:48:52.878 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 21:48:31.982986623 +0000 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 21:43:57.228215186 +0000 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 21:43:57.228215186 +0000 2024-09-15T21:48:52.934 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-15 21:43:57.228215186 +0000 2024-09-15T21:48:52.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T21:48:52.997 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T21:48:52.997 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T21:48:52.997 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000200606 s, 2.6 MB/s 2024-09-15T21:48:52.998 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T21:48:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: Reconfiguring mon.c (monmap changed)... 2024-09-15T21:48:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:53.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:53.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:52 smithi174 ceph-mon[35244]: Reconfiguring daemon mon.c on smithi203 2024-09-15T21:48:53.055 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-09-15T21:48:53.110 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T21:48:53.110 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:53.110 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-09-15T21:48:53.110 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:53.110 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:53.111 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 21:48:31.982986623 +0000 2024-09-15T21:48:53.111 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 21:43:57.544207460 +0000 2024-09-15T21:48:53.111 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 21:43:57.544207460 +0000 2024-09-15T21:48:53.111 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-15 21:43:57.544207460 +0000 2024-09-15T21:48:53.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T21:48:53.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:53.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:53.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:53.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: Reconfiguring mon.c (monmap changed)... 2024-09-15T21:48:53.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:53.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:53.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:53.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:52 smithi028 ceph-mon[30969]: Reconfiguring daemon mon.c on smithi203 2024-09-15T21:48:53.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:53.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:53.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:53.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: Reconfiguring mon.c (monmap changed)... 2024-09-15T21:48:53.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T21:48:53.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T21:48:53.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:53.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:52 smithi203 ceph-mon[35546]: Reconfiguring daemon mon.c on smithi203 2024-09-15T21:48:53.173 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T21:48:53.173 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T21:48:53.174 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000208823 s, 2.5 MB/s 2024-09-15T21:48:53.175 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T21:48:53.232 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-09-15T21:48:53.287 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 21:48:31.983986599 +0000 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 21:43:57.862199686 +0000 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 21:43:57.862199686 +0000 2024-09-15T21:48:53.288 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-15 21:43:57.862199686 +0000 2024-09-15T21:48:53.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T21:48:53.353 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T21:48:53.353 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T21:48:53.353 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000209061 s, 2.4 MB/s 2024-09-15T21:48:53.354 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T21:48:53.410 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-09-15T21:48:53.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T21:48:53.673 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T21:48:54.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:48:54.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:54.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:54 smithi028 ceph-mon[30969]: Deploying daemon mgr.z on smithi203 2024-09-15T21:48:54.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:54.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:54.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:48:54.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:48:54.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T21:48:54.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:48:54.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:54.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:54 smithi174 ceph-mon[35244]: Deploying daemon mgr.z on smithi203 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:48:54.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T21:48:54.543 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:48:54.543 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:54.543 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:54 smithi203 ceph-mon[35546]: Deploying daemon mgr.z on smithi203 2024-09-15T21:48:55.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 systemd[1]: Starting Ceph mgr.z for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:48:55.474 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:55 smithi203 ceph-mon[35546]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:55.474 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 podman[37339]: 2024-09-15 21:48:55.120482518 +0000 UTC m=+0.015901337 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:48:55.474 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 podman[37339]: 2024-09-15 21:48:55.225990091 +0000 UTC m=+0.121408904 container create 26da5c3b0f95d7e4106d5441fc018cfc17c829ae14bd1a66ab74213183408ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-15T21:48:55.474 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 podman[37339]: 2024-09-15 21:48:55.445120422 +0000 UTC m=+0.340539237 container init 26da5c3b0f95d7e4106d5441fc018cfc17c829ae14bd1a66ab74213183408ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-15T21:48:55.474 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 podman[37339]: 2024-09-15 21:48:55.447987245 +0000 UTC m=+0.343406060 container start 26da5c3b0f95d7e4106d5441fc018cfc17c829ae14bd1a66ab74213183408ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T21:48:55.474 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 ceph-mgr[37357]: -- 172.21.15.203:0/831314083 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bce82810e0 con 0x55bce825e800 2024-09-15T21:48:55.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:55 smithi174 ceph-mon[35244]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:55.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:55 smithi028 ceph-mon[30969]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:55.785 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 bash[37339]: 26da5c3b0f95d7e4106d5441fc018cfc17c829ae14bd1a66ab74213183408ead 2024-09-15T21:48:55.785 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 systemd[1]: Started Ceph mgr.z for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:48:55.785 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:55.577+0000 7f7b6ba19040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:48:55.785 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:55.637+0000 7f7b6ba19040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:48:55.785 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:55.782+0000 7f7b6ba19040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:48:55.998 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:48:56.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-09-15T21:48:56.136 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:55.848+0000 7f7b6ba19040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:48:56.136 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:55.994+0000 7f7b6ba19040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:48:56.136 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:56.134+0000 7f7b6ba19040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:48:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:48:56.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:56.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:48:56.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T21:48:56.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:48:56.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:56.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:56 smithi028 ceph-mon[30969]: Deploying daemon mgr.y on smithi174 2024-09-15T21:48:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:48:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T21:48:56.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:48:56.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:56.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-mon[35546]: Deploying daemon mgr.y on smithi174 2024-09-15T21:48:56.719 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:56.450+0000 7f7b6ba19040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:48:56.719 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:56.510+0000 7f7b6ba19040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:48:56.719 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:56 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:56.695+0000 7f7b6ba19040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 -- 172.21.15.28:0/1850887636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 msgr2=0x7f6d5c103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 --2- 172.21.15.28:0/1850887636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c103280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009ca0 tx=0x7f6d5402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 -- 172.21.15.28:0/1850887636 shutdown_connections 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 --2- 172.21.15.28:0/1850887636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d5c1037c0 0x7f6d5c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 --2- 172.21.15.28:0/1850887636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c103280 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 --2- 172.21.15.28:0/1850887636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d5c101c00 0x7f6d5c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 -- 172.21.15.28:0/1850887636 >> 172.21.15.28:0/1850887636 conn(0x7f6d5c0fd390 msgr2=0x7f6d5c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 -- 172.21.15.28:0/1850887636 shutdown_connections 2024-09-15T21:48:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 -- 172.21.15.28:0/1850887636 wait complete. 2024-09-15T21:48:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.790+0000 7f6d611a1640 1 Processor -- start 2024-09-15T21:48:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d611a1640 1 -- start start 2024-09-15T21:48:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d611a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d5c101c00 0x7f6d5c075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d611a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d611a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d5c1037c0 0x7f6d5c074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d611a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c10d880 con 0x7f6d5c102e00 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d611a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c10d700 con 0x7f6d5c101c00 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d611a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5c10da00 con 0x7f6d5c1037c0 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c0745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37758/0 (socket says 172.21.15.28:37758) 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 -- 172.21.15.28:0/552646602 learned_addr learned my addr 172.21.15.28:0/552646602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5b59e640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d5c1037c0 0x7f6d5c074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5ad9d640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d5c101c00 0x7f6d5c075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 -- 172.21.15.28:0/552646602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d5c1037c0 msgr2=0x7f6d5c074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d5c1037c0 0x7f6d5c074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 -- 172.21.15.28:0/552646602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d5c101c00 msgr2=0x7f6d5c075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d5c101c00 0x7f6d5c075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.791+0000 7f6d5a59c640 1 -- 172.21.15.28:0/552646602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5c0753e0 con 0x7f6d5c102e00 2024-09-15T21:48:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.792+0000 7f6d5ad9d640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d5c101c00 0x7f6d5c075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.792+0000 7f6d5a59c640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c0745e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6d54031a40 tx=0x7f6d54031ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.792+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5402faa0 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.792+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d5402fc40 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.792+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5c1a1fd0 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.792+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5403a790 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.792+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5c1a2480 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.793+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f6d5403a950 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.793+0000 7f6d43fff640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6d440394d0 0x7f6d4403b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.793+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5c108fb0 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.793+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6d5406dc90 con 0x7f6d5c102e00 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.793+0000 7f6d5ad9d640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6d440394d0 0x7f6d4403b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:48:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.794+0000 7f6d5ad9d640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6d440394d0 0x7f6d4403b990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0058c0 tx=0x7f6d4c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:48:56.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.798+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6d5403f1e0 con 0x7f6d5c102e00 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:48:56.950 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:56.951 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:56 smithi174 ceph-mon[35244]: Deploying daemon mgr.y on smithi174 2024-09-15T21:48:56.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:48:56.982+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6d5c062630 con 0x7f6d440394d0 2024-09-15T21:48:57.510 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 systemd[1]: Starting Ceph mgr.y for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:48:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:57 smithi028 ceph-mon[30969]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:48:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:48:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:57.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:57 smithi174 ceph-mon[35244]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:57.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:48:57.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:48:57.857 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:57.857 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 podman[36590]: 2024-09-15 21:48:57.495116388 +0000 UTC m=+0.015852355 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:48:57.858 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 podman[36590]: 2024-09-15 21:48:57.609230689 +0000 UTC m=+0.129966659 container create 539fe850ed94c0f77d00967966b124d01d33e21eabc78a0d3b9e6af3dcf295d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-15T21:48:57.858 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 podman[36590]: 2024-09-15 21:48:57.828074386 +0000 UTC m=+0.348810344 container init 539fe850ed94c0f77d00967966b124d01d33e21eabc78a0d3b9e6af3dcf295d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-15T21:48:57.858 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 podman[36590]: 2024-09-15 21:48:57.830691114 +0000 UTC m=+0.351427072 container start 539fe850ed94c0f77d00967966b124d01d33e21eabc78a0d3b9e6af3dcf295d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e) 2024-09-15T21:48:57.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:57 smithi203 ceph-mon[35546]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:48:57.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:48:57.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:48:57.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:48:57.969 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:57 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:57.587+0000 7f7b6ba19040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:48:57.970 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:57 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:57.656+0000 7f7b6ba19040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:48:58.179 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 ceph-mgr[36610]: -- 172.21.15.174:0/1284820825 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55921fc790e0 con 0x55921fc57400 2024-09-15T21:48:58.179 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 bash[36590]: 539fe850ed94c0f77d00967966b124d01d33e21eabc78a0d3b9e6af3dcf295d3 2024-09-15T21:48:58.180 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 systemd[1]: Started Ceph mgr.y for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:48:58.180 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:57 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:57.960+0000 7f2ab688f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:48:58.180 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:58.019+0000 7f2ab688f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:48:58.510 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:58.177+0000 7f2ab688f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:48:58.510 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:58.246+0000 7f2ab688f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:48:58.510 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:58.410+0000 7f2ab688f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:48:58.670 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:58.403+0000 7f7b6ba19040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:48:58.670 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:58.530+0000 7f7b6ba19040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:48:58.817 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:58.562+0000 7f2ab688f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:48:58.968 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:58.676+0000 7f7b6ba19040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:48:58.968 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:58.739+0000 7f7b6ba19040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:48:58.968 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:58.896+0000 7f7b6ba19040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:48:59.148 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:58.891+0000 7f2ab688f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:48:59.148 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:58.953+0000 7f2ab688f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:48:59.148 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-mon[35244]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:59.148 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.148 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.148 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.148 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.149 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:48:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:59.264 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-mon[35546]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:59.264 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.264 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.264 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.264 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.264 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:59.265 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:58.966+0000 7f7b6ba19040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:48:59.265 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:59.028+0000 7f7b6ba19040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:48:59.265 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:59.196+0000 7f7b6ba19040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:48:59.265 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:59.261+0000 7f7b6ba19040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:48:59.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:58 smithi028 ceph-mon[30969]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:48:59.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:48:59.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:48:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:48:59.510 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:48:59 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:48:59.146+0000 7f2ab688f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:48:59.535 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:59.331+0000 7f7b6ba19040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:48:59.535 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:59.416+0000 7f7b6ba19040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:48:59.535 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:59.475+0000 7f7b6ba19040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:48:59.535 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:48:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:48:59.533+0000 7f7b6ba19040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:49:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: Reconfiguring daemon mgr.x on smithi028 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2544375009' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5066bae-2a3f-4b91-9da9-39d341d3257e"}]: dispatch 2024-09-15T21:49:00.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2544375009' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5066bae-2a3f-4b91-9da9-39d341d3257e"}]': finished 2024-09-15T21:49:00.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T21:49:00.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:00.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:00 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1542800165' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:00.468 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:49:00.314+0000 7f7b6ba19040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:49:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: Reconfiguring daemon mgr.x on smithi028 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2544375009' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5066bae-2a3f-4b91-9da9-39d341d3257e"}]: dispatch 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2544375009' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5066bae-2a3f-4b91-9da9-39d341d3257e"}]': finished 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:00 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1542800165' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:00.510 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:00.029+0000 7f2ab688f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:49:00.510 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:00.095+0000 7f2ab688f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:49:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: Reconfiguring daemon mgr.x on smithi028 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2544375009' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5066bae-2a3f-4b91-9da9-39d341d3257e"}]: dispatch 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2544375009' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5066bae-2a3f-4b91-9da9-39d341d3257e"}]': finished 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T21:49:00.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:00.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1542800165' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:01.069 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:00.811+0000 7f2ab688f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:49:01.069 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:00 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:00.925+0000 7f2ab688f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:49:01.069 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.067+0000 7f2ab688f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:49:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:01.171+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 14) ==== 92408+0+0 (secure 0 0 0) 0x7f6d5403b920 con 0x7f6d5c102e00 2024-09-15T21:49:01.348 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.127+0000 7f2ab688f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.277+0000 7f2ab688f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.344+0000 7f2ab688f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-mon[35244]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-mon[35244]: Standby manager daemon z started 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T21:49:01.349 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:49:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:01 smithi028 ceph-mon[30969]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:01 smithi028 ceph-mon[30969]: Standby manager daemon z started 2024-09-15T21:49:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:01 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T21:49:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:01 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:49:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:01 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T21:49:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:01 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:49:01.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:01 smithi203 ceph-mon[35546]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:01.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:01 smithi203 ceph-mon[35546]: Standby manager daemon z started 2024-09-15T21:49:01.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:01 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T21:49:01.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:01 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:49:01.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:01 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T21:49:01.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:01 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/421773872' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:49:01.625 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.405+0000 7f2ab688f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:49:01.625 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.560+0000 7f2ab688f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:49:01.625 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.623+0000 7f2ab688f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:49:01.891 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.690+0000 7f2ab688f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:49:01.891 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.771+0000 7f2ab688f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:49:01.891 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.830+0000 7f2ab688f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:49:01.891 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:01.889+0000 7f2ab688f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:49:02.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:02 smithi203 ceph-mon[35546]: mgrmap e14: x(active, since 63s), standbys: z 2024-09-15T21:49:02.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:02 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T21:49:02.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:02 smithi174 ceph-mon[35244]: mgrmap e14: x(active, since 63s), standbys: z 2024-09-15T21:49:02.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:02 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T21:49:02.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:02 smithi028 ceph-mon[30969]: mgrmap e14: x(active, since 63s), standbys: z 2024-09-15T21:49:02.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:02 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T21:49:03.010 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:49:02 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:49:02.673+0000 7f2ab688f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:49:03.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:03 smithi203 ceph-mon[35546]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:03.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:03 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:03.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:03 smithi203 ceph-mon[35546]: Standby manager daemon y started 2024-09-15T21:49:03.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:03 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T21:49:03.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:03 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:49:03.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:03 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T21:49:03.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:03 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:49:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:03.504+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f6d540453e0 con 0x7f6d5c102e00 2024-09-15T21:49:03.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:03 smithi028 ceph-mon[30969]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:03.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:03 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:03.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:03 smithi028 ceph-mon[30969]: Standby manager daemon y started 2024-09-15T21:49:03.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:03 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T21:49:03.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:03 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:49:03.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:03 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T21:49:03.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:03 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:49:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:03 smithi174 ceph-mon[35244]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:03 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:03 smithi174 ceph-mon[35244]: Standby manager daemon y started 2024-09-15T21:49:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:03 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T21:49:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:03 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:49:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:03 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T21:49:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:03 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/1665381004' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:49:04.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:04 smithi028 ceph-mon[30969]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:04.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:04 smithi028 ceph-mon[30969]: mgrmap e15: x(active, since 66s), standbys: z, y 2024-09-15T21:49:04.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T21:49:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:04 smithi174 ceph-mon[35244]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:04 smithi174 ceph-mon[35244]: mgrmap e15: x(active, since 66s), standbys: z, y 2024-09-15T21:49:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T21:49:04.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:04 smithi203 ceph-mon[35546]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:04.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:04 smithi203 ceph-mon[35546]: mgrmap e15: x(active, since 66s), standbys: z, y 2024-09-15T21:49:04.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T21:49:07.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:07 smithi028 ceph-mon[30969]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:07.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:07.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:07.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:07.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:07 smithi203 ceph-mon[35546]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:07.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:07.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:07.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:07 smithi174 ceph-mon[35244]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:09.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:09 smithi028 ceph-mon[30969]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:09.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T21:49:09.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:09 smithi203 ceph-mon[35546]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:09 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T21:49:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:09 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:09 smithi174 ceph-mon[35244]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:09 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T21:49:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:09 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:10.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:10 smithi028 ceph-mon[30969]: Deploying daemon osd.0 on smithi028 2024-09-15T21:49:10.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:10 smithi203 ceph-mon[35546]: Deploying daemon osd.0 on smithi028 2024-09-15T21:49:10.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:10 smithi174 ceph-mon[35244]: Deploying daemon osd.0 on smithi028 2024-09-15T21:49:11.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:11 smithi028 ceph-mon[30969]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:11.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:11 smithi203 ceph-mon[35546]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:11.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:11 smithi174 ceph-mon[35244]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:13.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:13 smithi028 ceph-mon[30969]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:13.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:13 smithi203 ceph-mon[35546]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:13.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:13 smithi174 ceph-mon[35244]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:15.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:15 smithi028 ceph-mon[30969]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:15.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:15.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:15.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:15 smithi203 ceph-mon[35546]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:15 smithi174 ceph-mon[35244]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:17 smithi028 ceph-mon[30969]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:17.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:17.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:17.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:17 smithi203 ceph-mon[35546]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:17.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:17.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:17.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:17 smithi174 ceph-mon[35244]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:17.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:17.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:17.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:17.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:18.837 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-09-15T21:49:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.835+0000 7f6d43fff640 1 -- 172.21.15.28:0/552646602 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6d5c062630 con 0x7f6d440394d0 2024-09-15T21:49:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6d440394d0 msgr2=0x7f6d4403b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6d440394d0 0x7f6d4403b990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0058c0 tx=0x7f6d4c005850 comp rx=0 tx=0).stop 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 msgr2=0x7f6d5c0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c0745e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6d54031a40 tx=0x7f6d54031ec0 comp rx=0 tx=0).stop 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 shutdown_connections 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6d440394d0 0x7f6d4403b990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d5c1037c0 0x7f6d5c074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5c102e00 0x7f6d5c0745e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 --2- 172.21.15.28:0/552646602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d5c101c00 0x7f6d5c075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.837+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 >> 172.21.15.28:0/552646602 conn(0x7f6d5c0fd390 msgr2=0x7f6d5c106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.838+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 shutdown_connections 2024-09-15T21:49:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:49:18.838+0000 7f6d611a1640 1 -- 172.21.15.28:0/552646602 wait complete. 2024-09-15T21:49:19.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:19 smithi028 ceph-mon[30969]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:19.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:19.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:19.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:19.441 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.0.service 2024-09-15T21:49:19.444 INFO:tasks.cephadm:Deploying osd.1 on smithi174 with /dev/vg_nvme/lv_4... 2024-09-15T21:49:19.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T21:49:19.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:19 smithi203 ceph-mon[35546]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:19.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:19.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:19.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:19 smithi174 ceph-mon[35244]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:19.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:19.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:19.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:19.683 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:49:20.767 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:49:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0[44848]: 2024-09-15T21:49:20.479+0000 7f73f4d05640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:49:21.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:21 smithi028 ceph-mon[30969]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:21.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:21 smithi028 ceph-mon[30969]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T21:49:21.446 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:21 smithi174 ceph-mon[35244]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:21.446 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:21 smithi174 ceph-mon[35244]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T21:49:21.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:21 smithi203 ceph-mon[35546]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:21.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:21 smithi203 ceph-mon[35546]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T21:49:22.064 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:49:22.091 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-09-15T21:49:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:22 smithi028 ceph-mon[30969]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T21:49:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:22 smithi028 ceph-mon[30969]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T21:49:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:22 smithi028 ceph-mon[30969]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:49:22.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:22.280 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:49:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0[44848]: 2024-09-15T21:49:22.152+0000 7f73f0c88640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T21:49:22.281 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:49:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0[44848]: 2024-09-15T21:49:22.155+0000 7f73ecab0640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:49:22.363 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:22 smithi174 ceph-mon[35244]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T21:49:22.363 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:22 smithi174 ceph-mon[35244]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T21:49:22.363 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:22 smithi174 ceph-mon[35244]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:49:22.363 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:22.382 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:49:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:22 smithi203 ceph-mon[35546]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T21:49:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:22 smithi203 ceph-mon[35546]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T21:49:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:22 smithi203 ceph-mon[35546]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:49:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:22.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 -- 172.21.15.174:0/1392762664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b8100b10 msgr2=0x7f75b8100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:22.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/1392762664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b8100b10 0x7f75b8100f10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009900 tx=0x7f75b402a410 comp rx=0 tx=0).stop 2024-09-15T21:49:22.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 -- 172.21.15.174:0/1392762664 shutdown_connections 2024-09-15T21:49:22.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/1392762664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b81026d0 0x7f75b8108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:22.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/1392762664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75b8101d10 0x7f75b8102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:22.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/1392762664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b8100b10 0x7f75b8100f10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 -- 172.21.15.174:0/1392762664 >> 172.21.15.174:0/1392762664 conn(0x7f75b80fc2a0 msgr2=0x7f75b80fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 -- 172.21.15.174:0/1392762664 shutdown_connections 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 -- 172.21.15.174:0/1392762664 wait complete. 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 Processor -- start 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.855+0000 7f75bfb0a640 1 -- start start 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bfb0a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75b8100b10 0x7f75b8104aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bfb0a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b8101d10 0x7f75b81030f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bfb0a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b81026d0 0x7f75b8103630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bfb0a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75b810c7e0 con 0x7f75b8101d10 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bfb0a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f75b810c660 con 0x7f75b81026d0 2024-09-15T21:49:22.865 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bfb0a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75b810c960 con 0x7f75b8100b10 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b81026d0 0x7f75b8103630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bd8a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75b8100b10 0x7f75b8104aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b81026d0 0x7f75b8103630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:45462/0 (socket says 172.21.15.174:45462) 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 -- 172.21.15.174:0/4212766275 learned_addr learned my addr 172.21.15.174:0/4212766275 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bd0a5640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b8101d10 0x7f75b81030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 -- 172.21.15.174:0/4212766275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75b8100b10 msgr2=0x7f75b8104aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75b8100b10 0x7f75b8104aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 -- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b8101d10 msgr2=0x7f75b81030f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b8101d10 0x7f75b81030f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 -- 172.21.15.174:0/4212766275 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b8103ef0 con 0x7f75b81026d0 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bd0a5640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b8101d10 0x7f75b81030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:49:22.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75be0a7640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b81026d0 0x7f75b8103630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009fd0 tx=0x7f75b000b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75aeffd640 1 -- 172.21.15.174:0/4212766275 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0017070 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75aeffd640 1 -- 172.21.15.174:0/4212766275 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b0003e50 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75aeffd640 1 -- 172.21.15.174:0/4212766275 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b00106c0 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.856+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b81a2140 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.857+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b81a2600 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.857+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b81a2830 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.859+0000 7f75aeffd640 1 -- 172.21.15.174:0/4212766275 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f75b0003970 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.859+0000 7f75aeffd640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75a40a31b0 0x7f75a40a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.860+0000 7f75aeffd640 1 -- 172.21.15.174:0/4212766275 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1487+0+0 (secure 0 0 0) 0x7f75b00cda30 con 0x7f75b81026d0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.860+0000 7f75bd8a6640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75a40a31b0 0x7f75a40a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:22.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.860+0000 7f75bd8a6640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75a40a31b0 0x7f75a40a5670 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75b402aa00 tx=0x7f75b40315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:49:22.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:22.864+0000 7f75aeffd640 1 -- 172.21.15.174:0/4212766275 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f75b009c2e0 con 0x7f75b81026d0 2024-09-15T21:49:23.042 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:23.040+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f75b806af10 con 0x7f75a40a31b0 2024-09-15T21:49:23.374 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi028 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.482 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi028 to 17092M 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:49:23.483 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi028 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.484 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi028 to 17092M 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:49:23.485 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi028 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi028 to 17092M 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:49:23.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:24.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:24 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:49:24.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:24 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:49:24.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:24 smithi028 ceph-mon[30969]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:49:24.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:24 smithi028 ceph-mon[30969]: osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] boot 2024-09-15T21:49:24.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:24 smithi028 ceph-mon[30969]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T21:49:24.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:24 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:24.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:24 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:49:24.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:24 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:49:24.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:24 smithi203 ceph-mon[35546]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:49:24.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:24 smithi203 ceph-mon[35546]: osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] boot 2024-09-15T21:49:24.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:24 smithi203 ceph-mon[35546]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T21:49:24.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:24 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:24.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:24 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:49:24.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:24 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:49:24.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:24 smithi174 ceph-mon[35244]: from='client.24134 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:49:24.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:24 smithi174 ceph-mon[35244]: osd.0 [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] boot 2024-09-15T21:49:24.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:24 smithi174 ceph-mon[35244]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T21:49:24.470 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:24 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:49:25.164 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:25 smithi174 ceph-mon[35244]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:25.164 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:25 smithi174 ceph-mon[35244]: osdmap e9: 1 total, 1 up, 1 in 2024-09-15T21:49:25.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:25 smithi028 ceph-mon[30969]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:25.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:25 smithi028 ceph-mon[30969]: osdmap e9: 1 total, 1 up, 1 in 2024-09-15T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:25 smithi203 ceph-mon[35546]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:25 smithi203 ceph-mon[35546]: osdmap e9: 1 total, 1 up, 1 in 2024-09-15T21:49:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:26 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/3989116736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]: dispatch 2024-09-15T21:49:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:26 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]: dispatch 2024-09-15T21:49:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:26 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]': finished 2024-09-15T21:49:26.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:26 smithi028 ceph-mon[30969]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T21:49:26.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:26.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:26 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/3728527552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:26 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/3989116736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]: dispatch 2024-09-15T21:49:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:26 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]: dispatch 2024-09-15T21:49:26.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:26 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]': finished 2024-09-15T21:49:26.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:26 smithi203 ceph-mon[35546]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T21:49:26.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:26.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:26 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/3728527552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:26 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/3989116736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]: dispatch 2024-09-15T21:49:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:26 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]: dispatch 2024-09-15T21:49:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:26 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b85e5570-ddd1-4e02-858e-948a4517801e"}]': finished 2024-09-15T21:49:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:26 smithi174 ceph-mon[35244]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T21:49:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:26.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:26 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/3728527552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:27.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:27 smithi028 ceph-mon[30969]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:27.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:27 smithi203 ceph-mon[35546]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:27 smithi174 ceph-mon[35244]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:29.633 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:29 smithi174 ceph-mon[35244]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:29.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:29 smithi028 ceph-mon[30969]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:29.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:29 smithi203 ceph-mon[35546]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:31.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:31 smithi028 ceph-mon[30969]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:31.424 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:31 smithi174 ceph-mon[35244]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:31.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:31 smithi203 ceph-mon[35546]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:33.577 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:33 smithi174 ceph-mon[35244]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:33.577 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:33 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T21:49:33.577 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:33 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:33 smithi028 ceph-mon[30969]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:33 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T21:49:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:33 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:33.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:33 smithi203 ceph-mon[35546]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:33.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:33 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T21:49:33.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:33 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:34.509 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:34 smithi174 ceph-mon[35244]: Deploying daemon osd.1 on smithi174 2024-09-15T21:49:34.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:34 smithi028 ceph-mon[30969]: Deploying daemon osd.1 on smithi174 2024-09-15T21:49:34.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:34 smithi203 ceph-mon[35546]: Deploying daemon osd.1 on smithi174 2024-09-15T21:49:35.630 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:35 smithi174 ceph-mon[35244]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:35 smithi028 ceph-mon[30969]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:35.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:35 smithi203 ceph-mon[35546]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:37.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:37 smithi028 ceph-mon[30969]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:37.687 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:37 smithi174 ceph-mon[35244]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:37.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:37 smithi203 ceph-mon[35546]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:39.590 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:39 smithi174 ceph-mon[35244]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:39.590 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:39.590 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:39.590 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:39 smithi028 ceph-mon[30969]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:39 smithi203 ceph-mon[35546]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:41.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:41 smithi174 ceph-mon[35244]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:41.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:41 smithi028 ceph-mon[30969]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:41 smithi203 ceph-mon[35546]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:42.653 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.650+0000 7f75aeffd640 1 -- 172.21.15.174:0/4212766275 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f75b806af10 con 0x7f75a40a31b0 2024-09-15T21:49:42.654 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 1 on host 'smithi174' 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75a40a31b0 msgr2=0x7f75a40a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75a40a31b0 0x7f75a40a5670 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f75b402aa00 tx=0x7f75b40315b0 comp rx=0 tx=0).stop 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b81026d0 msgr2=0x7f75b8103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b81026d0 0x7f75b8103630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009fd0 tx=0x7f75b000b260 comp rx=0 tx=0).stop 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 shutdown_connections 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f75a40a31b0 0x7f75a40a5670 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75b81026d0 0x7f75b8103630 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:42.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b8101d10 0x7f75b81030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:42.657 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 --2- 172.21.15.174:0/4212766275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75b8100b10 0x7f75b8104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:42.657 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 >> 172.21.15.174:0/4212766275 conn(0x7f75b80fc2a0 msgr2=0x7f75b806a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:49:42.657 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 shutdown_connections 2024-09-15T21:49:42.657 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:49:42.653+0000 7f75bfb0a640 1 -- 172.21.15.174:0/4212766275 wait complete. 2024-09-15T21:49:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:43.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:43.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:43.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:43.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:49:43.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:43.355 DEBUG:teuthology.orchestra.run.smithi174:osd.1> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.1.service 2024-09-15T21:49:43.358 INFO:tasks.cephadm:Deploying osd.2 on smithi203 with /dev/vg_nvme/lv_4... 2024-09-15T21:49:43.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T21:49:43.613 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:49:44.983 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:44 smithi203 ceph-mon[35546]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:45.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:44 smithi174 ceph-mon[35244]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:45.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:44 smithi028 ceph-mon[30969]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:45.760 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:49:45 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1[39711]: 2024-09-15T21:49:45.470+0000 7fe0954e8640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:49:46.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:45 smithi203 ceph-mon[35546]: from='osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T21:49:46.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:45 smithi203 ceph-mon[35546]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T21:49:46.065 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:49:46.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-09-15T21:49:46.091 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:45 smithi174 ceph-mon[35244]: from='osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T21:49:46.091 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:45 smithi174 ceph-mon[35244]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T21:49:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:45 smithi028 ceph-mon[30969]: from='osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T21:49:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:45 smithi028 ceph-mon[30969]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T21:49:46.334 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:49:46.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 -- 172.21.15.203:0/2455535655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5581026d0 msgr2=0x7ff558108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:46.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 --2- 172.21.15.203:0/2455535655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5581026d0 0x7ff558108f60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff550009ca0 tx=0x7ff55002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:49:46.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 -- 172.21.15.203:0/2455535655 shutdown_connections 2024-09-15T21:49:46.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 --2- 172.21.15.203:0/2455535655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5581026d0 0x7ff558108f60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:46.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 --2- 172.21.15.203:0/2455535655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff558101d10 0x7ff558102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 --2- 172.21.15.203:0/2455535655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff558100b10 0x7ff558100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 -- 172.21.15.203:0/2455535655 >> 172.21.15.203:0/2455535655 conn(0x7ff5580fc2a0 msgr2=0x7ff5580fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 -- 172.21.15.203:0/2455535655 shutdown_connections 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.755+0000 7ff55f049640 1 -- 172.21.15.203:0/2455535655 wait complete. 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 Processor -- start 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 -- start start 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff558100b10 0x7ff55806b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff558101d10 0x7ff55819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5581026d0 0x7ff55819f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff55810c770 con 0x7ff5581026d0 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff55810c5f0 con 0x7ff558100b10 2024-09-15T21:49:46.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55f049640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff55810c8f0 con 0x7ff558101d10 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff558101d10 0x7ff55819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff558101d10 0x7ff55819cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48944/0 (socket says 172.21.15.203:48944) 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 -- 172.21.15.203:0/2532296502 learned_addr learned my addr 172.21.15.203:0/2532296502 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55cde5640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff558100b10 0x7ff55806b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff55d5e6640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5581026d0 0x7ff55819f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 -- 172.21.15.203:0/2532296502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff558100b10 msgr2=0x7ff55806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff558100b10 0x7ff55806b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 -- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5581026d0 msgr2=0x7ff55819f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5581026d0 0x7ff55819f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.756+0000 7ff557fff640 1 -- 172.21.15.203:0/2532296502 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55819fb20 con 0x7ff558101d10 2024-09-15T21:49:46.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.757+0000 7ff557fff640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff558101d10 0x7ff55819cbf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff54400d950 tx=0x7ff54400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:49:46.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.757+0000 7ff555ffb640 1 -- 172.21.15.203:0/2532296502 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544017030 con 0x7ff558101d10 2024-09-15T21:49:46.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.757+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55819fdb0 con 0x7ff558101d10 2024-09-15T21:49:46.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.757+0000 7ff555ffb640 1 -- 172.21.15.203:0/2532296502 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff544004020 con 0x7ff558101d10 2024-09-15T21:49:46.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.757+0000 7ff555ffb640 1 -- 172.21.15.203:0/2532296502 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5440047a0 con 0x7ff558101d10 2024-09-15T21:49:46.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.757+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5581a0260 con 0x7ff558101d10 2024-09-15T21:49:46.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.758+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff558107eb0 con 0x7ff558101d10 2024-09-15T21:49:46.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.759+0000 7ff555ffb640 1 -- 172.21.15.203:0/2532296502 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7ff5440041d0 con 0x7ff558101d10 2024-09-15T21:49:46.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.759+0000 7ff555ffb640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5480a31b0 0x7ff5480a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:49:46.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.760+0000 7ff555ffb640 1 -- 172.21.15.203:0/2532296502 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 1903+0+0 (secure 0 0 0) 0x7ff5440cd8c0 con 0x7ff558101d10 2024-09-15T21:49:46.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.760+0000 7ff55cde5640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5480a31b0 0x7ff5480a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:49:46.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.760+0000 7ff55cde5640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5480a31b0 0x7ff5480a5670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff558101b70 tx=0x7ff54c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:49:46.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.764+0000 7ff555ffb640 1 -- 172.21.15.203:0/2532296502 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff54409bfd0 con 0x7ff558101d10 2024-09-15T21:49:46.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:49:46.939+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff558062630 con 0x7ff5480a31b0 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi174 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:49:47.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi174 to 17088M 2024-09-15T21:49:47.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:47.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:47.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:47.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T21:49:47.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:49:47.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T21:49:47.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:47.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi174 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi174 to 17088M 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:47.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.262 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1[39711]: 2024-09-15T21:49:46.784+0000 7fe09146b640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T21:49:47.262 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1[39711]: 2024-09-15T21:49:46.786+0000 7fe08d293640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:49:47.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:47.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T21:49:47.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:49:47.262 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi174 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi174 to 17088M 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:49:47.263 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:49:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:49:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: osdmap e12: 2 total, 1 up, 2 in 2024-09-15T21:49:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:49:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:49:47.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:49:47.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: osdmap e12: 2 total, 1 up, 2 in 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:49:48.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:49:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: osdmap e12: 2 total, 1 up, 2 in 2024-09-15T21:49:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:49:48.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:49:48.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:49:48.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:47 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:49.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:48 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:49:49.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:48 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:49:49.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:48 smithi028 ceph-mon[30969]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:49.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:48 smithi028 ceph-mon[30969]: osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] boot 2024-09-15T21:49:49.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:48 smithi028 ceph-mon[30969]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T21:49:49.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:49.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:48 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:49:49.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:48 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:49:49.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:48 smithi203 ceph-mon[35546]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:49.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:48 smithi203 ceph-mon[35546]: osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] boot 2024-09-15T21:49:49.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:48 smithi203 ceph-mon[35546]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T21:49:49.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:49.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:48 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:49:49.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:48 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:49:49.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:48 smithi174 ceph-mon[35244]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-15T21:49:49.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:48 smithi174 ceph-mon[35244]: osd.1 [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] boot 2024-09-15T21:49:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:48 smithi174 ceph-mon[35244]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T21:49:49.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:49:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:49 smithi028 ceph-mon[30969]: osdmap e14: 2 total, 2 up, 2 in 2024-09-15T21:49:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:49 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/2248925918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]: dispatch 2024-09-15T21:49:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:49 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]: dispatch 2024-09-15T21:49:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:49 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]': finished 2024-09-15T21:49:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:49 smithi028 ceph-mon[30969]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T21:49:50.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:49:50.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:49 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/917187222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:49 smithi203 ceph-mon[35546]: osdmap e14: 2 total, 2 up, 2 in 2024-09-15T21:49:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:49 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/2248925918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]: dispatch 2024-09-15T21:49:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:49 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]: dispatch 2024-09-15T21:49:50.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:49 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]': finished 2024-09-15T21:49:50.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:49 smithi203 ceph-mon[35546]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T21:49:50.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:49:50.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:49 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/917187222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:49 smithi174 ceph-mon[35244]: osdmap e14: 2 total, 2 up, 2 in 2024-09-15T21:49:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:49 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/2248925918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]: dispatch 2024-09-15T21:49:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:49 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]: dispatch 2024-09-15T21:49:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:49 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41e4d0e4-7b72-4ad8-9c2e-a260d638777f"}]': finished 2024-09-15T21:49:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:49 smithi174 ceph-mon[35244]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T21:49:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:49:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:49 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/917187222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:49:51.495 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:51 smithi174 ceph-mon[35244]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:51.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:51 smithi028 ceph-mon[30969]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:51 smithi203 ceph-mon[35546]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:53.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:53 smithi174 ceph-mon[35244]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:53.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:53 smithi028 ceph-mon[30969]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:53 smithi203 ceph-mon[35546]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:55.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:55 smithi203 ceph-mon[35546]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:55.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:55 smithi174 ceph-mon[35244]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:55.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:55 smithi028 ceph-mon[30969]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:57 smithi203 ceph-mon[35546]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:57 smithi174 ceph-mon[35244]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:57.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:57 smithi028 ceph-mon[30969]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:58.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T21:49:58.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:58.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T21:49:58.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:58.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T21:49:58.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:49:59.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:59 smithi174 ceph-mon[35244]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:59.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:49:59 smithi174 ceph-mon[35244]: Deploying daemon osd.2 on smithi203 2024-09-15T21:49:59.522 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:59 smithi203 ceph-mon[35546]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:59.522 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:49:59 smithi203 ceph-mon[35546]: Deploying daemon osd.2 on smithi203 2024-09-15T21:49:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:59 smithi028 ceph-mon[30969]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:49:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:49:59 smithi028 ceph-mon[30969]: Deploying daemon osd.2 on smithi203 2024-09-15T21:50:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:00 smithi203 ceph-mon[35546]: overall HEALTH_OK 2024-09-15T21:50:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:00 smithi174 ceph-mon[35244]: overall HEALTH_OK 2024-09-15T21:50:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:00 smithi028 ceph-mon[30969]: overall HEALTH_OK 2024-09-15T21:50:01.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:01 smithi203 ceph-mon[35546]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:01 smithi174 ceph-mon[35244]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:01 smithi028 ceph-mon[30969]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:03.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:03 smithi203 ceph-mon[35546]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:03 smithi174 ceph-mon[35244]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:03.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:03 smithi028 ceph-mon[30969]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:04.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:04.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:04.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:04.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:04.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:04.515 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:04.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:04.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:04.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:05.362 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:05 smithi203 ceph-mon[35546]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:05.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:05 smithi174 ceph-mon[35244]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:05.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:05 smithi028 ceph-mon[30969]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:07.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:07 smithi174 ceph-mon[35244]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:07.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:07.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:07.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.520 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:07 smithi203 ceph-mon[35546]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:07.520 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.520 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.520 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:07.520 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:07.521 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:07 smithi028 ceph-mon[30969]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:07.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:07.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:07.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:07.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:08.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.178+0000 7ff555ffb640 1 -- 172.21.15.203:0/2532296502 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff558062630 con 0x7ff5480a31b0 2024-09-15T21:50:08.181 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 2 on host 'smithi203' 2024-09-15T21:50:08.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.180+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5480a31b0 msgr2=0x7ff5480a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:08.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.180+0000 7ff55f049640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5480a31b0 0x7ff5480a5670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff558101b70 tx=0x7ff54c00a2d0 comp rx=0 tx=0).stop 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.180+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff558101d10 msgr2=0x7ff55819cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.180+0000 7ff55f049640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff558101d10 0x7ff55819cbf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff54400d950 tx=0x7ff54400de10 comp rx=0 tx=0).stop 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 shutdown_connections 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5480a31b0 0x7ff5480a5670 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5581026d0 0x7ff55819f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff558101d10 0x7ff55819cbf0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 --2- 172.21.15.203:0/2532296502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff558100b10 0x7ff55806b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 >> 172.21.15.203:0/2532296502 conn(0x7ff5580fc2a0 msgr2=0x7ff558105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 shutdown_connections 2024-09-15T21:50:08.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:50:08.181+0000 7ff55f049640 1 -- 172.21.15.203:0/2532296502 wait complete. 2024-09-15T21:50:08.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:08 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:08.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:08 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:08.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:08 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:08.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:08 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:08.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:08 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:08.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:08 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:08.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:08 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:08.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:08 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:08.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:08 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:08.820 DEBUG:teuthology.orchestra.run.smithi203:osd.2> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.2.service 2024-09-15T21:50:08.822 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-09-15T21:50:08.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T21:50:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:50:09.330 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:09 smithi203 ceph-mon[35546]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:09.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:09 smithi028 ceph-mon[30969]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:09.661 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:09 smithi174 ceph-mon[35244]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:10.114 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:09 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2[40488]: 2024-09-15T21:50:09.970+0000 7f064ad6b640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:50:10.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:10 smithi203 ceph-mon[35546]: from='osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T21:50:10.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:10 smithi203 ceph-mon[35546]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T21:50:10.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:10 smithi028 ceph-mon[30969]: from='osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T21:50:10.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:10 smithi028 ceph-mon[30969]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T21:50:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:10 smithi174 ceph-mon[35244]: from='osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T21:50:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:10 smithi174 ceph-mon[35244]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T21:50:11.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:11 smithi028 ceph-mon[30969]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:11.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:11 smithi028 ceph-mon[30969]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T21:50:11.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:11 smithi028 ceph-mon[30969]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T21:50:11.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:11 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:11.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:11 smithi028 ceph-mon[30969]: from='osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:50:11.328 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:11 smithi028 ceph-mon[30969]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:50:11.718 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2[40488]: 2024-09-15T21:50:11.275+0000 7f0646cee640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T21:50:11.718 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2[40488]: 2024-09-15T21:50:11.279+0000 7f0642b16640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:50:11.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-mon[35546]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:11.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-mon[35546]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T21:50:11.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-mon[35546]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T21:50:11.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:11.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-mon[35546]: from='osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:50:11.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:11 smithi203 ceph-mon[35546]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:50:11.731 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:50:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:11 smithi174 ceph-mon[35244]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:11 smithi174 ceph-mon[35244]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T21:50:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:11 smithi174 ceph-mon[35244]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T21:50:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:11 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:11 smithi174 ceph-mon[35244]: from='osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:50:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:11 smithi174 ceph-mon[35244]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:50:11.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-09-15T21:50:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:50:12.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: osdmap e17: 3 total, 2 up, 3 in 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:12.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:12.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.499+0000 7facf6e8d640 1 -- 172.21.15.28:0/3315354888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0102e00 msgr2=0x7facf0103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.499+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3315354888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0102e00 0x7facf0103280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7face4009900 tx=0x7face402a3f0 comp rx=0 tx=0).stop 2024-09-15T21:50:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 -- 172.21.15.28:0/3315354888 shutdown_connections 2024-09-15T21:50:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3315354888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facf01037c0 0x7facf010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3315354888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0102e00 0x7facf0103280 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3315354888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf0101c00 0x7facf0102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 -- 172.21.15.28:0/3315354888 >> 172.21.15.28:0/3315354888 conn(0x7facf00fd390 msgr2=0x7facf00ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:50:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 -- 172.21.15.28:0/3315354888 shutdown_connections 2024-09-15T21:50:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 -- 172.21.15.28:0/3315354888 wait complete. 2024-09-15T21:50:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 Processor -- start 2024-09-15T21:50:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.500+0000 7facf6e8d640 1 -- start start 2024-09-15T21:50:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf6e8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0101c00 0x7facf0197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf6e8d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf0102e00 0x7facf01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf6e8d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facf01037c0 0x7facf019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf6e8d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7facf010d8e0 con 0x7facf0101c00 2024-09-15T21:50:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf6e8d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7facf010d760 con 0x7facf01037c0 2024-09-15T21:50:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf6e8d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7facf010da60 con 0x7facf0102e00 2024-09-15T21:50:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0101c00 0x7facf0197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0101c00 0x7facf0197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48048/0 (socket says 172.21.15.28:48048) 2024-09-15T21:50:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 -- 172.21.15.28:0/3875104977 learned_addr learned my addr 172.21.15.28:0/3875104977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:50:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7faceffff640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf0102e00 0x7facf01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf542a640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facf01037c0 0x7facf019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 -- 172.21.15.28:0/3875104977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf0102e00 msgr2=0x7facf01984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf0102e00 0x7facf01984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 -- 172.21.15.28:0/3875104977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facf01037c0 msgr2=0x7facf019f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facf01037c0 0x7facf019f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 -- 172.21.15.28:0/3875104977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf019fb90 con 0x7facf0101c00 2024-09-15T21:50:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7faceffff640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf0102e00 0x7facf01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:50:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf4c29640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0101c00 0x7facf0197f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7face000d770 tx=0x7face000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:50:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0017070 con 0x7facf0101c00 2024-09-15T21:50:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face0009c30 con 0x7facf0101c00 2024-09-15T21:50:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf019fe80 con 0x7facf0101c00 2024-09-15T21:50:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.501+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0003dd0 con 0x7facf0101c00 2024-09-15T21:50:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.502+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf01a0350 con 0x7facf0101c00 2024-09-15T21:50:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.503+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf0108fd0 con 0x7facf0101c00 2024-09-15T21:50:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.504+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7face0003f70 con 0x7facf0101c00 2024-09-15T21:50:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.505+0000 7facedffb640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7facd80a31e0 0x7facd80a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.505+0000 7faceffff640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7facd80a31e0 0x7facd80a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.505+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2851+0+0 (secure 0 0 0) 0x7face00cdfe0 con 0x7facf0101c00 2024-09-15T21:50:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.506+0000 7faceffff640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7facd80a31e0 0x7facd80a56a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7face4000c00 tx=0x7face40095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:50:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.509+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7face0009de0 con 0x7facf0101c00 2024-09-15T21:50:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:12.694+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7facf0062630 con 0x7facd80a31e0 2024-09-15T21:50:12.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:50:12.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: osdmap e17: 3 total, 2 up, 3 in 2024-09-15T21:50:12.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:12.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:12.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:12.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:12.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:12.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:50:12.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: osdmap e17: 3 total, 2 up, 3 in 2024-09-15T21:50:12.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:12.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:12.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:12.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:12.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:12.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:12.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:13.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:50:13.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi203 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi203 to 17088M 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] boot 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:50:13.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:13.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:50:13.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:50:13.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:13.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi203 2024-09-15T21:50:13.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi203 to 17088M 2024-09-15T21:50:13.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] boot 2024-09-15T21:50:13.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T21:50:13.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:13.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:50:13.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:50:13.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:50:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:50:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-15T21:50:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi203 2024-09-15T21:50:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi203 to 17088M 2024-09-15T21:50:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: osd.2 [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] boot 2024-09-15T21:50:13.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T21:50:13.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:50:13.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:50:13.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:50:13.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:14.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:14 smithi028 ceph-mon[30969]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:50:14.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:14 smithi028 ceph-mon[30969]: osdmap e19: 3 total, 3 up, 3 in 2024-09-15T21:50:14.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T21:50:14.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:14 smithi203 ceph-mon[35546]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:50:14.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:14 smithi203 ceph-mon[35546]: osdmap e19: 3 total, 3 up, 3 in 2024-09-15T21:50:14.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T21:50:14.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:14 smithi174 ceph-mon[35244]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:50:14.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:14 smithi174 ceph-mon[35244]: osdmap e19: 3 total, 3 up, 3 in 2024-09-15T21:50:14.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T21:50:15.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:15.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T21:50:15.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: osdmap e20: 3 total, 3 up, 3 in 2024-09-15T21:50:15.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T21:50:15.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/382759557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd998a1a-b9b4-4950-8925-5b3d2da0e73c"}]: dispatch 2024-09-15T21:50:15.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T21:50:15.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/382759557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd998a1a-b9b4-4950-8925-5b3d2da0e73c"}]': finished 2024-09-15T21:50:15.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T21:50:15.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:15.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:15.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T21:50:15.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: osdmap e20: 3 total, 3 up, 3 in 2024-09-15T21:50:15.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T21:50:15.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/382759557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd998a1a-b9b4-4950-8925-5b3d2da0e73c"}]: dispatch 2024-09-15T21:50:15.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T21:50:15.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/382759557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd998a1a-b9b4-4950-8925-5b3d2da0e73c"}]': finished 2024-09-15T21:50:15.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T21:50:15.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T21:50:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: osdmap e20: 3 total, 3 up, 3 in 2024-09-15T21:50:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T21:50:15.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/382759557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bd998a1a-b9b4-4950-8925-5b3d2da0e73c"}]: dispatch 2024-09-15T21:50:15.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T21:50:15.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/382759557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd998a1a-b9b4-4950-8925-5b3d2da0e73c"}]': finished 2024-09-15T21:50:15.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T21:50:15.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:15.854 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:15 smithi028 sudo[49418]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T21:50:15.855 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:15 smithi028 sudo[49418]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:15.855 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:15 smithi028 sudo[49418]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:15.855 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:15 smithi028 sudo[49418]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:16.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:16.304+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7face0004260 con 0x7facf0101c00 2024-09-15T21:50:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:16 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3470461911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:50:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:16 smithi174 ceph-mon[35244]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T21:50:16.553 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:16 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:16.554 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43863]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T21:50:16.554 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43863]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:16.554 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43863]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:16.554 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43863]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:16.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:16 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3470461911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:50:16.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:16 smithi028 ceph-mon[30969]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T21:50:16.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:16.672 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:16 smithi028 sudo[49467]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T21:50:16.672 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:16 smithi028 sudo[49467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:16.672 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:16 smithi028 sudo[49467]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:16.672 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 21:50:16 smithi028 sudo[49467]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:16.721 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:16 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3470461911' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:50:16.721 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:16 smithi203 ceph-mon[35546]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T21:50:16.721 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:16 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:17.010 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43866]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T21:50:17.010 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:17.010 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43866]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:17.010 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 21:50:16 smithi174 sudo[43866]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:17.317 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:17 smithi203 ceph-mon[35546]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:17.317 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:17.317 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:17 smithi203 ceph-mon[35546]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T21:50:17.317 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:17.318 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44555]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T21:50:17.318 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44555]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:17.318 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44555]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:17.318 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44555]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:17.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 ceph-mon[30969]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:17.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:17.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 ceph-mon[30969]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T21:50:17.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:17.718 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44558]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T21:50:17.718 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44558]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:17.718 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44558]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:17.718 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 21:50:17 smithi203 sudo[44558]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:17 smithi174 ceph-mon[35244]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:17 smithi174 ceph-mon[35244]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T21:50:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 sudo[49844]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T21:50:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 sudo[49844]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 sudo[49844]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:17 smithi028 sudo[49844]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:18.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 sudo[49849]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T21:50:18.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 sudo[49849]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:18.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 sudo[49849]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:18.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 sudo[49849]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:18.734 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44561]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T21:50:18.734 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44561]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:18.734 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44561]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:18.734 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44561]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:19.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:19.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44564]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T21:50:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44564]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44564]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 sudo[44564]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:19.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:19.234 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:18 smithi174 sudo[43869]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T21:50:19.234 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:18 smithi174 sudo[43869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:19.234 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:18 smithi174 sudo[43869]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:19.234 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:18 smithi174 sudo[43869]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:19.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 sudo[43872]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 sudo[43872]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 sudo[43872]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 sudo[43872]: pam_unix(sudo:session): session closed for user root 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:19.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:19.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:20.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:20.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:20.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:19 smithi174 ceph-mon[35244]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:20.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:20.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:20.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:20.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:20.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:20.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:20.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:20.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:19 smithi028 ceph-mon[30969]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T21:50:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:50:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:50:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:50:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:19 smithi203 ceph-mon[35546]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T21:50:21.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:20 smithi174 ceph-mon[35244]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:21.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:20 smithi028 ceph-mon[30969]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:21.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:20 smithi203 ceph-mon[35546]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:23.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:22 smithi174 ceph-mon[35244]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:23.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:22 smithi028 ceph-mon[30969]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:23.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:22 smithi203 ceph-mon[35546]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:25.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:24 smithi174 ceph-mon[35244]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:25.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:24 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T21:50:25.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:24 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:25.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:24 smithi174 ceph-mon[35244]: Deploying daemon osd.3 on smithi028 2024-09-15T21:50:25.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:24 smithi028 ceph-mon[30969]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:25.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:24 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T21:50:25.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:24 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:25.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:24 smithi028 ceph-mon[30969]: Deploying daemon osd.3 on smithi028 2024-09-15T21:50:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:24 smithi203 ceph-mon[35546]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:24 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T21:50:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:24 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:24 smithi203 ceph-mon[35546]: Deploying daemon osd.3 on smithi028 2024-09-15T21:50:26.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:26 smithi028 ceph-mon[30969]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-09-15T21:50:27.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:26 smithi174 ceph-mon[35244]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-09-15T21:50:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:26 smithi203 ceph-mon[35546]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-09-15T21:50:29.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:28 smithi028 ceph-mon[30969]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-09-15T21:50:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:28 smithi203 ceph-mon[35546]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-09-15T21:50:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:28 smithi174 ceph-mon[35244]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-09-15T21:50:31.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:30 smithi028 ceph-mon[30969]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-15T21:50:31.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:31.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:31.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:30 smithi203 ceph-mon[35546]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-15T21:50:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:30 smithi174 ceph-mon[35244]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-15T21:50:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:32 smithi028 ceph-mon[30969]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:32 smithi203 ceph-mon[35546]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:32 smithi174 ceph-mon[35244]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:34.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:34.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.410+0000 7facedffb640 1 -- 172.21.15.28:0/3875104977 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7facf0062630 con 0x7facd80a31e0 2024-09-15T21:50:34.413 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7facd80a31e0 msgr2=0x7facd80a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7facd80a31e0 0x7facd80a56a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7face4000c00 tx=0x7face40095f0 comp rx=0 tx=0).stop 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0101c00 msgr2=0x7facf0197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0101c00 0x7facf0197f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7face000d770 tx=0x7face000dc30 comp rx=0 tx=0).stop 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 shutdown_connections 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7facd80a31e0 0x7facd80a56a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facf01037c0 0x7facf019f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facf0102e00 0x7facf01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:34.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 --2- 172.21.15.28:0/3875104977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf0101c00 0x7facf0197f60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:34.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 >> 172.21.15.28:0/3875104977 conn(0x7facf00fd390 msgr2=0x7facf01061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:50:34.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 shutdown_connections 2024-09-15T21:50:34.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:50:34.414+0000 7facf6e8d640 1 -- 172.21.15.28:0/3875104977 wait complete. 2024-09-15T21:50:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:35.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:35 smithi028 ceph-mon[30969]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:35 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:35 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:35 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:35.065 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.3.service 2024-09-15T21:50:35.067 INFO:tasks.cephadm:Deploying osd.4 on smithi174 with /dev/vg_nvme/lv_3... 2024-09-15T21:50:35.067 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T21:50:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:50:35.442 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:35 smithi174 ceph-mon[35244]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:35.443 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:35 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:35.443 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:35 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:35.443 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:35 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:35 smithi203 ceph-mon[35546]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:35 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:35 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:35 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:36.541 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 21:50:36 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3[51427]: 2024-09-15T21:50:36.187+0000 7fbc378bd640 -1 osd.3 0 log_to_monitors true 2024-09-15T21:50:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:37 smithi174 ceph-mon[35244]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:37 smithi174 ceph-mon[35244]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T21:50:37.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:37 smithi028 ceph-mon[30969]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:37.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:37 smithi028 ceph-mon[30969]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T21:50:37.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:37 smithi203 ceph-mon[35546]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-09-15T21:50:37.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:37 smithi203 ceph-mon[35546]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T21:50:38.106 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T21:50:38.106 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: osdmap e23: 4 total, 3 up, 4 in 2024-09-15T21:50:38.106 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:50:38.106 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:38.106 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.106 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.106 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:38.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:38.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:38.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:38.107 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.109 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:50:38.135 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-09-15T21:50:38.171 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3[51427]: 2024-09-15T21:50:38.030+0000 7fbc33840640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T21:50:38.172 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3[51427]: 2024-09-15T21:50:38.035+0000 7fbc2f668640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:50:38.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T21:50:38.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: osdmap e23: 4 total, 3 up, 4 in 2024-09-15T21:50:38.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:50:38.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:38.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:38.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:38.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:38.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:38.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.385 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:50:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T21:50:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: osdmap e23: 4 total, 3 up, 4 in 2024-09-15T21:50:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:50:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:50:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:38.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 -- 172.21.15.174:0/543477251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304102690 msgr2=0x7f1304108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 --2- 172.21.15.174:0/543477251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304102690 0x7f1304108f20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009ca0 tx=0x7f12fc02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 -- 172.21.15.174:0/543477251 shutdown_connections 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 --2- 172.21.15.174:0/543477251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304102690 0x7f1304108f20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 --2- 172.21.15.174:0/543477251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304101cd0 0x7f1304102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 --2- 172.21.15.174:0/543477251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1304100ad0 0x7f1304100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 -- 172.21.15.174:0/543477251 >> 172.21.15.174:0/543477251 conn(0x7f13040fc280 msgr2=0x7f13040fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:50:38.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 -- 172.21.15.174:0/543477251 shutdown_connections 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.848+0000 7f130a08c640 1 -- 172.21.15.174:0/543477251 wait complete. 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 Processor -- start 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 -- start start 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304100ad0 0x7f130406aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1304101cd0 0x7f130419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304102690 0x7f13041a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f130410c730 con 0x7f1304102690 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f130410c5b0 con 0x7f1304100ad0 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f130a08c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f130410c8b0 con 0x7f1304101cd0 2024-09-15T21:50:38.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f13037fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304100ad0 0x7f130406aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f13037fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304100ad0 0x7f130406aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:59388/0 (socket says 172.21.15.174:59388) 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f13037fe640 1 -- 172.21.15.174:0/3370474166 learned_addr learned my addr 172.21.15.174:0/3370474166 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f1302ffd640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1304101cd0 0x7f130419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.849+0000 7f1303fff640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304102690 0x7f13041a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f13037fe640 1 -- 172.21.15.174:0/3370474166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1304101cd0 msgr2=0x7f130419ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f13037fe640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1304101cd0 0x7f130419ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f13037fe640 1 -- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304102690 msgr2=0x7f13041a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f13037fe640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304102690 0x7f13041a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f13037fe640 1 -- 172.21.15.174:0/3370474166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13041a1c60 con 0x7f1304100ad0 2024-09-15T21:50:38.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f1303fff640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304102690 0x7f13041a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f13037fe640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304100ad0 0x7f130406aad0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f12f400c970 tx=0x7f12f400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f1300ff9640 1 -- 172.21.15.174:0/3370474166 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f4007c10 con 0x7f1304100ad0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f1300ff9640 1 -- 172.21.15.174:0/3370474166 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12f4007db0 con 0x7f1304100ad0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13041a1ef0 con 0x7f1304100ad0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f1300ff9640 1 -- 172.21.15.174:0/3370474166 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f400f8a0 con 0x7f1304100ad0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.850+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13041a23a0 con 0x7f1304100ad0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.851+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1304107e80 con 0x7f1304100ad0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.852+0000 7f1300ff9640 1 -- 172.21.15.174:0/3370474166 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f12f400fa40 con 0x7f1304100ad0 2024-09-15T21:50:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.853+0000 7f1300ff9640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12ec0a3270 0x7f12ec0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:50:38.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.853+0000 7f1300ff9640 1 -- 172.21.15.174:0/3370474166 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 3389+0+0 (secure 0 0 0) 0x7f12f40ce900 con 0x7f1304100ad0 2024-09-15T21:50:38.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.853+0000 7f1302ffd640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12ec0a3270 0x7f12ec0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:50:38.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.854+0000 7f1302ffd640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12ec0a3270 0x7f12ec0a5730 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f130406b050 tx=0x7f12f800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:50:38.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:38.857+0000 7f1300ff9640 1 -- 172.21.15.174:0/3370474166 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f12f409c9f0 con 0x7f1304100ad0 2024-09-15T21:50:39.035 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:50:39.033+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1304062630 con 0x7f12ec0a3270 2024-09-15T21:50:39.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:39.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi028 2024-09-15T21:50:39.370 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi028 to 8546M 2024-09-15T21:50:39.370 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:50:39.370 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: osdmap e24: 4 total, 3 up, 4 in 2024-09-15T21:50:39.370 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.370 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.370 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:39.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi028 2024-09-15T21:50:39.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi028 to 8546M 2024-09-15T21:50:39.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:50:39.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: osdmap e24: 4 total, 3 up, 4 in 2024-09-15T21:50:39.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi028 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi028 to 8546M 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: from='osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: osdmap e24: 4 total, 3 up, 4 in 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:39.459 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] boot 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:50:40.157 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:50:40.158 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:40.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:50:40.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:50:40.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:50:40.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] boot 2024-09-15T21:50:40.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T21:50:40.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:40.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:50:40.386 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:50:40.386 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:40.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:50:40.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:50:40.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:50:40.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: osd.3 [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] boot 2024-09-15T21:50:40.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T21:50:40.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:50:40.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:50:40.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:50:40.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:41.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:41 smithi028 ceph-mon[30969]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:41.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:41 smithi028 ceph-mon[30969]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T21:50:41.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:41 smithi203 ceph-mon[35546]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:41.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:41 smithi203 ceph-mon[35546]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T21:50:41.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:41 smithi174 ceph-mon[35244]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-15T21:50:41.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:41 smithi174 ceph-mon[35244]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T21:50:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:42 smithi174 ceph-mon[35244]: osdmap e27: 4 total, 4 up, 4 in 2024-09-15T21:50:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:42 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/652607649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]: dispatch 2024-09-15T21:50:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:42 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]: dispatch 2024-09-15T21:50:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:42 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]': finished 2024-09-15T21:50:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:42 smithi174 ceph-mon[35244]: osdmap e28: 5 total, 4 up, 5 in 2024-09-15T21:50:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:50:42.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:42 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/488168582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:50:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:42 smithi028 ceph-mon[30969]: osdmap e27: 4 total, 4 up, 4 in 2024-09-15T21:50:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:42 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/652607649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]: dispatch 2024-09-15T21:50:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:42 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]: dispatch 2024-09-15T21:50:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:42 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]': finished 2024-09-15T21:50:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:42 smithi028 ceph-mon[30969]: osdmap e28: 5 total, 4 up, 5 in 2024-09-15T21:50:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:50:42.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:42 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/488168582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:50:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:42 smithi203 ceph-mon[35546]: osdmap e27: 4 total, 4 up, 4 in 2024-09-15T21:50:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:42 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/652607649' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]: dispatch 2024-09-15T21:50:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:42 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]: dispatch 2024-09-15T21:50:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:42 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6"}]': finished 2024-09-15T21:50:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:42 smithi203 ceph-mon[35546]: osdmap e28: 5 total, 4 up, 5 in 2024-09-15T21:50:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:50:42.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:42 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/488168582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:50:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:43 smithi203 ceph-mon[35546]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:43 smithi203 ceph-mon[35546]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T21:50:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:50:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:43 smithi174 ceph-mon[35244]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:43 smithi174 ceph-mon[35244]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T21:50:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:50:43.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:43 smithi028 ceph-mon[30969]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:43.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:43 smithi028 ceph-mon[30969]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T21:50:43.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:50:45.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:45 smithi203 ceph-mon[35546]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:45.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:45 smithi174 ceph-mon[35244]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:45.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:45 smithi028 ceph-mon[30969]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:47.705 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:47 smithi174 ceph-mon[35244]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-15T21:50:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:47 smithi203 ceph-mon[35546]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-15T21:50:47.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:47 smithi028 ceph-mon[30969]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-15T21:50:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:49 smithi203 ceph-mon[35546]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-15T21:50:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:49 smithi174 ceph-mon[35244]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-15T21:50:49.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:49 smithi028 ceph-mon[30969]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-15T21:50:50.706 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T21:50:50.706 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:50.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T21:50:50.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:50.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T21:50:50.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:51 smithi203 ceph-mon[35546]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-15T21:50:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:51 smithi203 ceph-mon[35546]: Deploying daemon osd.4 on smithi174 2024-09-15T21:50:51.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:51 smithi174 ceph-mon[35244]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-15T21:50:51.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:51 smithi174 ceph-mon[35244]: Deploying daemon osd.4 on smithi174 2024-09-15T21:50:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:51 smithi028 ceph-mon[30969]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-15T21:50:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:51 smithi028 ceph-mon[30969]: Deploying daemon osd.4 on smithi174 2024-09-15T21:50:53.704 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:53 smithi174 ceph-mon[35244]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T21:50:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:53 smithi203 ceph-mon[35546]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T21:50:53.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:53 smithi028 ceph-mon[30969]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T21:50:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:55 smithi203 ceph-mon[35546]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-15T21:50:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:55 smithi174 ceph-mon[35244]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-15T21:50:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:55 smithi028 ceph-mon[30969]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-15T21:50:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:50:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:57.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:57 smithi203 ceph-mon[35546]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:50:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:57 smithi174 ceph-mon[35244]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:50:57.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:57 smithi028 ceph-mon[30969]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:50:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:59 smithi203 ceph-mon[35546]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:50:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:59 smithi174 ceph-mon[35244]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:50:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:59 smithi028 ceph-mon[30969]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:50:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:50:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:50:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:50:59.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:50:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:00.560 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.558+0000 7f1300ff9640 1 -- 172.21.15.174:0/3370474166 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1304062630 con 0x7f12ec0a3270 2024-09-15T21:51:00.561 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 4 on host 'smithi174' 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12ec0a3270 msgr2=0x7f12ec0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12ec0a3270 0x7f12ec0a5730 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f130406b050 tx=0x7f12f800b390 comp rx=0 tx=0).stop 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304100ad0 msgr2=0x7f130406aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304100ad0 0x7f130406aad0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f12f400c970 tx=0x7f12f400ce30 comp rx=0 tx=0).stop 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 shutdown_connections 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12ec0a3270 0x7f12ec0a5730 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1304102690 0x7f13041a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1304101cd0 0x7f130419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 --2- 172.21.15.174:0/3370474166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1304100ad0 0x7f130406aad0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:00.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 >> 172.21.15.174:0/3370474166 conn(0x7f13040fc280 msgr2=0x7f1304105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:51:00.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 shutdown_connections 2024-09-15T21:51:00.564 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:00.561+0000 7f130a08c640 1 -- 172.21.15.174:0/3370474166 wait complete. 2024-09-15T21:51:01.220 DEBUG:teuthology.orchestra.run.smithi174:osd.4> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.4.service 2024-09-15T21:51:01.223 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-09-15T21:51:01.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T21:51:01.475 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:51:01.596 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:01 smithi203 ceph-mon[35546]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:01.596 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:01.596 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:01.596 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:01.731 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:01 smithi174 ceph-mon[35244]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:01.731 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:01.731 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:01.731 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:01.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:01 smithi028 ceph-mon[30969]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:01.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:01.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:01.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:02.010 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 21:51:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4[46611]: 2024-09-15T21:51:01.728+0000 7f93155b8640 -1 osd.4 0 log_to_monitors true 2024-09-15T21:51:02.604 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:02 smithi174 ceph-mon[35244]: from='osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T21:51:02.604 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:02 smithi174 ceph-mon[35244]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T21:51:02.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:02 smithi203 ceph-mon[35546]: from='osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T21:51:02.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:02 smithi203 ceph-mon[35546]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T21:51:02.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:02 smithi028 ceph-mon[30969]: from='osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T21:51:02.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:02 smithi028 ceph-mon[30969]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T21:51:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:03 smithi203 ceph-mon[35546]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:03 smithi203 ceph-mon[35546]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T21:51:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:03 smithi203 ceph-mon[35546]: osdmap e30: 5 total, 4 up, 5 in 2024-09-15T21:51:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:03 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:03 smithi203 ceph-mon[35546]: from='osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:51:03.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:03 smithi203 ceph-mon[35546]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:51:03.760 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4[46611]: 2024-09-15T21:51:03.466+0000 7f931153b640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T21:51:03.760 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4[46611]: 2024-09-15T21:51:03.475+0000 7f930d363640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:51:03.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-mon[35244]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:03.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-mon[35244]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T21:51:03.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-mon[35244]: osdmap e30: 5 total, 4 up, 5 in 2024-09-15T21:51:03.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:03.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-mon[35244]: from='osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:51:03.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:03 smithi174 ceph-mon[35244]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:51:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:03 smithi028 ceph-mon[30969]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:03 smithi028 ceph-mon[30969]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T21:51:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:03 smithi028 ceph-mon[30969]: osdmap e30: 5 total, 4 up, 5 in 2024-09-15T21:51:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:03 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:03 smithi028 ceph-mon[30969]: from='osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:51:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:03 smithi028 ceph-mon[30969]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:51:04.111 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:51:04.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-09-15T21:51:04.398 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:51:04.751 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:51:04.751 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: osdmap e31: 5 total, 4 up, 5 in 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:04.752 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:04 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: osdmap e31: 5 total, 4 up, 5 in 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:04.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:04.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:04.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:04 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 -- 172.21.15.203:0/2098498664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc101cd0 msgr2=0x7f71cc102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:04.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 --2- 172.21.15.203:0/2098498664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc101cd0 0x7f71cc102150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009900 tx=0x7f71c802a410 comp rx=0 tx=0).stop 2024-09-15T21:51:04.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 -- 172.21.15.203:0/2098498664 shutdown_connections 2024-09-15T21:51:04.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 --2- 172.21.15.203:0/2098498664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71cc102690 0x7f71cc108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:04.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 --2- 172.21.15.203:0/2098498664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc101cd0 0x7f71cc102150 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 --2- 172.21.15.203:0/2098498664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71cc100ad0 0x7f71cc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 -- 172.21.15.203:0/2098498664 >> 172.21.15.203:0/2098498664 conn(0x7f71cc0fc280 msgr2=0x7f71cc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 -- 172.21.15.203:0/2098498664 shutdown_connections 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 -- 172.21.15.203:0/2098498664 wait complete. 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.835+0000 7f71d3d95640 1 Processor -- start 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 -- start start 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc100ad0 0x7f71cc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71cc101cd0 0x7f71cc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71cc102690 0x7f71cc19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc10c730 con 0x7f71cc101cd0 2024-09-15T21:51:04.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc10c5b0 con 0x7f71cc102690 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc10c8b0 con 0x7f71cc100ad0 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc100ad0 0x7f71cc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc100ad0 0x7f71cc197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:34284/0 (socket says 172.21.15.203:34284) 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 -- 172.21.15.203:0/1725475071 learned_addr learned my addr 172.21.15.203:0/1725475071 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1330640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71cc101cd0 0x7f71cc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d2332640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71cc102690 0x7f71cc19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 -- 172.21.15.203:0/1725475071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71cc102690 msgr2=0x7f71cc19f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71cc102690 0x7f71cc19f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 -- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71cc101cd0 msgr2=0x7f71cc198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71cc101cd0 0x7f71cc198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 -- 172.21.15.203:0/1725475071 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc19fb30 con 0x7f71cc100ad0 2024-09-15T21:51:04.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1330640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71cc101cd0 0x7f71cc198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d1b31640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc100ad0 0x7f71cc197f00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00d950 tx=0x7f71bc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71c2ffd640 1 -- 172.21.15.203:0/1725475071 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc017020 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71cc19fe20 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71c2ffd640 1 -- 172.21.15.203:0/1725475071 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71bc009c30 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71c2ffd640 1 -- 172.21.15.203:0/1725475071 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00f6b0 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.836+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71cc1a0280 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.838+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71cc1a0510 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.838+0000 7f71c2ffd640 1 -- 172.21.15.203:0/1725475071 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f71bc01a020 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.839+0000 7f71c2ffd640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f71b80a3270 0x7f71b80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.839+0000 7f71c2ffd640 1 -- 172.21.15.203:0/1725475071 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f71bc0ce990 con 0x7f71cc100ad0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.839+0000 7f71d1330640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f71b80a3270 0x7f71b80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.840+0000 7f71d1330640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f71b80a3270 0x7f71b80a5730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c80095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:51:04.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:04.843+0000 7f71c2ffd640 1 -- 172.21.15.203:0/1725475071 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f71bc021020 con 0x7f71cc100ad0 2024-09-15T21:51:04.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: osdmap e31: 5 total, 4 up, 5 in 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:04.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:04 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:05.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:05.017+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f71cc1a06e0 con 0x7f71b80a3270 2024-09-15T21:51:05.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:51:05.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi174 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi174 to 8544M 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] boot 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: osdmap e32: 5 total, 5 up, 5 in 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:05.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:51:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:51:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi174 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi174 to 8544M 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] boot 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: osdmap e32: 5 total, 5 up, 5 in 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:51:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:51:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-15T21:51:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi174 2024-09-15T21:51:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi174 to 8544M 2024-09-15T21:51:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: osd.4 [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] boot 2024-09-15T21:51:05.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: osdmap e32: 5 total, 5 up, 5 in 2024-09-15T21:51:05.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:51:05.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:05.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:05.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:05.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:06.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:06 smithi203 ceph-mon[35546]: osdmap e33: 5 total, 5 up, 5 in 2024-09-15T21:51:06.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:06 smithi203 ceph-mon[35546]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:06.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:06 smithi174 ceph-mon[35244]: osdmap e33: 5 total, 5 up, 5 in 2024-09-15T21:51:06.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:06 smithi174 ceph-mon[35244]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:06.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:06 smithi028 ceph-mon[30969]: osdmap e33: 5 total, 5 up, 5 in 2024-09-15T21:51:06.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:06 smithi028 ceph-mon[30969]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:07 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/434669120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]: dispatch 2024-09-15T21:51:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:07 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]: dispatch 2024-09-15T21:51:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:07 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]': finished 2024-09-15T21:51:07.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:07 smithi174 ceph-mon[35244]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T21:51:07.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:07.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:07 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/434669120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]: dispatch 2024-09-15T21:51:07.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:07 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]: dispatch 2024-09-15T21:51:07.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:07 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]': finished 2024-09-15T21:51:07.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:07 smithi203 ceph-mon[35546]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T21:51:07.762 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:07 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/434669120' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]: dispatch 2024-09-15T21:51:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:07 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]: dispatch 2024-09-15T21:51:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:07 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40516595-c52a-4503-b534-2b5c855d431e"}]': finished 2024-09-15T21:51:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:07 smithi028 ceph-mon[30969]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T21:51:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:08.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:08 smithi174 ceph-mon[35244]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:08.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:08 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/1581947234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:51:08.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:08 smithi028 ceph-mon[30969]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:08.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:08 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/1581947234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:51:08.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:08 smithi203 ceph-mon[35546]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:08.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:08 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/1581947234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:51:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:10 smithi174 ceph-mon[35244]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:10.770 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:10 smithi203 ceph-mon[35546]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:10.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:10 smithi028 ceph-mon[30969]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:12.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:12 smithi174 ceph-mon[35244]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:12.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:12 smithi028 ceph-mon[30969]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:12.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:12 smithi203 ceph-mon[35546]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:14.676 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:14 smithi203 ceph-mon[35546]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:14.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:14 smithi174 ceph-mon[35244]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:14.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:14 smithi028 ceph-mon[30969]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:16 smithi203 ceph-mon[35546]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:16 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T21:51:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:16 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:16 smithi203 ceph-mon[35546]: Deploying daemon osd.5 on smithi203 2024-09-15T21:51:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:16 smithi174 ceph-mon[35244]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:16 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T21:51:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:16 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:16 smithi174 ceph-mon[35244]: Deploying daemon osd.5 on smithi203 2024-09-15T21:51:16.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:16 smithi028 ceph-mon[30969]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:16.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T21:51:16.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:16.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:16 smithi028 ceph-mon[30969]: Deploying daemon osd.5 on smithi203 2024-09-15T21:51:18.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:18 smithi203 ceph-mon[35546]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:18.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:18 smithi174 ceph-mon[35244]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:18.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:18 smithi028 ceph-mon[30969]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:20.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:20 smithi174 ceph-mon[35244]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:20.832 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:20 smithi203 ceph-mon[35546]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:20.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:20 smithi028 ceph-mon[30969]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:22.863 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:22 smithi203 ceph-mon[35546]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:22.864 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:22.864 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:22.864 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:22.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:22 smithi028 ceph-mon[30969]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:22.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:22.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:22.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:23.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:22 smithi174 ceph-mon[35244]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:23.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:23.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:23.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:24.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:24 smithi203 ceph-mon[35546]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:24.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:24 smithi028 ceph-mon[30969]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:25.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:24 smithi174 ceph-mon[35244]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:26.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:26.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:26.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.433 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-09-15T21:51:26.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.431+0000 7f71c2ffd640 1 -- 172.21.15.203:0/1725475071 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f71cc1a06e0 con 0x7f71b80a3270 2024-09-15T21:51:26.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f71b80a3270 msgr2=0x7f71b80a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:26.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f71b80a3270 0x7f71b80a5730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71c8000c00 tx=0x7f71c80095f0 comp rx=0 tx=0).stop 2024-09-15T21:51:26.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc100ad0 msgr2=0x7f71cc197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:26.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc100ad0 0x7f71cc197f00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00d950 tx=0x7f71bc00de10 comp rx=0 tx=0).stop 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 shutdown_connections 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f71b80a3270 0x7f71b80a5730 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71cc102690 0x7f71cc19f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71cc101cd0 0x7f71cc198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 --2- 172.21.15.203:0/1725475071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71cc100ad0 0x7f71cc197f00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.433+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 >> 172.21.15.203:0/1725475071 conn(0x7f71cc0fc280 msgr2=0x7f71cc06a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.434+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 shutdown_connections 2024-09-15T21:51:26.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:51:26.434+0000 7f71d3d95640 1 -- 172.21.15.203:0/1725475071 wait complete. 2024-09-15T21:51:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:27.157 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.5.service 2024-09-15T21:51:27.159 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-09-15T21:51:27.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T21:51:27.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:27 smithi203 ceph-mon[35546]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:27.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:27 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:27.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:27 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:27.274 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:27 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:27.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:27 smithi028 ceph-mon[30969]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:27.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:27 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:27.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:27 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:27.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:27 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:27.425 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:51:27.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:27 smithi174 ceph-mon[35244]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:27.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:27 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:27.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:27 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:27.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:27 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:28.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:28 smithi203 ceph-mon[35546]: from='osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T21:51:28.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:28 smithi203 ceph-mon[35546]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T21:51:28.218 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 21:51:27 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5[47404]: 2024-09-15T21:51:27.948+0000 7f2b711b0640 -1 osd.5 0 log_to_monitors true 2024-09-15T21:51:28.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:28 smithi028 ceph-mon[30969]: from='osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T21:51:28.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:28 smithi028 ceph-mon[30969]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T21:51:28.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:28 smithi174 ceph-mon[35244]: from='osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T21:51:28.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:28 smithi174 ceph-mon[35244]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T21:51:29.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:29 smithi028 ceph-mon[30969]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:29.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:29 smithi028 ceph-mon[30969]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T21:51:29.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:29 smithi028 ceph-mon[30969]: osdmap e35: 6 total, 5 up, 6 in 2024-09-15T21:51:29.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:29 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:29.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:29 smithi028 ceph-mon[30969]: from='osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:51:29.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:29 smithi028 ceph-mon[30969]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:51:29.471 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5[47404]: 2024-09-15T21:51:29.080+0000 7f2b6d133640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T21:51:29.471 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5[47404]: 2024-09-15T21:51:29.087+0000 7f2b68f5b640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:51:29.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-mon[35546]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:29.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-mon[35546]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T21:51:29.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-mon[35546]: osdmap e35: 6 total, 5 up, 6 in 2024-09-15T21:51:29.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:29.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-mon[35546]: from='osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:51:29.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:29 smithi203 ceph-mon[35546]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:51:29.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:29 smithi174 ceph-mon[35244]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:29.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:29 smithi174 ceph-mon[35244]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T21:51:29.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:29 smithi174 ceph-mon[35244]: osdmap e35: 6 total, 5 up, 6 in 2024-09-15T21:51:29.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:29 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:29.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:29 smithi174 ceph-mon[35244]: from='osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:51:29.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:29 smithi174 ceph-mon[35244]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:51:30.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:51:30.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: osdmap e36: 6 total, 5 up, 6 in 2024-09-15T21:51:30.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:30.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:30.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:30.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:30.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:30.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:30.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:30 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:51:30.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-09-15T21:51:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:51:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: osdmap e36: 6 total, 5 up, 6 in 2024-09-15T21:51:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:30.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:30.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:30.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:30.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:30 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:51:30.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: osdmap e36: 6 total, 5 up, 6 in 2024-09-15T21:51:30.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:30.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:30.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:30.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:30.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:30.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:30.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:30 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:51:31.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.001+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/3011543919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34101c00 msgr2=0x7f7e34102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.001+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/3011543919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34101c00 0x7f7e34102000 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009900 tx=0x7f7e2402a410 comp rx=0 tx=0).stop 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.001+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/3011543919 shutdown_connections 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.001+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/3011543919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e341037c0 0x7f7e3410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.001+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/3011543919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e34102e00 0x7f7e34103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.001+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/3011543919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34101c00 0x7f7e34102000 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.001+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/3011543919 >> 172.21.15.28:0/3011543919 conn(0x7f7e340fd390 msgr2=0x7f7e340ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/3011543919 shutdown_connections 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/3011543919 wait complete. 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 Processor -- start 2024-09-15T21:51:31.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 -- start start 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e34101c00 0x7f7e3419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34102e00 0x7f7e3419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e341037c0 0x7f7e341a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3410d890 con 0x7f7e34102e00 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3410d710 con 0x7f7e34101c00 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.002+0000 7f7e3a5c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3410da10 con 0x7f7e341037c0 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34102e00 0x7f7e3419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34102e00 0x7f7e3419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49022/0 (socket says 172.21.15.28:49022) 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 -- 172.21.15.28:0/4147329497 learned_addr learned my addr 172.21.15.28:0/4147329497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:51:31.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e33fff640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e34101c00 0x7f7e3419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e38b66640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e341037c0 0x7f7e341a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 -- 172.21.15.28:0/4147329497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e341037c0 msgr2=0x7f7e341a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e341037c0 0x7f7e341a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 -- 172.21.15.28:0/4147329497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e34101c00 msgr2=0x7f7e3419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e34101c00 0x7f7e3419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 -- 172.21.15.28:0/4147329497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e341a1d50 con 0x7f7e34102e00 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e33fff640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e34101c00 0x7f7e3419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e337fe640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34102e00 0x7f7e3419a660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800e9f0 tx=0x7f7e2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e317fa640 1 -- 172.21.15.28:0/4147329497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800cd80 con 0x7f7e34102e00 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e317fa640 1 -- 172.21.15.28:0/4147329497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e28010910 con 0x7f7e34102e00 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e341a1fe0 con 0x7f7e34102e00 2024-09-15T21:51:31.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e317fa640 1 -- 172.21.15.28:0/4147329497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28018990 con 0x7f7e34102e00 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.003+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e341a2540 con 0x7f7e34102e00 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.004+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e341a2770 con 0x7f7e34102e00 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.005+0000 7f7e317fa640 1 -- 172.21.15.28:0/4147329497 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7e28010ac0 con 0x7f7e34102e00 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.006+0000 7f7e317fa640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7e1c0a32f0 0x7f7e1c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.007+0000 7f7e33fff640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7e1c0a32f0 0x7f7e1c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.007+0000 7f7e317fa640 1 -- 172.21.15.28:0/4147329497 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4562+0+0 (secure 0 0 0) 0x7f7e280cee70 con 0x7f7e34102e00 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.007+0000 7f7e33fff640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7e1c0a32f0 0x7f7e1c0a57b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000c00 tx=0x7f7e24031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:51:31.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.011+0000 7f7e317fa640 1 -- 172.21.15.28:0/4147329497 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7e2809c990 con 0x7f7e34102e00 2024-09-15T21:51:31.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:51:31.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:51:31.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:31.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi203 2024-09-15T21:51:31.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi203 to 8544M 2024-09-15T21:51:31.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] boot 2024-09-15T21:51:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: osdmap e37: 6 total, 6 up, 6 in 2024-09-15T21:51:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:31.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:31 smithi028 ceph-mon[30969]: osdmap e38: 6 total, 6 up, 6 in 2024-09-15T21:51:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:31.196+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7e34109d10 con 0x7f7e1c0a32f0 2024-09-15T21:51:31.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi203 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi203 to 8544M 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] boot 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: osdmap e37: 6 total, 6 up, 6 in 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:31.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:31 smithi203 ceph-mon[35546]: osdmap e38: 6 total, 6 up, 6 in 2024-09-15T21:51:31.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:51:31.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi203 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi203 to 8544M 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: osd.5 [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] boot 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: osdmap e37: 6 total, 6 up, 6 in 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:51:31.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:31 smithi174 ceph-mon[35244]: osdmap e38: 6 total, 6 up, 6 in 2024-09-15T21:51:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:32 smithi028 ceph-mon[30969]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:32 smithi028 ceph-mon[30969]: osdmap e39: 6 total, 6 up, 6 in 2024-09-15T21:51:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:32 smithi203 ceph-mon[35546]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:32 smithi203 ceph-mon[35546]: osdmap e39: 6 total, 6 up, 6 in 2024-09-15T21:51:32.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:32 smithi174 ceph-mon[35244]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:32.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:32.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:32.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:32.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:32 smithi174 ceph-mon[35244]: osdmap e39: 6 total, 6 up, 6 in 2024-09-15T21:51:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:33 smithi028 ceph-mon[30969]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:33 smithi028 ceph-mon[30969]: osdmap e40: 6 total, 6 up, 6 in 2024-09-15T21:51:33.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:33 smithi203 ceph-mon[35546]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:33.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:33 smithi203 ceph-mon[35546]: osdmap e40: 6 total, 6 up, 6 in 2024-09-15T21:51:33.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:33 smithi174 ceph-mon[35244]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:33.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:33 smithi174 ceph-mon[35244]: osdmap e40: 6 total, 6 up, 6 in 2024-09-15T21:51:34.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:34 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1135918036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4db3191e-2eb9-4ce3-b007-11d4305d0c91"}]: dispatch 2024-09-15T21:51:34.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:34 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1135918036' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4db3191e-2eb9-4ce3-b007-11d4305d0c91"}]': finished 2024-09-15T21:51:34.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:34 smithi028 ceph-mon[30969]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T21:51:34.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:34 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1135918036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4db3191e-2eb9-4ce3-b007-11d4305d0c91"}]: dispatch 2024-09-15T21:51:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:34 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1135918036' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4db3191e-2eb9-4ce3-b007-11d4305d0c91"}]': finished 2024-09-15T21:51:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:34 smithi203 ceph-mon[35546]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T21:51:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:34.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:34 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1135918036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4db3191e-2eb9-4ce3-b007-11d4305d0c91"}]: dispatch 2024-09-15T21:51:34.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:34 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1135918036' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4db3191e-2eb9-4ce3-b007-11d4305d0c91"}]': finished 2024-09-15T21:51:34.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:34 smithi174 ceph-mon[35244]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T21:51:34.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:35 smithi028 ceph-mon[30969]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:35 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/505516199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:51:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:35 smithi203 ceph-mon[35546]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:35 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/505516199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:51:35.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:35 smithi174 ceph-mon[35244]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:35.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:35 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/505516199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:51:37.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:37 smithi028 ceph-mon[30969]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-15T21:51:37.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:37 smithi203 ceph-mon[35546]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-15T21:51:37.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:37 smithi174 ceph-mon[35244]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-09-15T21:51:39.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:39 smithi028 ceph-mon[30969]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:51:39.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:39 smithi203 ceph-mon[35546]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:51:39.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:39 smithi174 ceph-mon[35244]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:51:41.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:41 smithi028 ceph-mon[30969]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-15T21:51:41.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:41 smithi203 ceph-mon[35546]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-15T21:51:41.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:41 smithi174 ceph-mon[35244]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-15T21:51:43.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:43 smithi028 ceph-mon[30969]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-15T21:51:43.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T21:51:43.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:43.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:43 smithi028 ceph-mon[30969]: Deploying daemon osd.6 on smithi028 2024-09-15T21:51:43.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:43 smithi203 ceph-mon[35546]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-15T21:51:43.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T21:51:43.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:43.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:43 smithi203 ceph-mon[35546]: Deploying daemon osd.6 on smithi028 2024-09-15T21:51:43.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:43 smithi174 ceph-mon[35244]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-15T21:51:43.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T21:51:43.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:43.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:43 smithi174 ceph-mon[35244]: Deploying daemon osd.6 on smithi028 2024-09-15T21:51:45.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:45 smithi028 ceph-mon[30969]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T21:51:45.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:45 smithi203 ceph-mon[35546]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T21:51:45.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:45 smithi174 ceph-mon[35244]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T21:51:47.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:47 smithi028 ceph-mon[30969]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:51:47.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:47 smithi203 ceph-mon[35546]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:51:47.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:47 smithi174 ceph-mon[35244]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:51:49.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:49 smithi028 ceph-mon[30969]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:49.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:49.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:49.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:49.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:49 smithi203 ceph-mon[35546]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:49.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:49.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:49.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:49.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:49 smithi174 ceph-mon[35244]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:49.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:49.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:49.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:51.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:51 smithi028 ceph-mon[30969]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:51.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:51 smithi203 ceph-mon[35546]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:51.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:51 smithi174 ceph-mon[35244]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:52.264 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-09-15T21:51:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.262+0000 7f7e317fa640 1 -- 172.21.15.28:0/4147329497 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7e34109d10 con 0x7f7e1c0a32f0 2024-09-15T21:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.264+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7e1c0a32f0 msgr2=0x7f7e1c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.264+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7e1c0a32f0 0x7f7e1c0a57b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e24000c00 tx=0x7f7e24031400 comp rx=0 tx=0).stop 2024-09-15T21:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.264+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34102e00 msgr2=0x7f7e3419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.264+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34102e00 0x7f7e3419a660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800e9f0 tx=0x7f7e2800eeb0 comp rx=0 tx=0).stop 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 shutdown_connections 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7e1c0a32f0 0x7f7e1c0a57b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e341037c0 0x7f7e341a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34102e00 0x7f7e3419a660 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 --2- 172.21.15.28:0/4147329497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e34101c00 0x7f7e3419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 >> 172.21.15.28:0/4147329497 conn(0x7f7e340fd390 msgr2=0x7f7e341085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 shutdown_connections 2024-09-15T21:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:51:52.265+0000 7f7e3a5c9640 1 -- 172.21.15.28:0/4147329497 wait complete. 2024-09-15T21:51:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:52 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:52.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:52.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:52 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:52.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:52.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:52.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:52 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:53.044 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.6.service 2024-09-15T21:51:53.046 INFO:tasks.cephadm:Deploying osd.7 on smithi174 with /dev/vg_nvme/lv_2... 2024-09-15T21:51:53.046 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T21:51:53.296 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:51:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:53 smithi028 ceph-mon[30969]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:53.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:53.417 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:53 smithi174 ceph-mon[35244]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:53.417 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:53.417 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:53.417 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:53.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:53 smithi203 ceph-mon[35546]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:53.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:51:53.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:53.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:54.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:54 smithi028 ceph-mon[30969]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T21:51:54.392 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 21:51:54 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6[57911]: 2024-09-15T21:51:54.077+0000 7fe26db9c640 -1 osd.6 0 log_to_monitors true 2024-09-15T21:51:54.446 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:54 smithi174 ceph-mon[35244]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T21:51:54.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:54 smithi203 ceph-mon[35546]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T21:51:55.328 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:55 smithi174 ceph-mon[35244]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:55.328 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:55 smithi174 ceph-mon[35244]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T21:51:55.329 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:55 smithi174 ceph-mon[35244]: osdmap e42: 7 total, 6 up, 7 in 2024-09-15T21:51:55.329 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:55 smithi174 ceph-mon[35244]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:51:55.329 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:55 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:55.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:55 smithi028 ceph-mon[30969]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:55.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:55 smithi028 ceph-mon[30969]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T21:51:55.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:55 smithi028 ceph-mon[30969]: osdmap e42: 7 total, 6 up, 7 in 2024-09-15T21:51:55.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:55 smithi028 ceph-mon[30969]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:51:55.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:55 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:55.422 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 21:51:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6[57911]: 2024-09-15T21:51:55.166+0000 7fe269b1f640 -1 osd.6 0 waiting for initial osdmap 2024-09-15T21:51:55.422 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 21:51:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6[57911]: 2024-09-15T21:51:55.179+0000 7fe265947640 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:51:55.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:55 smithi203 ceph-mon[35546]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:55.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:55 smithi203 ceph-mon[35546]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T21:51:55.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:55 smithi203 ceph-mon[35546]: osdmap e42: 7 total, 6 up, 7 in 2024-09-15T21:51:55.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:55 smithi203 ceph-mon[35546]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:51:55.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:55 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:56.172 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:51:56.199 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-09-15T21:51:56.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:51:56.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: osdmap e43: 7 total, 6 up, 7 in 2024-09-15T21:51:56.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:56.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:56.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.456 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:51:56.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:51:56.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: osdmap e43: 7 total, 6 up, 7 in 2024-09-15T21:51:56.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:56.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:56.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:56.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:56.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:51:56.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: osdmap e43: 7 total, 6 up, 7 in 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:51:56.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:51:56.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 -- 172.21.15.174:0/1762858144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 msgr2=0x7f7e08109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:56.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 --2- 172.21.15.174:0/1762858144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e08109d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009ca0 tx=0x7f7e0002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 -- 172.21.15.174:0/1762858144 shutdown_connections 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 --2- 172.21.15.174:0/1762858144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e08109d50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 --2- 172.21.15.174:0/1762858144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e08104fd0 0x7f7e081073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 --2- 172.21.15.174:0/1762858144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e081026a0 0x7f7e08104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 -- 172.21.15.174:0/1762858144 >> 172.21.15.174:0/1762858144 conn(0x7f7e080fc2a0 msgr2=0x7f7e080fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 -- 172.21.15.174:0/1762858144 shutdown_connections 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.951+0000 7f7e0d553640 1 -- 172.21.15.174:0/1762858144 wait complete. 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 Processor -- start 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 -- start start 2024-09-15T21:51:56.979 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e081026a0 0x7f7e08197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e08104fd0 0x7f7e08198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e0819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0810d8a0 con 0x7f7e081026a0 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0810d720 con 0x7f7e08107900 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e0d553640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0810da20 con 0x7f7e08104fd0 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e0819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e0819f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:45778/0 (socket says 172.21.15.174:45778) 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 -- 172.21.15.174:0/746521899 learned_addr learned my addr 172.21.15.174:0/746521899 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e067fc640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e08104fd0 0x7f7e08198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e06ffd640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e081026a0 0x7f7e08197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:56.980 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 -- 172.21.15.174:0/746521899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e08104fd0 msgr2=0x7f7e08198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e08104fd0 0x7f7e08198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 -- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e081026a0 msgr2=0x7f7e08197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e081026a0 0x7f7e08197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e077fe640 1 -- 172.21.15.174:0/746521899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0819fb00 con 0x7f7e08107900 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e06ffd640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e081026a0 0x7f7e08197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.952+0000 7f7e067fc640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e08104fd0 0x7f7e08198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.953+0000 7f7e077fe640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e0819f490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e00005e30 tx=0x7f7e00031ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.953+0000 7f7deffff640 1 -- 172.21.15.174:0/746521899 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00003c30 con 0x7f7e08107900 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.953+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0819fd90 con 0x7f7e08107900 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.953+0000 7f7deffff640 1 -- 172.21.15.174:0/746521899 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e00003dd0 con 0x7f7e08107900 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.953+0000 7f7deffff640 1 -- 172.21.15.174:0/746521899 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0002fde0 con 0x7f7e08107900 2024-09-15T21:51:56.981 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.953+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e081a0270 con 0x7f7e08107900 2024-09-15T21:51:56.982 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.954+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e080fdd90 con 0x7f7e08107900 2024-09-15T21:51:56.982 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.955+0000 7f7deffff640 1 -- 172.21.15.174:0/746521899 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7e00004050 con 0x7f7e08107900 2024-09-15T21:51:56.982 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.956+0000 7f7deffff640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a3270 0x7f7df00a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:51:56.982 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.956+0000 7f7deffff640 1 -- 172.21.15.174:0/746521899 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f7e000f0100 con 0x7f7e08107900 2024-09-15T21:51:56.982 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.956+0000 7f7e06ffd640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a3270 0x7f7df00a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:51:56.982 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.957+0000 7f7e06ffd640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a3270 0x7f7df00a5730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009b40 tx=0x7f7df8005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:51:56.982 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:56.959+0000 7f7deffff640 1 -- 172.21.15.174:0/746521899 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7e000bdc50 con 0x7f7e08107900 2024-09-15T21:51:57.141 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:51:57.139+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7e08062630 con 0x7f7df00a3270 2024-09-15T21:51:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:51:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:51:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi028 2024-09-15T21:51:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi028 to 5697M 2024-09-15T21:51:57.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] boot 2024-09-15T21:51:57.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: osdmap e44: 7 total, 7 up, 7 in 2024-09-15T21:51:57.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:57.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:57.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:57.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi028 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi028 to 5697M 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] boot 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: osdmap e44: 7 total, 7 up, 7 in 2024-09-15T21:51:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:57.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:57.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:57.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:51:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:51:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-15T21:51:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi028 2024-09-15T21:51:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi028 to 5697M 2024-09-15T21:51:57.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: osd.6 [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] boot 2024-09-15T21:51:57.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: osdmap e44: 7 total, 7 up, 7 in 2024-09-15T21:51:57.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:51:57.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:51:57.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:51:57.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:51:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:58 smithi203 ceph-mon[35546]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:58 smithi203 ceph-mon[35546]: osdmap e45: 7 total, 7 up, 7 in 2024-09-15T21:51:58.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:58 smithi174 ceph-mon[35244]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:58.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:58 smithi174 ceph-mon[35244]: osdmap e45: 7 total, 7 up, 7 in 2024-09-15T21:51:58.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:58 smithi028 ceph-mon[30969]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:51:58.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:58 smithi028 ceph-mon[30969]: osdmap e45: 7 total, 7 up, 7 in 2024-09-15T21:51:59.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:51:59 smithi203 ceph-mon[35546]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:51:59.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:51:59 smithi174 ceph-mon[35244]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:51:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:51:59 smithi028 ceph-mon[30969]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:00 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/867649987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]: dispatch 2024-09-15T21:52:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:00 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]: dispatch 2024-09-15T21:52:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:00 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]': finished 2024-09-15T21:52:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:00 smithi203 ceph-mon[35546]: osdmap e46: 8 total, 7 up, 8 in 2024-09-15T21:52:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:00.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:00 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/1817943490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:00 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/867649987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]: dispatch 2024-09-15T21:52:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:00 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]: dispatch 2024-09-15T21:52:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:00 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]': finished 2024-09-15T21:52:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:00 smithi174 ceph-mon[35244]: osdmap e46: 8 total, 7 up, 8 in 2024-09-15T21:52:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:00.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:00 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/1817943490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:00 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/867649987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]: dispatch 2024-09-15T21:52:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:00 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]: dispatch 2024-09-15T21:52:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:00 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6259a57-5604-481c-b739-f187239955d0"}]': finished 2024-09-15T21:52:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:00 smithi028 ceph-mon[30969]: osdmap e46: 8 total, 7 up, 8 in 2024-09-15T21:52:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:00 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/1817943490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:01.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:01 smithi203 ceph-mon[35546]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:01 smithi174 ceph-mon[35244]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:01 smithi028 ceph-mon[30969]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:03.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:03 smithi203 ceph-mon[35546]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:03.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:03 smithi174 ceph-mon[35244]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:03.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:03 smithi028 ceph-mon[30969]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:05.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:05 smithi203 ceph-mon[35546]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:05.469 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:05 smithi174 ceph-mon[35244]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:05.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:05 smithi028 ceph-mon[30969]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:07.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:07 smithi174 ceph-mon[35244]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:07 smithi203 ceph-mon[35546]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:07.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:07 smithi028 ceph-mon[30969]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:09 smithi203 ceph-mon[35546]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:09 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T21:52:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:09 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:09 smithi203 ceph-mon[35546]: Deploying daemon osd.7 on smithi174 2024-09-15T21:52:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:09 smithi174 ceph-mon[35244]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:09 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T21:52:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:09 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:09 smithi174 ceph-mon[35244]: Deploying daemon osd.7 on smithi174 2024-09-15T21:52:09.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:09 smithi028 ceph-mon[30969]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:09.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T21:52:09.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:09.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:09 smithi028 ceph-mon[30969]: Deploying daemon osd.7 on smithi174 2024-09-15T21:52:11.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:11 smithi174 ceph-mon[35244]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:11.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:11 smithi203 ceph-mon[35546]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:11 smithi028 ceph-mon[30969]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:13.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:13 smithi203 ceph-mon[35546]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:13.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:13 smithi174 ceph-mon[35244]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:13.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:13 smithi028 ceph-mon[30969]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:15 smithi203 ceph-mon[35546]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:15.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:15 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:15 smithi174 ceph-mon[35244]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:15.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:15.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:15 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:15 smithi028 ceph-mon[30969]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:15 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:17.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:17 smithi203 ceph-mon[35546]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:17.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:17 smithi174 ceph-mon[35244]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:17.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:17 smithi028 ceph-mon[30969]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:18.331 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.328+0000 7f7deffff640 1 -- 172.21.15.174:0/746521899 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7e08062630 con 0x7f7df00a3270 2024-09-15T21:52:18.331 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 7 on host 'smithi174' 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a3270 msgr2=0x7f7df00a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a3270 0x7f7df00a5730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009b40 tx=0x7f7df8005e50 comp rx=0 tx=0).stop 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 msgr2=0x7f7e0819f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e0819f490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e00005e30 tx=0x7f7e00031ee0 comp rx=0 tx=0).stop 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 shutdown_connections 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a3270 0x7f7df00a5730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e08107900 0x7f7e0819f490 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e08104fd0 0x7f7e08198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:18.333 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 --2- 172.21.15.174:0/746521899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e081026a0 0x7f7e08197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:18.334 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 >> 172.21.15.174:0/746521899 conn(0x7f7e080fc2a0 msgr2=0x7f7e08102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:52:18.334 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 shutdown_connections 2024-09-15T21:52:18.334 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:52:18.331+0000 7f7e0d553640 1 -- 172.21.15.174:0/746521899 wait complete. 2024-09-15T21:52:18.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:18.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:18.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:18 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.633 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:18 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.633 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:18 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.633 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:18 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:18.633 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:18 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:18.633 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:18 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:18.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:18.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:18 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:18.969 DEBUG:teuthology.orchestra.run.smithi174:osd.7> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.7.service 2024-09-15T21:52:18.972 INFO:tasks.cephadm:Deploying osd.8 on smithi203 with /dev/vg_nvme/lv_2... 2024-09-15T21:52:18.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T21:52:19.229 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:52:19.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:19 smithi174 ceph-mon[35244]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:19.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:19.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:19.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:19.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:19 smithi028 ceph-mon[30969]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:19.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:19.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:19.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:19.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:19 smithi203 ceph-mon[35546]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:19.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:19.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:19.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:20.182 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 21:52:19 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7[53595]: 2024-09-15T21:52:19.978+0000 7f4ad1d6b640 -1 osd.7 0 log_to_monitors true 2024-09-15T21:52:20.486 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:20 smithi174 ceph-mon[35244]: from='osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T21:52:20.486 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:20 smithi174 ceph-mon[35244]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T21:52:20.638 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:20 smithi203 ceph-mon[35546]: from='osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T21:52:20.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:20 smithi203 ceph-mon[35546]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T21:52:20.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:20 smithi028 ceph-mon[30969]: from='osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T21:52:20.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:20 smithi028 ceph-mon[30969]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T21:52:21.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-mon[35244]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:21.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-mon[35244]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T21:52:21.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-mon[35244]: osdmap e47: 8 total, 7 up, 8 in 2024-09-15T21:52:21.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:21.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-mon[35244]: from='osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:52:21.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-mon[35244]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:52:21.569 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:21 smithi203 ceph-mon[35546]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:21.569 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:21 smithi203 ceph-mon[35546]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T21:52:21.569 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:21 smithi203 ceph-mon[35546]: osdmap e47: 8 total, 7 up, 8 in 2024-09-15T21:52:21.569 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:21 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:21.569 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:21 smithi203 ceph-mon[35546]: from='osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:52:21.569 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:21 smithi203 ceph-mon[35546]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:52:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:21 smithi028 ceph-mon[30969]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:21 smithi028 ceph-mon[30969]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T21:52:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:21 smithi028 ceph-mon[30969]: osdmap e47: 8 total, 7 up, 8 in 2024-09-15T21:52:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:21 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:21 smithi028 ceph-mon[30969]: from='osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:52:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:21 smithi028 ceph-mon[30969]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:52:21.923 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:52:21.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-09-15T21:52:22.209 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:52:22.260 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7[53595]: 2024-09-15T21:52:21.959+0000 7f4acdcee640 -1 osd.7 0 waiting for initial osdmap 2024-09-15T21:52:22.260 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 21:52:21 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7[53595]: 2024-09-15T21:52:21.967+0000 7f4ac5b0e640 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:52:22.566 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: osdmap e48: 8 total, 7 up, 8 in 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:22.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:22.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:22 smithi203 ceph-mon[35546]: from='osd.7 ' entity='osd.7' 2024-09-15T21:52:22.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:52:22.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: osdmap e48: 8 total, 7 up, 8 in 2024-09-15T21:52:22.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:22.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:22.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:22 smithi028 ceph-mon[30969]: from='osd.7 ' entity='osd.7' 2024-09-15T21:52:22.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2537677167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 msgr2=0x7f92e4100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:22.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2537677167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4100690 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f92d400aa10 tx=0x7f92d402b720 comp rx=0 tx=0).stop 2024-09-15T21:52:22.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2537677167 shutdown_connections 2024-09-15T21:52:22.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2537677167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92e4101e50 0x7f92e41086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:22.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2537677167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101490 0x7f92e4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:22.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2537677167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4100690 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:22.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2537677167 >> 172.21.15.203:0/2537677167 conn(0x7f92e40fba40 msgr2=0x7f92e40fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2537677167 shutdown_connections 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.657+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2537677167 wait complete. 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 Processor -- start 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 -- start start 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92e4101490 0x7f92e41962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101e50 0x7f92e419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92e410b6b0 con 0x7f92e4101e50 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f92e410b530 con 0x7f92e4101490 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e8bb9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f92e410b830 con 0x7f92e4100290 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:52:22.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:40984/0 (socket says 172.21.15.203:40984) 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 -- 172.21.15.203:0/2916590835 learned_addr learned my addr 172.21.15.203:0/2916590835 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e1d9b640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92e4101490 0x7f92e41962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 -- 172.21.15.203:0/2916590835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92e4101490 msgr2=0x7f92e41962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92e4101490 0x7f92e41962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 -- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101e50 msgr2=0x7f92e419d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101e50 0x7f92e419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.658+0000 7f92e259c640 1 -- 172.21.15.203:0/2916590835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e419d9e0 con 0x7f92e4100290 2024-09-15T21:52:22.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.659+0000 7f92e259c640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4195db0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92d4005ab0 tx=0x7f92d40042d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:52:22.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.659+0000 7f92d37fe640 1 -- 172.21.15.203:0/2916590835 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4040070 con 0x7f92e4100290 2024-09-15T21:52:22.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.659+0000 7f92d37fe640 1 -- 172.21.15.203:0/2916590835 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92d40044c0 con 0x7f92e4100290 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.659+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e419dc70 con 0x7f92e4100290 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.659+0000 7f92d37fe640 1 -- 172.21.15.203:0/2916590835 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d4036030 con 0x7f92e4100290 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.659+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e419e150 con 0x7f92e4100290 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.660+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e4107730 con 0x7f92e4100290 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.661+0000 7f92d37fe640 1 -- 172.21.15.203:0/2916590835 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f92d4043070 con 0x7f92e4100290 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.662+0000 7f92d37fe640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f92cc0a3270 0x7f92cc0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.662+0000 7f92d37fe640 1 -- 172.21.15.203:0/2916590835 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5430+0+0 (secure 0 0 0) 0x7f92d40f0840 con 0x7f92e4100290 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.662+0000 7f92e1d9b640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f92cc0a3270 0x7f92cc0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.662+0000 7f92e1d9b640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f92cc0a3270 0x7f92cc0a5730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f92e41972d0 tx=0x7f92d800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:52:22.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.666+0000 7f92d37fe640 1 -- 172.21.15.203:0/2916590835 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f92d40be0a0 con 0x7f92e4100290 2024-09-15T21:52:22.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: osdmap e48: 8 total, 7 up, 8 in 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:22.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:22.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:22 smithi174 ceph-mon[35244]: from='osd.7 ' entity='osd.7' 2024-09-15T21:52:22.843 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:22.841+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f92e4062630 con 0x7f92cc0a3270 2024-09-15T21:52:23.586 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:52:23.586 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:52:23.586 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi174 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi174 to 5696M 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] boot 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: osdmap e49: 8 total, 8 up, 8 in 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:52:23.587 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi174 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi174 to 5696M 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] boot 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: osdmap e49: 8 total, 8 up, 8 in 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:52:23.622 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:52:23.623 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:52:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:52:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-15T21:52:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi174 2024-09-15T21:52:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi174 to 5696M 2024-09-15T21:52:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: osd.7 [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] boot 2024-09-15T21:52:23.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: osdmap e49: 8 total, 8 up, 8 in 2024-09-15T21:52:23.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:52:23.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:52:23.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:52:23.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:24.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:24 smithi174 ceph-mon[35244]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:52:24.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:24 smithi174 ceph-mon[35244]: osdmap e50: 8 total, 8 up, 8 in 2024-09-15T21:52:24.548 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:24 smithi203 ceph-mon[35546]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:52:24.548 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:24 smithi203 ceph-mon[35546]: osdmap e50: 8 total, 8 up, 8 in 2024-09-15T21:52:24.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:24 smithi028 ceph-mon[30969]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:52:24.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:24 smithi028 ceph-mon[30969]: osdmap e50: 8 total, 8 up, 8 in 2024-09-15T21:52:25.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:25 smithi203 ceph-mon[35546]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:25.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:25 smithi203 ceph-mon[35546]: osdmap e51: 8 total, 8 up, 8 in 2024-09-15T21:52:25.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:25 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/222290169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]: dispatch 2024-09-15T21:52:25.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:25 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]: dispatch 2024-09-15T21:52:25.594 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:25 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]': finished 2024-09-15T21:52:25.594 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:25 smithi203 ceph-mon[35546]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T21:52:25.594 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:25 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:25 smithi028 ceph-mon[30969]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:25 smithi028 ceph-mon[30969]: osdmap e51: 8 total, 8 up, 8 in 2024-09-15T21:52:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:25 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/222290169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]: dispatch 2024-09-15T21:52:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:25 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]: dispatch 2024-09-15T21:52:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:25 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]': finished 2024-09-15T21:52:25.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:25 smithi028 ceph-mon[30969]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T21:52:25.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:25 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:25 smithi174 ceph-mon[35244]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:25 smithi174 ceph-mon[35244]: osdmap e51: 8 total, 8 up, 8 in 2024-09-15T21:52:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:25 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/222290169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]: dispatch 2024-09-15T21:52:25.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:25 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]: dispatch 2024-09-15T21:52:25.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:25 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8"}]': finished 2024-09-15T21:52:25.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:25 smithi174 ceph-mon[35244]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T21:52:25.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:25 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:26.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:26 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/1831217764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:26.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:26 smithi028 ceph-mon[30969]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T21:52:26.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:26.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:26 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/1831217764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:26.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:26 smithi203 ceph-mon[35546]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T21:52:26.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:26.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:26 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/1831217764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:26.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:26 smithi174 ceph-mon[35244]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T21:52:26.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:27.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:27 smithi028 ceph-mon[30969]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:27.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:27 smithi203 ceph-mon[35546]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:27 smithi174 ceph-mon[35244]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:29.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:29 smithi028 ceph-mon[30969]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:29.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:29 smithi203 ceph-mon[35546]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:29.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:29 smithi174 ceph-mon[35244]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:31.647 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:31 smithi203 ceph-mon[35546]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:31.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:31 smithi028 ceph-mon[30969]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:31 smithi174 ceph-mon[35244]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:33 smithi028 ceph-mon[30969]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:33.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:33 smithi203 ceph-mon[35546]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:33.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:33 smithi174 ceph-mon[35244]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T21:52:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:34.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T21:52:34.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:34.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T21:52:34.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:35 smithi203 ceph-mon[35546]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:35 smithi203 ceph-mon[35546]: Deploying daemon osd.8 on smithi203 2024-09-15T21:52:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:35 smithi028 ceph-mon[30969]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:35 smithi028 ceph-mon[30969]: Deploying daemon osd.8 on smithi203 2024-09-15T21:52:35.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:35 smithi174 ceph-mon[35244]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:35.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:35 smithi174 ceph-mon[35244]: Deploying daemon osd.8 on smithi203 2024-09-15T21:52:37.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:37 smithi203 ceph-mon[35546]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:37 smithi028 ceph-mon[30969]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:37 smithi174 ceph-mon[35244]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:39.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:39 smithi203 ceph-mon[35546]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:39 smithi028 ceph-mon[30969]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:39 smithi174 ceph-mon[35244]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:40.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:40.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:40.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:41.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:41 smithi203 ceph-mon[35546]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:41.661 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:41 smithi174 ceph-mon[35244]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:41.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:41 smithi028 ceph-mon[30969]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:43.545 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:43 smithi203 ceph-mon[35546]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:43.545 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.545 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.545 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:43.545 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:43.545 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.585 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 8 on host 'smithi203' 2024-09-15T21:52:43.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.582+0000 7f92d37fe640 1 -- 172.21.15.203:0/2916590835 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f92e4062630 con 0x7f92cc0a3270 2024-09-15T21:52:43.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f92cc0a3270 msgr2=0x7f92cc0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:43.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f92cc0a3270 0x7f92cc0a5730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f92e41972d0 tx=0x7f92d800b390 comp rx=0 tx=0).stop 2024-09-15T21:52:43.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 msgr2=0x7f92e4195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:43.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4195db0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92d4005ab0 tx=0x7f92d40042d0 comp rx=0 tx=0).stop 2024-09-15T21:52:43.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 shutdown_connections 2024-09-15T21:52:43.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f92cc0a3270 0x7f92cc0a5730 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:43.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101e50 0x7f92e419d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:43.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92e4101490 0x7f92e41962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:43.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 --2- 172.21.15.203:0/2916590835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92e4100290 0x7f92e4195db0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:43.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 >> 172.21.15.203:0/2916590835 conn(0x7f92e40fba40 msgr2=0x7f92e4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:52:43.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 shutdown_connections 2024-09-15T21:52:43.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:52:43.585+0000 7f92e8bb9640 1 -- 172.21.15.203:0/2916590835 wait complete. 2024-09-15T21:52:43.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:43 smithi028 ceph-mon[30969]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:43.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:43.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:43.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:43 smithi174 ceph-mon[35244]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:44.287 DEBUG:teuthology.orchestra.run.smithi203:osd.8> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.8.service 2024-09-15T21:52:44.290 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-09-15T21:52:44.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T21:52:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:44 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:44 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:44.530 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:44 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:44.560 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:52:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:44 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:44 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:44 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:44.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:44 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:52:44.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:44 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:44.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:44 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:45.541 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:45 smithi203 ceph-mon[35546]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:45.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:45 smithi028 ceph-mon[30969]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:45.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:45 smithi174 ceph-mon[35244]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:45.859 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 21:52:45 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T21:52:45.655+0000 7f4548e1a640 -1 osd.8 0 log_to_monitors true 2024-09-15T21:52:46.585 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:46 smithi028 ceph-mon[30969]: from='osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T21:52:46.585 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:46 smithi028 ceph-mon[30969]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T21:52:46.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:46 smithi203 ceph-mon[35546]: from='osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T21:52:46.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:46 smithi203 ceph-mon[35546]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T21:52:46.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:46 smithi174 ceph-mon[35244]: from='osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T21:52:46.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:46 smithi174 ceph-mon[35244]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T21:52:47.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:52:47.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-09-15T21:52:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:47 smithi028 ceph-mon[30969]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:47 smithi028 ceph-mon[30969]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T21:52:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:47 smithi028 ceph-mon[30969]: osdmap e54: 9 total, 8 up, 9 in 2024-09-15T21:52:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:47 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:47 smithi028 ceph-mon[30969]: from='osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:52:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:47 smithi028 ceph-mon[30969]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:52:47.679 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:52:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:47 smithi203 ceph-mon[35546]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:47 smithi203 ceph-mon[35546]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T21:52:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:47 smithi203 ceph-mon[35546]: osdmap e54: 9 total, 8 up, 9 in 2024-09-15T21:52:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:47 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:47 smithi203 ceph-mon[35546]: from='osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:52:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:47 smithi203 ceph-mon[35546]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:52:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:47 smithi174 ceph-mon[35244]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:47 smithi174 ceph-mon[35244]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T21:52:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:47 smithi174 ceph-mon[35244]: osdmap e54: 9 total, 8 up, 9 in 2024-09-15T21:52:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:47 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:47 smithi174 ceph-mon[35244]: from='osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:52:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:47 smithi174 ceph-mon[35244]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 -- 172.21.15.28:0/3729609858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 msgr2=0x7f862c10a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3729609858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c10a050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8624009ca0 tx=0x7f862402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 -- 172.21.15.28:0/3729609858 shutdown_connections 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3729609858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c10a050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3729609858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f862c102e00 0x7f862c103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3729609858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c101c00 0x7f862c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 -- 172.21.15.28:0/3729609858 >> 172.21.15.28:0/3729609858 conn(0x7f862c0fd390 msgr2=0x7f862c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 -- 172.21.15.28:0/3729609858 shutdown_connections 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.159+0000 7f8632ab9640 1 -- 172.21.15.28:0/3729609858 wait complete. 2024-09-15T21:52:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 Processor -- start 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 -- start start 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c101c00 0x7f862c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f862c102e00 0x7f862c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f862c10d880 con 0x7f862c1037c0 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f862c10d700 con 0x7f862c101c00 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8632ab9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f862c10da00 con 0x7f862c102e00 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8631056640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8631056640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c19f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50556/0 (socket says 172.21.15.28:50556) 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f8631056640 1 -- 172.21.15.28:0/3608511644 learned_addr learned my addr 172.21.15.28:0/3608511644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:52:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.160+0000 7f862bfff640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f862c102e00 0x7f862c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8630855640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c101c00 0x7f862c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8631056640 1 -- 172.21.15.28:0/3608511644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f862c102e00 msgr2=0x7f862c1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8631056640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f862c102e00 0x7f862c1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8631056640 1 -- 172.21.15.28:0/3608511644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c101c00 msgr2=0x7f862c197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8631056640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c101c00 0x7f862c197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8631056640 1 -- 172.21.15.28:0/3608511644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c19fad0 con 0x7f862c1037c0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8631056640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c19f460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8624031ae0 tx=0x7f8624031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8629ffb640 1 -- 172.21.15.28:0/3608511644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624003c30 con 0x7f862c1037c0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8629ffb640 1 -- 172.21.15.28:0/3608511644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8624003dd0 con 0x7f862c1037c0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862c19fd60 con 0x7f862c1037c0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8629ffb640 1 -- 172.21.15.28:0/3608511644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862402fcb0 con 0x7f862c1037c0 2024-09-15T21:52:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.161+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f862c1a0260 con 0x7f862c1037c0 2024-09-15T21:52:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.162+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f862c108f80 con 0x7f862c1037c0 2024-09-15T21:52:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.163+0000 7f8629ffb640 1 -- 172.21.15.28:0/3608511644 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f8624004050 con 0x7f862c1037c0 2024-09-15T21:52:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.164+0000 7f8629ffb640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f861c0a32f0 0x7f861c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:52:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.164+0000 7f8630855640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f861c0a32f0 0x7f861c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:52:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.164+0000 7f8629ffb640 1 -- 172.21.15.28:0/3608511644 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5549+0+0 (secure 0 0 0) 0x7f86240f0370 con 0x7f862c1037c0 2024-09-15T21:52:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.164+0000 7f8630855640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f861c0a32f0 0x7f861c0a57b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8620005d00 tx=0x7f8620005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:52:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.167+0000 7f8629ffb640 1 -- 172.21.15.28:0/3608511644 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f86240bdac0 con 0x7f862c1037c0 2024-09-15T21:52:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:52:48.355+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f862c062630 con 0x7f861c0a32f0 2024-09-15T21:52:48.386 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T21:52:48.046+0000 7f4544d9d640 -1 osd.8 0 waiting for initial osdmap 2024-09-15T21:52:48.387 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T21:52:48.056+0000 7f453cbbd640 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:52:48.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:52:48.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: osdmap e55: 9 total, 8 up, 9 in 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi203 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi203 to 5696M 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:48.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:48.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='osd.8 ' entity='osd.8' 2024-09-15T21:52:48.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:52:48.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:52:48.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:48 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:48.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:52:48.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: osdmap e55: 9 total, 8 up, 9 in 2024-09-15T21:52:48.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:48.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:48.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi203 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi203 to 5696M 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='osd.8 ' entity='osd.8' 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:52:48.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:52:48.720 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:48 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:48.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:52:48.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: osdmap e55: 9 total, 8 up, 9 in 2024-09-15T21:52:48.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:48.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi203 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi203 to 5696M 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:52:48.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:52:48.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='osd.8 ' entity='osd.8' 2024-09-15T21:52:48.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:52:48.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:52:48.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:48 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:52:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:52:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:52:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] boot 2024-09-15T21:52:49.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: osdmap e56: 9 total, 9 up, 9 in 2024-09-15T21:52:49.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:52:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:52:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:52:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] boot 2024-09-15T21:52:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: osdmap e56: 9 total, 9 up, 9 in 2024-09-15T21:52:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: osd.8 [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] boot 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: osdmap e56: 9 total, 9 up, 9 in 2024-09-15T21:52:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:52:50.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:50 smithi028 ceph-mon[30969]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T21:52:50.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:50 smithi203 ceph-mon[35546]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T21:52:50.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:50 smithi174 ceph-mon[35244]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T21:52:51.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:51 smithi028 ceph-mon[30969]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:52:51.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:51 smithi028 ceph-mon[30969]: osdmap e58: 9 total, 9 up, 9 in 2024-09-15T21:52:51.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:51 smithi028 ceph-mon[30969]: osdmap e59: 9 total, 9 up, 9 in 2024-09-15T21:52:51.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:51 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4051352240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe08e2a-856e-4836-bfa3-fa269e447a9f"}]: dispatch 2024-09-15T21:52:51.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:51 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4051352240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe08e2a-856e-4836-bfa3-fa269e447a9f"}]': finished 2024-09-15T21:52:51.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:51 smithi028 ceph-mon[30969]: osdmap e60: 10 total, 9 up, 10 in 2024-09-15T21:52:51.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:52:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:51 smithi203 ceph-mon[35546]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:52:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:51 smithi203 ceph-mon[35546]: osdmap e58: 9 total, 9 up, 9 in 2024-09-15T21:52:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:51 smithi203 ceph-mon[35546]: osdmap e59: 9 total, 9 up, 9 in 2024-09-15T21:52:51.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:51 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4051352240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe08e2a-856e-4836-bfa3-fa269e447a9f"}]: dispatch 2024-09-15T21:52:51.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:51 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4051352240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe08e2a-856e-4836-bfa3-fa269e447a9f"}]': finished 2024-09-15T21:52:51.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:51 smithi203 ceph-mon[35546]: osdmap e60: 10 total, 9 up, 10 in 2024-09-15T21:52:51.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:52:51.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:51 smithi174 ceph-mon[35244]: pgmap v193: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:52:51.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:51 smithi174 ceph-mon[35244]: osdmap e58: 9 total, 9 up, 9 in 2024-09-15T21:52:51.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:51 smithi174 ceph-mon[35244]: osdmap e59: 9 total, 9 up, 9 in 2024-09-15T21:52:51.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:51 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4051352240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe08e2a-856e-4836-bfa3-fa269e447a9f"}]: dispatch 2024-09-15T21:52:51.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:51 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4051352240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe08e2a-856e-4836-bfa3-fa269e447a9f"}]': finished 2024-09-15T21:52:51.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:51 smithi174 ceph-mon[35244]: osdmap e60: 10 total, 9 up, 10 in 2024-09-15T21:52:51.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:52:52.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:52 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3372323203' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:52.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:52 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3372323203' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:52.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:52 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3372323203' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:52:53.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:53 smithi028 ceph-mon[30969]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:52:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:53 smithi203 ceph-mon[35546]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:52:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:53 smithi174 ceph-mon[35244]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:52:55.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:55 smithi028 ceph-mon[30969]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-15T21:52:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:55 smithi203 ceph-mon[35546]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-15T21:52:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:55 smithi174 ceph-mon[35244]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-09-15T21:52:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:57 smithi028 ceph-mon[30969]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:52:57.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:57 smithi203 ceph-mon[35546]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:52:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:57 smithi174 ceph-mon[35244]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:52:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:59 smithi028 ceph-mon[30969]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-15T21:52:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T21:52:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:52:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:59 smithi203 ceph-mon[35546]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-15T21:52:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T21:52:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:52:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:52:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:59 smithi174 ceph-mon[35244]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-15T21:52:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T21:52:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:52:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:00.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:00 smithi203 ceph-mon[35546]: Deploying daemon osd.9 on smithi028 2024-09-15T21:53:00.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:00 smithi174 ceph-mon[35244]: Deploying daemon osd.9 on smithi028 2024-09-15T21:53:00.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:00 smithi028 ceph-mon[30969]: Deploying daemon osd.9 on smithi028 2024-09-15T21:53:01.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:01 smithi028 ceph-mon[30969]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-15T21:53:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:01 smithi203 ceph-mon[35546]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-15T21:53:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:01 smithi174 ceph-mon[35244]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-15T21:53:03.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:03 smithi028 ceph-mon[30969]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-15T21:53:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:03 smithi203 ceph-mon[35546]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-15T21:53:03.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:03 smithi174 ceph-mon[35244]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-15T21:53:05.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:05 smithi203 ceph-mon[35546]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:53:05.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:05 smithi174 ceph-mon[35244]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:53:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:05.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:05 smithi028 ceph-mon[30969]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T21:53:05.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:06.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:06 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:06.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:06 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:06.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:06 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:06.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:06 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:06.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:06 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:06.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:06 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:07.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:07 smithi028 ceph-mon[30969]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:07 smithi203 ceph-mon[35546]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:07 smithi174 ceph-mon[35244]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.851+0000 7f8629ffb640 1 -- 172.21.15.28:0/3608511644 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f862c062630 con 0x7f861c0a32f0 2024-09-15T21:53:08.854 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-09-15T21:53:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f861c0a32f0 msgr2=0x7f861c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f861c0a32f0 0x7f861c0a57b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8620005d00 tx=0x7f8620005c50 comp rx=0 tx=0).stop 2024-09-15T21:53:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 msgr2=0x7f862c19f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c19f460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8624031ae0 tx=0x7f8624031ee0 comp rx=0 tx=0).stop 2024-09-15T21:53:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 shutdown_connections 2024-09-15T21:53:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f861c0a32f0 0x7f861c0a57b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862c1037c0 0x7f862c19f460 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f862c102e00 0x7f862c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 --2- 172.21.15.28:0/3608511644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c101c00 0x7f862c197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 >> 172.21.15.28:0/3608511644 conn(0x7f862c0fd390 msgr2=0x7f862c106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:53:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 shutdown_connections 2024-09-15T21:53:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:53:08.855+0000 7f8632ab9640 1 -- 172.21.15.28:0/3608511644 wait complete. 2024-09-15T21:53:09.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:09 smithi028 ceph-mon[30969]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:09.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:09.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:09 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:09.651 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.9.service 2024-09-15T21:53:09.654 INFO:tasks.cephadm:Deploying osd.10 on smithi174 with /dev/vg_nvme/lv_1... 2024-09-15T21:53:09.654 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T21:53:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:09 smithi203 ceph-mon[35546]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:09 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:09 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:09 smithi174 ceph-mon[35244]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:09 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:09 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:09.910 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:53:11.111 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:10 smithi174 ceph-mon[35244]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:11.111 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:10 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:11.112 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:10 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:11.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:10 smithi028 ceph-mon[30969]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:11.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:10 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:11.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:10 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:11.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:10 smithi203 ceph-mon[35546]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:11.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:10 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:11.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:10 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:11.636 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 21:53:11 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T21:53:11.279+0000 7fe0f4107640 -1 osd.9 0 log_to_monitors true 2024-09-15T21:53:11.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:11 smithi028 ceph-mon[30969]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T21:53:12.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:11 smithi203 ceph-mon[35546]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T21:53:12.218 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:11 smithi174 ceph-mon[35244]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T21:53:12.522 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T21:53:12.549 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_1 2024-09-15T21:53:12.808 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T21:53:13.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:12 smithi028 ceph-mon[30969]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:13.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:12 smithi028 ceph-mon[30969]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T21:53:13.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:12 smithi028 ceph-mon[30969]: osdmap e61: 10 total, 9 up, 10 in 2024-09-15T21:53:13.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:12 smithi028 ceph-mon[30969]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:53:13.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:12 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:13.091 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:12 smithi174 ceph-mon[35244]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:13.091 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:12 smithi174 ceph-mon[35244]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T21:53:13.091 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:12 smithi174 ceph-mon[35244]: osdmap e61: 10 total, 9 up, 10 in 2024-09-15T21:53:13.091 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:12 smithi174 ceph-mon[35244]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:53:13.092 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:12 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:13.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:12 smithi203 ceph-mon[35546]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:13.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:12 smithi203 ceph-mon[35546]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T21:53:13.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:12 smithi203 ceph-mon[35546]: osdmap e61: 10 total, 9 up, 10 in 2024-09-15T21:53:13.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:12 smithi203 ceph-mon[35546]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-09-15T21:53:13.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:12 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:13.341 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 -- 172.21.15.174:0/784698104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278102690 msgr2=0x7fd278108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:13.341 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 --2- 172.21.15.174:0/784698104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278102690 0x7fd278108f20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd270009ca0 tx=0x7fd27002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:53:13.341 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 -- 172.21.15.174:0/784698104 shutdown_connections 2024-09-15T21:53:13.341 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 --2- 172.21.15.174:0/784698104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278102690 0x7fd278108f20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:13.341 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 --2- 172.21.15.174:0/784698104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd278101cd0 0x7fd278102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:13.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 --2- 172.21.15.174:0/784698104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd278100ad0 0x7fd278100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:13.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 -- 172.21.15.174:0/784698104 >> 172.21.15.174:0/784698104 conn(0x7fd2780fc280 msgr2=0x7fd2780fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:53:13.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 -- 172.21.15.174:0/784698104 shutdown_connections 2024-09-15T21:53:13.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.315+0000 7fd27e9be640 1 -- 172.21.15.174:0/784698104 wait complete. 2024-09-15T21:53:13.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.316+0000 7fd27e9be640 1 Processor -- start 2024-09-15T21:53:13.343 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.316+0000 7fd27e9be640 1 -- start start 2024-09-15T21:53:13.343 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.316+0000 7fd27e9be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278100ad0 0x7fd278197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:13.343 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.316+0000 7fd27e9be640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd278101cd0 0x7fd278198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:13.343 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd27e9be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd278102690 0x7fd27819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:13.343 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd27e9be640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd27810c730 con 0x7fd278102690 2024-09-15T21:53:13.344 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd27e9be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd27810c5b0 con 0x7fd278100ad0 2024-09-15T21:53:13.344 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd27e9be640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd27810c8b0 con 0x7fd278101cd0 2024-09-15T21:53:13.344 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278100ad0 0x7fd278197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:13.344 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd2777fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd278101cd0 0x7fd278198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:13.344 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278100ad0 0x7fd278197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:39724/0 (socket says 172.21.15.174:39724) 2024-09-15T21:53:13.345 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 -- 172.21.15.174:0/2739634912 learned_addr learned my addr 172.21.15.174:0/2739634912 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T21:53:13.345 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd27cf5b640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd278102690 0x7fd27819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:13.345 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 -- 172.21.15.174:0/2739634912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd278101cd0 msgr2=0x7fd278198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:13.345 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd278101cd0 0x7fd278198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:13.345 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 -- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd278102690 msgr2=0x7fd27819f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:13.345 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd278102690 0x7fd27819f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:13.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 -- 172.21.15.174:0/2739634912 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27819fa80 con 0x7fd278100ad0 2024-09-15T21:53:13.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd2777fe640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd278101cd0 0x7fd278198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:53:13.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd277fff640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278100ad0 0x7fd278197e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd26800d950 tx=0x7fd26800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:53:13.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd2757fa640 1 -- 172.21.15.174:0/2739634912 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268017020 con 0x7fd278100ad0 2024-09-15T21:53:13.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27819fd70 con 0x7fd278100ad0 2024-09-15T21:53:13.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd2757fa640 1 -- 172.21.15.174:0/2739634912 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd268009c30 con 0x7fd278100ad0 2024-09-15T21:53:13.347 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.317+0000 7fd2757fa640 1 -- 172.21.15.174:0/2739634912 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26800f740 con 0x7fd278100ad0 2024-09-15T21:53:13.347 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.318+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2781a0220 con 0x7fd278100ad0 2024-09-15T21:53:13.347 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.319+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2781a04b0 con 0x7fd278100ad0 2024-09-15T21:53:13.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.320+0000 7fd2757fa640 1 -- 172.21.15.174:0/2739634912 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd26801a020 con 0x7fd278100ad0 2024-09-15T21:53:13.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.321+0000 7fd2757fa640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2600a3270 0x7fd2600a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:13.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.321+0000 7fd2757fa640 1 -- 172.21.15.174:0/2739634912 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5981+0+0 (secure 0 0 0) 0x7fd2680ce950 con 0x7fd278100ad0 2024-09-15T21:53:13.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.321+0000 7fd2777fe640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2600a3270 0x7fd2600a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:13.348 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.321+0000 7fd2777fe640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2600a3270 0x7fd2600a5730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd278199370 tx=0x7fd26c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:53:13.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.325+0000 7fd2757fa640 1 -- 172.21.15.174:0/2739634912 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd268021070 con 0x7fd278100ad0 2024-09-15T21:53:13.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:13.510+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd2780fe4b0 con 0x7fd2600a3270 2024-09-15T21:53:13.671 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T21:53:13.390+0000 7fe0f008a640 -1 osd.9 0 waiting for initial osdmap 2024-09-15T21:53:13.671 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T21:53:13.400+0000 7fe0e7eaa640 -1 osd.9 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:53:14.082 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:53:14.082 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: osdmap e62: 10 total, 9 up, 10 in 2024-09-15T21:53:14.082 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.082 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.082 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi028 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi028 to 4273M 2024-09-15T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' 2024-09-15T21:53:14.084 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:53:14.085 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:53:14.085 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.085 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:13 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:53:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: osdmap e62: 10 total, 9 up, 10 in 2024-09-15T21:53:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi028 2024-09-15T21:53:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi028 to 4273M 2024-09-15T21:53:14.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:14.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:14.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:14.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' 2024-09-15T21:53:14.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:53:14.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:53:14.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:13 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-09-15T21:53:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: osdmap e62: 10 total, 9 up, 10 in 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi028 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi028 to 4273M 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678]' entity='osd.9' 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:13 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:14.989 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:14 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:53:14.989 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:14 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:53:14.989 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:14 smithi174 ceph-mon[35244]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:53:14.989 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:14 smithi174 ceph-mon[35244]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:14.989 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:14 smithi174 ceph-mon[35244]: osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] boot 2024-09-15T21:53:14.990 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:14 smithi174 ceph-mon[35244]: osdmap e63: 10 total, 10 up, 10 in 2024-09-15T21:53:14.990 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:15.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:14 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:53:15.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:14 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:53:15.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:14 smithi028 ceph-mon[30969]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:53:15.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:14 smithi028 ceph-mon[30969]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:15.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:14 smithi028 ceph-mon[30969]: osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] boot 2024-09-15T21:53:15.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:14 smithi028 ceph-mon[30969]: osdmap e63: 10 total, 10 up, 10 in 2024-09-15T21:53:15.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:14 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:53:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:14 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:53:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:14 smithi203 ceph-mon[35546]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:53:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:14 smithi203 ceph-mon[35546]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-15T21:53:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:14 smithi203 ceph-mon[35546]: osd.9 [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] boot 2024-09-15T21:53:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:14 smithi203 ceph-mon[35546]: osdmap e63: 10 total, 10 up, 10 in 2024-09-15T21:53:15.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:53:17.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:17.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: osdmap e64: 10 total, 10 up, 10 in 2024-09-15T21:53:17.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/1361046131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]: dispatch 2024-09-15T21:53:17.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]: dispatch 2024-09-15T21:53:17.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]': finished 2024-09-15T21:53:17.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: osdmap e65: 11 total, 10 up, 11 in 2024-09-15T21:53:17.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:17.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:16 smithi174 ceph-mon[35244]: from='client.? 172.21.15.174:0/2014574461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:53:17.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:17.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: osdmap e64: 10 total, 10 up, 10 in 2024-09-15T21:53:17.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/1361046131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]: dispatch 2024-09-15T21:53:17.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]: dispatch 2024-09-15T21:53:17.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]': finished 2024-09-15T21:53:17.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: osdmap e65: 11 total, 10 up, 11 in 2024-09-15T21:53:17.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:17.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:16 smithi028 ceph-mon[30969]: from='client.? 172.21.15.174:0/2014574461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:53:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: osdmap e64: 10 total, 10 up, 10 in 2024-09-15T21:53:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/1361046131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]: dispatch 2024-09-15T21:53:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]: dispatch 2024-09-15T21:53:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff5afcc0-0e93-459e-87a8-1a34428aef91"}]': finished 2024-09-15T21:53:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: osdmap e65: 11 total, 10 up, 11 in 2024-09-15T21:53:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:17.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:16 smithi203 ceph-mon[35546]: from='client.? 172.21.15.174:0/2014574461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:53:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:18 smithi028 ceph-mon[30969]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:18 smithi203 ceph-mon[35546]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:19.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:18 smithi174 ceph-mon[35244]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:20 smithi028 ceph-mon[30969]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:21.207 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:20 smithi174 ceph-mon[35244]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:21.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:20 smithi203 ceph-mon[35546]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:23.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:22 smithi174 ceph-mon[35244]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:23.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:22 smithi028 ceph-mon[30969]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:23.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:22 smithi203 ceph-mon[35546]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:25.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:24 smithi028 ceph-mon[30969]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:24 smithi203 ceph-mon[35546]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:25.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:24 smithi174 ceph-mon[35244]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:25 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T21:53:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:25 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:25 smithi028 ceph-mon[30969]: Deploying daemon osd.10 on smithi174 2024-09-15T21:53:26.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:25 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T21:53:26.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:25 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:26.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:25 smithi203 ceph-mon[35546]: Deploying daemon osd.10 on smithi174 2024-09-15T21:53:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:25 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T21:53:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:25 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:25 smithi174 ceph-mon[35244]: Deploying daemon osd.10 on smithi174 2024-09-15T21:53:27.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:26 smithi174 ceph-mon[35244]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:27.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:26 smithi028 ceph-mon[30969]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:26 smithi203 ceph-mon[35546]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:29.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:28 smithi028 ceph-mon[30969]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:28 smithi203 ceph-mon[35546]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:28 smithi174 ceph-mon[35244]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:30.910 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:30 smithi174 ceph-mon[35244]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:31.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:30 smithi028 ceph-mon[30969]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:30 smithi203 ceph-mon[35546]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:31 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:31 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:31 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:31 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:31 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:31 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:31 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:31 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:31 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:33.315 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:33 smithi174 ceph-mon[35244]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:33.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:33 smithi028 ceph-mon[30969]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:33.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:33 smithi203 ceph-mon[35546]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:34.163 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.161+0000 7fd2757fa640 1 -- 172.21.15.174:0/2739634912 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd2780fe4b0 con 0x7fd2600a3270 2024-09-15T21:53:34.164 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 10 on host 'smithi174' 2024-09-15T21:53:34.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.163+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2600a3270 msgr2=0x7fd2600a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:34.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.163+0000 7fd27e9be640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2600a3270 0x7fd2600a5730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd278199370 tx=0x7fd26c00b410 comp rx=0 tx=0).stop 2024-09-15T21:53:34.166 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.163+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278100ad0 msgr2=0x7fd278197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:34.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.163+0000 7fd27e9be640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278100ad0 0x7fd278197e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd26800d950 tx=0x7fd26800de10 comp rx=0 tx=0).stop 2024-09-15T21:53:34.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.163+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 shutdown_connections 2024-09-15T21:53:34.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.163+0000 7fd27e9be640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2600a3270 0x7fd2600a5730 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:34.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.163+0000 7fd27e9be640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd278102690 0x7fd27819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:34.167 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.164+0000 7fd27e9be640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd278101cd0 0x7fd278198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:34.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.164+0000 7fd27e9be640 1 --2- 172.21.15.174:0/2739634912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd278100ad0 0x7fd278197e50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:34.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.164+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 >> 172.21.15.174:0/2739634912 conn(0x7fd2780fc280 msgr2=0x7fd2780fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:53:34.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.164+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 shutdown_connections 2024-09-15T21:53:34.168 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T21:53:34.164+0000 7fd27e9be640 1 -- 172.21.15.174:0/2739634912 wait complete. 2024-09-15T21:53:35.180 DEBUG:teuthology.orchestra.run.smithi174:osd.10> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.10.service 2024-09-15T21:53:35.182 INFO:tasks.cephadm:Deploying osd.11 on smithi203 with /dev/vg_nvme/lv_1... 2024-09-15T21:53:35.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T21:53:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:35 smithi028 ceph-mon[30969]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:35 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:35 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:35.440 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:53:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:35 smithi203 ceph-mon[35546]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:35 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:35 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:35.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:35 smithi174 ceph-mon[35244]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:35.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:35 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:35.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:35 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:36.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:36.669 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:36.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:36.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:36.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:36.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:37.346 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 21:53:37 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T21:53:37.076+0000 7f77ad40e640 -1 osd.10 0 log_to_monitors true 2024-09-15T21:53:37.548 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:37 smithi203 ceph-mon[35546]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:37.548 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:37 smithi203 ceph-mon[35546]: from='osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T21:53:37.549 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:37 smithi203 ceph-mon[35546]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T21:53:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:37 smithi028 ceph-mon[30969]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:37 smithi028 ceph-mon[30969]: from='osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T21:53:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:37 smithi028 ceph-mon[30969]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T21:53:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:37 smithi174 ceph-mon[35244]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:37 smithi174 ceph-mon[35244]: from='osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T21:53:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:37 smithi174 ceph-mon[35244]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T21:53:38.354 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T21:53:38.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-09-15T21:53:38.635 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:53:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:38 smithi028 ceph-mon[30969]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T21:53:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:38 smithi028 ceph-mon[30969]: osdmap e66: 11 total, 10 up, 11 in 2024-09-15T21:53:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:38 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:38 smithi028 ceph-mon[30969]: from='osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:53:38.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:38 smithi028 ceph-mon[30969]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:53:38.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:38 smithi203 ceph-mon[35546]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T21:53:38.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:38 smithi203 ceph-mon[35546]: osdmap e66: 11 total, 10 up, 11 in 2024-09-15T21:53:38.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:38 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:38.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:38 smithi203 ceph-mon[35546]: from='osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:53:38.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:38 smithi203 ceph-mon[35546]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:53:38.745 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 21:53:38 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T21:53:38.361+0000 7f77a9391640 -1 osd.10 0 waiting for initial osdmap 2024-09-15T21:53:38.745 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 21:53:38 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T21:53:38.370+0000 7f77a51b9640 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:53:38.745 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:38 smithi174 ceph-mon[35244]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T21:53:38.746 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:38 smithi174 ceph-mon[35244]: osdmap e66: 11 total, 10 up, 11 in 2024-09-15T21:53:38.746 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:38 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:38.746 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:38 smithi174 ceph-mon[35244]: from='osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:53:38.746 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:38 smithi174 ceph-mon[35244]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T21:53:39.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 -- 172.21.15.203:0/956028485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 msgr2=0x7fe9fc109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:39.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 --2- 172.21.15.203:0/956028485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc109d30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009ca0 tx=0x7fe9f402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 -- 172.21.15.203:0/956028485 shutdown_connections 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 --2- 172.21.15.203:0/956028485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc109d30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 --2- 172.21.15.203:0/956028485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc105010 0x7fe9fc107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 --2- 172.21.15.203:0/956028485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc1026e0 0x7fe9fc104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 -- 172.21.15.203:0/956028485 >> 172.21.15.203:0/956028485 conn(0x7fe9fc0fc280 msgr2=0x7fe9fc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 -- 172.21.15.203:0/956028485 shutdown_connections 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 -- 172.21.15.203:0/956028485 wait complete. 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.120+0000 7fea0331a640 1 Processor -- start 2024-09-15T21:53:39.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea0331a640 1 -- start start 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea0331a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc1026e0 0x7fe9fc197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea0331a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc105010 0x7fe9fc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea0331a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea0331a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10d880 con 0x7fe9fc105010 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea0331a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10d700 con 0x7fe9fc1026e0 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea0331a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10da00 con 0x7fe9fc107940 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc19f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:45922/0 (socket says 172.21.15.203:45922) 2024-09-15T21:53:39.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 -- 172.21.15.203:0/2462771262 learned_addr learned my addr 172.21.15.203:0/2462771262 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea010b6640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc1026e0 0x7fe9fc197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea008b5640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc105010 0x7fe9fc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 -- 172.21.15.203:0/2462771262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc1026e0 msgr2=0x7fe9fc197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc1026e0 0x7fe9fc197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 -- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc105010 msgr2=0x7fe9fc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc105010 0x7fe9fc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 -- 172.21.15.203:0/2462771262 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc19fb00 con 0x7fe9fc107940 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fea018b7640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc19f490 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4005e30 tx=0x7fe9f4031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.121+0000 7fe9f27fc640 1 -- 172.21.15.203:0/2462771262 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4003c30 con 0x7fe9fc107940 2024-09-15T21:53:39.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.122+0000 7fe9f27fc640 1 -- 172.21.15.203:0/2462771262 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f4003dd0 con 0x7fe9fc107940 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.122+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc19fd90 con 0x7fe9fc107940 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.122+0000 7fe9f27fc640 1 -- 172.21.15.203:0/2462771262 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f402fde0 con 0x7fe9fc107940 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.122+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc1a0270 con 0x7fe9fc107940 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.123+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9fc0fdd70 con 0x7fe9fc107940 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.124+0000 7fe9f27fc640 1 -- 172.21.15.203:0/2462771262 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe9f4004050 con 0x7fe9fc107940 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.124+0000 7fe9f27fc640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe9e80a3270 0x7fe9e80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.124+0000 7fe9f27fc640 1 -- 172.21.15.203:0/2462771262 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6414+0+0 (secure 0 0 0) 0x7fe9f40f0320 con 0x7fe9fc107940 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.124+0000 7fea010b6640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe9e80a3270 0x7fe9e80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:53:39.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.125+0000 7fea010b6640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe9e80a3270 0x7fe9e80a5730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8005d00 tx=0x7fe9f800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:53:39.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.129+0000 7fe9f27fc640 1 -- 172.21.15.203:0/2462771262 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe9f40bd7b0 con 0x7fe9fc107940 2024-09-15T21:53:39.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:53:39.304+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe9fc062630 con 0x7fe9e80a3270 2024-09-15T21:53:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:53:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: osdmap e67: 11 total, 10 up, 11 in 2024-09-15T21:53:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:39.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:39.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:53:39.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:53:39.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:39 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: osdmap e67: 11 total, 10 up, 11 in 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:53:39.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:39 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-09-15T21:53:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T21:53:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: osdmap e67: 11 total, 10 up, 11 in 2024-09-15T21:53:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:39.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:39.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:39.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:53:39.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:39.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T21:53:39.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T21:53:39.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:39 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:53:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:53:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi174 2024-09-15T21:53:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi174 to 4272M 2024-09-15T21:53:40.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:53:40.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] boot 2024-09-15T21:53:40.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: osdmap e68: 11 total, 11 up, 11 in 2024-09-15T21:53:40.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi174 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi174 to 4272M 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] boot 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: osdmap e68: 11 total, 11 up, 11 in 2024-09-15T21:53:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:53:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:53:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi174 2024-09-15T21:53:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi174 to 4272M 2024-09-15T21:53:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:53:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: osd.10 [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] boot 2024-09-15T21:53:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: osdmap e68: 11 total, 11 up, 11 in 2024-09-15T21:53:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:53:41.519 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:41 smithi203 ceph-mon[35546]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:41.519 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:41 smithi203 ceph-mon[35546]: osdmap e69: 11 total, 11 up, 11 in 2024-09-15T21:53:41.661 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:41 smithi174 ceph-mon[35244]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:41.661 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:41 smithi174 ceph-mon[35244]: osdmap e69: 11 total, 11 up, 11 in 2024-09-15T21:53:41.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:41 smithi028 ceph-mon[30969]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:41.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:41 smithi028 ceph-mon[30969]: osdmap e69: 11 total, 11 up, 11 in 2024-09-15T21:53:42.566 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:42 smithi203 ceph-mon[35546]: osdmap e70: 11 total, 11 up, 11 in 2024-09-15T21:53:42.566 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:42 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/1681670403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]: dispatch 2024-09-15T21:53:42.566 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:42 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]: dispatch 2024-09-15T21:53:42.566 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:42 smithi203 ceph-mon[35546]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]': finished 2024-09-15T21:53:42.566 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:42 smithi203 ceph-mon[35546]: osdmap e71: 12 total, 11 up, 12 in 2024-09-15T21:53:42.566 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:42 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:42 smithi028 ceph-mon[30969]: osdmap e70: 11 total, 11 up, 11 in 2024-09-15T21:53:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:42 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/1681670403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]: dispatch 2024-09-15T21:53:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:42 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]: dispatch 2024-09-15T21:53:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:42 smithi028 ceph-mon[30969]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]': finished 2024-09-15T21:53:42.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:42 smithi028 ceph-mon[30969]: osdmap e71: 12 total, 11 up, 12 in 2024-09-15T21:53:42.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:42 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:42.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:42 smithi174 ceph-mon[35244]: osdmap e70: 11 total, 11 up, 11 in 2024-09-15T21:53:42.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:42 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/1681670403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]: dispatch 2024-09-15T21:53:42.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:42 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]: dispatch 2024-09-15T21:53:42.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:42 smithi174 ceph-mon[35244]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3393ee20-fde1-48f1-b9f8-407c434f1386"}]': finished 2024-09-15T21:53:42.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:42 smithi174 ceph-mon[35244]: osdmap e71: 12 total, 11 up, 12 in 2024-09-15T21:53:42.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:42 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:43.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:43 smithi028 ceph-mon[30969]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:43.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:43 smithi028 ceph-mon[30969]: from='client.? 172.21.15.203:0/1637979788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:53:43.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:43 smithi028 ceph-mon[30969]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T21:53:43.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:43 smithi203 ceph-mon[35546]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:43 smithi203 ceph-mon[35546]: from='client.? 172.21.15.203:0/1637979788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:53:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:43 smithi203 ceph-mon[35546]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T21:53:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:43 smithi174 ceph-mon[35244]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:43 smithi174 ceph-mon[35244]: from='client.? 172.21.15.203:0/1637979788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T21:53:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:43 smithi174 ceph-mon[35244]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T21:53:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:45.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:44 smithi174 ceph-mon[35244]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-15T21:53:45.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:44 smithi174 ceph-mon[35244]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T21:53:45.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:44 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:45.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:44 smithi028 ceph-mon[30969]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-15T21:53:45.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:44 smithi028 ceph-mon[30969]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T21:53:45.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:44 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:45.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:44 smithi203 ceph-mon[35546]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-15T21:53:45.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:44 smithi203 ceph-mon[35546]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T21:53:45.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:44 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:53:47.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:46 smithi203 ceph-mon[35546]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-09-15T21:53:47.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:46 smithi174 ceph-mon[35244]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-09-15T21:53:47.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:46 smithi028 ceph-mon[30969]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-09-15T21:53:49.099 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:48 smithi203 ceph-mon[35546]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:53:49.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:48 smithi174 ceph-mon[35244]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:53:49.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:48 smithi028 ceph-mon[30969]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T21:53:51.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:50 smithi203 ceph-mon[35546]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-15T21:53:51.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T21:53:51.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:50 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:51.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:50 smithi174 ceph-mon[35244]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-15T21:53:51.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T21:53:51.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:50 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:50 smithi028 ceph-mon[30969]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-15T21:53:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T21:53:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:50 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:53:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:51 smithi174 ceph-mon[35244]: Deploying daemon osd.11 on smithi203 2024-09-15T21:53:52.312 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:51 smithi203 ceph-mon[35546]: Deploying daemon osd.11 on smithi203 2024-09-15T21:53:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:51 smithi028 ceph-mon[30969]: Deploying daemon osd.11 on smithi203 2024-09-15T21:53:53.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:53 smithi203 ceph-mon[35546]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-15T21:53:53.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:53 smithi174 ceph-mon[35244]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-15T21:53:53.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:53 smithi028 ceph-mon[30969]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-15T21:53:55.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:55 smithi028 ceph-mon[30969]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:55.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:55 smithi203 ceph-mon[35546]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:55.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:55 smithi174 ceph-mon[35244]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:57.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:57 smithi028 ceph-mon[30969]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:57 smithi203 ceph-mon[35546]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:57 smithi174 ceph-mon[35244]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:58.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:58.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:58.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:58.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:53:58.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:58.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:53:59.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:53:59 smithi203 ceph-mon[35546]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:59.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:53:59 smithi174 ceph-mon[35244]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:53:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:53:59 smithi028 ceph-mon[30969]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:00.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.991+0000 7fe9f27fc640 1 -- 172.21.15.203:0/2462771262 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe9fc062630 con 0x7fe9e80a3270 2024-09-15T21:54:00.993 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 11 on host 'smithi203' 2024-09-15T21:54:00.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe9e80a3270 msgr2=0x7fe9e80a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:00.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe9e80a3270 0x7fe9e80a5730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8005d00 tx=0x7fe9f800b410 comp rx=0 tx=0).stop 2024-09-15T21:54:00.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 msgr2=0x7fe9fc19f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc19f490 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4005e30 tx=0x7fe9f4031ee0 comp rx=0 tx=0).stop 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 shutdown_connections 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe9e80a3270 0x7fe9e80a5730 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc107940 0x7fe9fc19f490 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc105010 0x7fe9fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 --2- 172.21.15.203:0/2462771262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc1026e0 0x7fe9fc197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 >> 172.21.15.203:0/2462771262 conn(0x7fe9fc0fc280 msgr2=0x7fe9fc102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 shutdown_connections 2024-09-15T21:54:00.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:00.993+0000 7fea0331a640 1 -- 172.21.15.203:0/2462771262 wait complete. 2024-09-15T21:54:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:01.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:01 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:01.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:01.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:01 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:01.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:01 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:01.732 DEBUG:teuthology.orchestra.run.smithi203:osd.11> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.11.service 2024-09-15T21:54:01.735 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-15T21:54:01.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T21:54:02.002 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:02.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 -- 172.21.15.28:0/4290545244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3181037c0 msgr2=0x7fa31810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 --2- 172.21.15.28:0/4290545244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3181037c0 0x7fa31810a050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa310009ca0 tx=0x7fa31002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 -- 172.21.15.28:0/4290545244 shutdown_connections 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 --2- 172.21.15.28:0/4290545244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3181037c0 0x7fa31810a050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 --2- 172.21.15.28:0/4290545244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa318102e00 0x7fa318103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 --2- 172.21.15.28:0/4290545244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa318101c00 0x7fa318102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 -- 172.21.15.28:0/4290545244 >> 172.21.15.28:0/4290545244 conn(0x7fa3180fd390 msgr2=0x7fa3180ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 -- 172.21.15.28:0/4290545244 shutdown_connections 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.573+0000 7fa31f47c640 1 -- 172.21.15.28:0/4290545244 wait complete. 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 Processor -- start 2024-09-15T21:54:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 -- start start 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa318101c00 0x7fa318197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa318102e00 0x7fa318198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3181037c0 0x7fa31819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa31810d8e0 con 0x7fa318102e00 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa31810d760 con 0x7fa318101c00 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31f47c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa31810da60 con 0x7fa3181037c0 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa318102e00 0x7fa318198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa318102e00 0x7fa318198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57646/0 (socket says 172.21.15.28:57646) 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 -- 172.21.15.28:0/945203925 learned_addr learned my addr 172.21.15.28:0/945203925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31d218640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa318101c00 0x7fa318197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31da19640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3181037c0 0x7fa31819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 -- 172.21.15.28:0/945203925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3181037c0 msgr2=0x7fa31819f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3181037c0 0x7fa31819f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 -- 172.21.15.28:0/945203925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa318101c00 msgr2=0x7fa318197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa318101c00 0x7fa318197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31ca17640 1 -- 172.21.15.28:0/945203925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31819fa80 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.574+0000 7fa31da19640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3181037c0 0x7fa31819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.575+0000 7fa31ca17640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa318102e00 0x7fa318198390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00b4a0 tx=0x7fa30c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.575+0000 7fa30a7fc640 1 -- 172.21.15.28:0/945203925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c014070 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.575+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31819fd70 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.575+0000 7fa30a7fc640 1 -- 172.21.15.28:0/945203925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa30c010af0 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.575+0000 7fa30a7fc640 1 -- 172.21.15.28:0/945203925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c017d10 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.575+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3181a02d0 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.576+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa318108f30 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.577+0000 7fa30a7fc640 1 -- 172.21.15.28:0/945203925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa30c017070 con 0x7fa318102e00 2024-09-15T21:54:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.578+0000 7fa30a7fc640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa3040a3270 0x7fa3040a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.578+0000 7fa30a7fc640 1 -- 172.21.15.28:0/945203925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6809+0+0 (secure 0 0 0) 0x7fa30c0ceae0 con 0x7fa318102e00 2024-09-15T21:54:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.578+0000 7fa31d218640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa3040a3270 0x7fa3040a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.579+0000 7fa31d218640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa3040a3270 0x7fa3040a5730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa318102c60 tx=0x7fa314005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.582+0000 7fa30a7fc640 1 -- 172.21.15.28:0/945203925 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa30c09bd40 con 0x7fa318102e00 2024-09-15T21:54:02.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.757+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa31802da50 con 0x7fa318102e00 2024-09-15T21:54:02.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.757+0000 7fa30a7fc640 1 -- 172.21.15.28:0/945203925 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+133 (secure 0 0 0) 0x7fa30c097030 con 0x7fa318102e00 2024-09-15T21:54:02.759 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa3040a3270 msgr2=0x7fa3040a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa3040a3270 0x7fa3040a5730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa318102c60 tx=0x7fa314005f90 comp rx=0 tx=0).stop 2024-09-15T21:54:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa318102e00 msgr2=0x7fa318198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa318102e00 0x7fa318198390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00b4a0 tx=0x7fa30c00b960 comp rx=0 tx=0).stop 2024-09-15T21:54:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 shutdown_connections 2024-09-15T21:54:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa3040a3270 0x7fa3040a5730 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3181037c0 0x7fa31819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa318102e00 0x7fa318198390 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 --2- 172.21.15.28:0/945203925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa318101c00 0x7fa318197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 >> 172.21.15.28:0/945203925 conn(0x7fa3180fd390 msgr2=0x7fa318106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 shutdown_connections 2024-09-15T21:54:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:02.759+0000 7fa31f47c640 1 -- 172.21.15.28:0/945203925 wait complete. 2024-09-15T21:54:03.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:03 smithi028 ceph-mon[30969]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:03.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:03 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/945203925' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:03.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:03 smithi203 ceph-mon[35546]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:03.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:03 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/945203925' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:03.461 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 21:54:03 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T21:54:03.290+0000 7f46d92b5640 -1 osd.11 0 log_to_monitors true 2024-09-15T21:54:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:03 smithi174 ceph-mon[35244]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:03.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:03 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/945203925' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:03.520 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1726437219,"num_in_osds":12,"osd_in_since":1726437221,"num_remapped_pgs":0} 2024-09-15T21:54:04.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:04 smithi203 ceph-mon[35546]: from='osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T21:54:04.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:04 smithi203 ceph-mon[35546]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T21:54:04.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:04 smithi028 ceph-mon[30969]: from='osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T21:54:04.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:04 smithi028 ceph-mon[30969]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T21:54:04.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:04 smithi174 ceph-mon[35244]: from='osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T21:54:04.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:04 smithi174 ceph-mon[35244]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T21:54:04.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T21:54:04.796 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:05.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:05.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T21:54:05.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: osdmap e74: 12 total, 11 up, 12 in 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:05.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:05.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:05 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 -- 172.21.15.28:0/3562127723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c102e00 msgr2=0x7fb26c103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 --2- 172.21.15.28:0/3562127723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c102e00 0x7fb26c103280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb258009900 tx=0x7fb25802a410 comp rx=0 tx=0).stop 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 -- 172.21.15.28:0/3562127723 shutdown_connections 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 --2- 172.21.15.28:0/3562127723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb26c1037c0 0x7fb26c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 --2- 172.21.15.28:0/3562127723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c102e00 0x7fb26c103280 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 --2- 172.21.15.28:0/3562127723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb26c101c00 0x7fb26c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 -- 172.21.15.28:0/3562127723 >> 172.21.15.28:0/3562127723 conn(0x7fb26c0fd390 msgr2=0x7fb26c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 -- 172.21.15.28:0/3562127723 shutdown_connections 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.287+0000 7fb27090f640 1 -- 172.21.15.28:0/3562127723 wait complete. 2024-09-15T21:54:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 Processor -- start 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 -- start start 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb26c101c00 0x7fb26c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb26c102e00 0x7fb26c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c1037c0 0x7fb26c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb26c10d8e0 con 0x7fb26c1037c0 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb26c10d760 con 0x7fb26c101c00 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb27090f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb26c10da60 con 0x7fb26c102e00 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c1037c0 0x7fb26c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c1037c0 0x7fb26c19f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57672/0 (socket says 172.21.15.28:57672) 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 -- 172.21.15.28:0/3752920771 learned_addr learned my addr 172.21.15.28:0/3752920771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb269d9b640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb26c102e00 0x7fb26c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26a59c640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb26c101c00 0x7fb26c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 -- 172.21.15.28:0/3752920771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb26c102e00 msgr2=0x7fb26c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb26c102e00 0x7fb26c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 -- 172.21.15.28:0/3752920771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb26c101c00 msgr2=0x7fb26c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb26c101c00 0x7fb26c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.288+0000 7fb26ad9d640 1 -- 172.21.15.28:0/3752920771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb26c19fb90 con 0x7fb26c1037c0 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.289+0000 7fb26ad9d640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c1037c0 0x7fb26c19f520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb26400b970 tx=0x7fb26400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.289+0000 7fb2637fe640 1 -- 172.21.15.28:0/3752920771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb264019070 con 0x7fb26c1037c0 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.289+0000 7fb2637fe640 1 -- 172.21.15.28:0/3752920771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb26400cb40 con 0x7fb26c1037c0 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.289+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb26c19fe80 con 0x7fb26c1037c0 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.289+0000 7fb2637fe640 1 -- 172.21.15.28:0/3752920771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb264007650 con 0x7fb26c1037c0 2024-09-15T21:54:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.289+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb26c1a0360 con 0x7fb26c1037c0 2024-09-15T21:54:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.290+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb26c108fd0 con 0x7fb26c1037c0 2024-09-15T21:54:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.291+0000 7fb2637fe640 1 -- 172.21.15.28:0/3752920771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb26401c070 con 0x7fb26c1037c0 2024-09-15T21:54:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.291+0000 7fb2637fe640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2500a3270 0x7fb2500a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.292+0000 7fb26a59c640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2500a3270 0x7fb2500a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.292+0000 7fb2637fe640 1 -- 172.21.15.28:0/3752920771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6847+0+0 (secure 0 0 0) 0x7fb2640ce8e0 con 0x7fb26c1037c0 2024-09-15T21:54:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.292+0000 7fb26a59c640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2500a3270 0x7fb2500a5730 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb26c102c60 tx=0x7fb25c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:05.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.296+0000 7fb2637fe640 1 -- 172.21.15.28:0/3752920771 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb26409b570 con 0x7fb26c1037c0 2024-09-15T21:54:05.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T21:54:05.167+0000 7f46d5238640 -1 osd.11 0 waiting for initial osdmap 2024-09-15T21:54:05.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T21:54:05.177+0000 7f46d1060640 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:54:05.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:05.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: osdmap e74: 12 total, 11 up, 12 in 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:05.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:05.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:05 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.472+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb26c02d9c0 con 0x7fb26c1037c0 2024-09-15T21:54:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.473+0000 7fb2637fe640 1 -- 172.21.15.28:0/3752920771 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) ==== 74+0+133 (secure 0 0 0) 0x7fb26409f870 con 0x7fb26c1037c0 2024-09-15T21:54:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2500a3270 msgr2=0x7fb2500a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2500a3270 0x7fb2500a5730 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb26c102c60 tx=0x7fb25c00a2d0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c1037c0 msgr2=0x7fb26c19f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c1037c0 0x7fb26c19f520 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb26400b970 tx=0x7fb26400be30 comp rx=0 tx=0).stop 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 shutdown_connections 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2500a3270 0x7fb2500a5730 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c1037c0 0x7fb26c19f520 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb26c102e00 0x7fb26c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 --2- 172.21.15.28:0/3752920771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb26c101c00 0x7fb26c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 >> 172.21.15.28:0/3752920771 conn(0x7fb26c0fd390 msgr2=0x7fb26c1061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 shutdown_connections 2024-09-15T21:54:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:05.475+0000 7fb27090f640 1 -- 172.21.15.28:0/3752920771 wait complete. 2024-09-15T21:54:05.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:05.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: osdmap e74: 12 total, 11 up, 12 in 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:05.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:05.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:05.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:05 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:06 smithi028 ceph-mon[30969]: Detected new or changed devices on smithi203 2024-09-15T21:54:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:06 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi203 to 4272M 2024-09-15T21:54:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:06 smithi028 ceph-mon[30969]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:54:06.222 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:06 smithi028 ceph-mon[30969]: osdmap e75: 12 total, 11 up, 12 in 2024-09-15T21:54:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:06 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:06 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:06.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:06 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3752920771' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:06.254 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1726437219,"num_in_osds":12,"osd_in_since":1726437221,"num_remapped_pgs":0} 2024-09-15T21:54:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:06 smithi203 ceph-mon[35546]: Detected new or changed devices on smithi203 2024-09-15T21:54:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:06 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi203 to 4272M 2024-09-15T21:54:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:06 smithi203 ceph-mon[35546]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:54:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:06 smithi203 ceph-mon[35546]: osdmap e75: 12 total, 11 up, 12 in 2024-09-15T21:54:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:06 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:06 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:06 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3752920771' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:06.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:06 smithi174 ceph-mon[35244]: Detected new or changed devices on smithi203 2024-09-15T21:54:06.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:06 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi203 to 4272M 2024-09-15T21:54:06.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:06 smithi174 ceph-mon[35244]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-15T21:54:06.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:06 smithi174 ceph-mon[35244]: osdmap e75: 12 total, 11 up, 12 in 2024-09-15T21:54:06.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:06 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:06.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:06 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:06.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:06 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3752920771' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:07.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T21:54:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:07 smithi203 ceph-mon[35546]: purged_snaps scrub starts 2024-09-15T21:54:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:07 smithi203 ceph-mon[35546]: purged_snaps scrub ok 2024-09-15T21:54:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:07 smithi203 ceph-mon[35546]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:07 smithi203 ceph-mon[35546]: osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] boot 2024-09-15T21:54:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:07 smithi203 ceph-mon[35546]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T21:54:07.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:07 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:07 smithi174 ceph-mon[35244]: purged_snaps scrub starts 2024-09-15T21:54:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:07 smithi174 ceph-mon[35244]: purged_snaps scrub ok 2024-09-15T21:54:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:07 smithi174 ceph-mon[35244]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:07 smithi174 ceph-mon[35244]: osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] boot 2024-09-15T21:54:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:07 smithi174 ceph-mon[35244]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T21:54:07.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:07 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:07.525 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:07 smithi028 ceph-mon[30969]: purged_snaps scrub starts 2024-09-15T21:54:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:07 smithi028 ceph-mon[30969]: purged_snaps scrub ok 2024-09-15T21:54:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:07 smithi028 ceph-mon[30969]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-09-15T21:54:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:07 smithi028 ceph-mon[30969]: osd.11 [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] boot 2024-09-15T21:54:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:07 smithi028 ceph-mon[30969]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T21:54:07.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:07 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 -- 172.21.15.28:0/4100434877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 msgr2=0x7fdd98102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 --2- 172.21.15.28:0/4100434877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98102000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009900 tx=0x7fdd8802a410 comp rx=0 tx=0).stop 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 -- 172.21.15.28:0/4100434877 shutdown_connections 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 --2- 172.21.15.28:0/4100434877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd981037c0 0x7fdd9810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 --2- 172.21.15.28:0/4100434877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd98102e00 0x7fdd98103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 --2- 172.21.15.28:0/4100434877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98102000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 -- 172.21.15.28:0/4100434877 >> 172.21.15.28:0/4100434877 conn(0x7fdd980fd390 msgr2=0x7fdd980ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.020+0000 7fdd9de74640 1 -- 172.21.15.28:0/4100434877 shutdown_connections 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.021+0000 7fdd9de74640 1 -- 172.21.15.28:0/4100434877 wait complete. 2024-09-15T21:54:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.021+0000 7fdd9de74640 1 Processor -- start 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.021+0000 7fdd9de74640 1 -- start start 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd9de74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd9de74640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd98102e00 0x7fdd98198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd9de74640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd981037c0 0x7fdd9819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd9de74640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9810d890 con 0x7fdd98101c00 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd9de74640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9810d710 con 0x7fdd981037c0 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd9de74640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9810da10 con 0x7fdd98102e00 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57682/0 (socket says 172.21.15.28:57682) 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 -- 172.21.15.28:0/2332380250 learned_addr learned my addr 172.21.15.28:0/2332380250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd96ffd640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd98102e00 0x7fdd98198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd97fff640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd981037c0 0x7fdd9819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 -- 172.21.15.28:0/2332380250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd98102e00 msgr2=0x7fdd98198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd98102e00 0x7fdd98198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 -- 172.21.15.28:0/2332380250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd981037c0 msgr2=0x7fdd9819f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd981037c0 0x7fdd9819f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 -- 172.21.15.28:0/2332380250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9819fa80 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd977fe640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98197e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdd88031610 tx=0x7fdd880043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd94ff9640 1 -- 172.21.15.28:0/2332380250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8803f030 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd94ff9640 1 -- 172.21.15.28:0/2332380250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd88003ce0 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.022+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9819fd10 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.023+0000 7fdd94ff9640 1 -- 172.21.15.28:0/2332380250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88035070 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.023+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd981a01a0 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.023+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd98108fb0 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.025+0000 7fdd94ff9640 1 -- 172.21.15.28:0/2332380250 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdd88042030 con 0x7fdd98101c00 2024-09-15T21:54:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.025+0000 7fdd94ff9640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdd800a3270 0x7fdd800a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.026+0000 7fdd96ffd640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdd800a3270 0x7fdd800a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.026+0000 7fdd94ff9640 1 -- 172.21.15.28:0/2332380250 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7fdd880efcb0 con 0x7fdd98101c00 2024-09-15T21:54:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.026+0000 7fdd96ffd640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdd800a3270 0x7fdd800a5730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd98199370 tx=0x7fdd8c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.029+0000 7fdd94ff9640 1 -- 172.21.15.28:0/2332380250 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdd880bce70 con 0x7fdd98101c00 2024-09-15T21:54:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.204+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fdd9802d9e0 con 0x7fdd98101c00 2024-09-15T21:54:08.206 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:08 smithi028 ceph-mon[30969]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T21:54:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.204+0000 7fdd94ff9640 1 -- 172.21.15.28:0/2332380250 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) ==== 74+0+133 (secure 0 0 0) 0x7fdd880bc770 con 0x7fdd98101c00 2024-09-15T21:54:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.206+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdd800a3270 msgr2=0x7fdd800a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.206+0000 7fdd9de74640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdd800a3270 0x7fdd800a5730 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd98199370 tx=0x7fdd8c00a2d0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.206+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 msgr2=0x7fdd98197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.206+0000 7fdd9de74640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98197e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdd88031610 tx=0x7fdd880043b0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 shutdown_connections 2024-09-15T21:54:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fdd800a3270 0x7fdd800a5730 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd981037c0 0x7fdd9819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd98102e00 0x7fdd98198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 --2- 172.21.15.28:0/2332380250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98101c00 0x7fdd98197e50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 >> 172.21.15.28:0/2332380250 conn(0x7fdd980fd390 msgr2=0x7fdd98106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 shutdown_connections 2024-09-15T21:54:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:08.207+0000 7fdd9de74640 1 -- 172.21.15.28:0/2332380250 wait complete. 2024-09-15T21:54:08.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:08 smithi203 ceph-mon[35546]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T21:54:08.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:08 smithi174 ceph-mon[35244]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T21:54:08.982 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1726437246,"num_in_osds":12,"osd_in_since":1726437221,"num_remapped_pgs":0} 2024-09-15T21:54:08.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:54:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:09.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:09 smithi028 ceph-mon[30969]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:09.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:09 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2332380250' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:09 smithi203 ceph-mon[35546]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:09.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:09 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2332380250' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:09 smithi174 ceph-mon[35244]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:09.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:09 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2332380250' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T21:54:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.697+0000 7f5c78546640 1 -- 172.21.15.28:0/3432810305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c70102d30 msgr2=0x7f5c70103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.697+0000 7f5c78546640 1 --2- 172.21.15.28:0/3432810305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c70102d30 0x7f5c70103110 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5c68009930 tx=0x7f5c6802a440 comp rx=0 tx=0).stop 2024-09-15T21:54:09.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 -- 172.21.15.28:0/3432810305 shutdown_connections 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 --2- 172.21.15.28:0/3432810305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c701082a0 0x7f5c7010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 --2- 172.21.15.28:0/3432810305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c701036e0 0x7f5c70107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 --2- 172.21.15.28:0/3432810305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c70102d30 0x7f5c70103110 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 -- 172.21.15.28:0/3432810305 >> 172.21.15.28:0/3432810305 conn(0x7f5c700fc7d0 msgr2=0x7f5c700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 -- 172.21.15.28:0/3432810305 shutdown_connections 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 -- 172.21.15.28:0/3432810305 wait complete. 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 Processor -- start 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 -- start start 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c70102d30 0x7f5c701985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.698+0000 7f5c78546640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c701036e0 0x7f5c70198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:09.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c78546640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c701082a0 0x7f5c7019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c78546640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c70112130 con 0x7f5c701036e0 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c78546640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c70111fb0 con 0x7f5c701082a0 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c78546640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c701122b0 con 0x7f5c70102d30 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c701036e0 0x7f5c70198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c701036e0 0x7f5c70198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57702/0 (socket says 172.21.15.28:57702) 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 -- 172.21.15.28:0/498853453 learned_addr learned my addr 172.21.15.28:0/498853453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c762e2640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c70102d30 0x7f5c701985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 -- 172.21.15.28:0/498853453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c70102d30 msgr2=0x7f5c701985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c70102d30 0x7f5c701985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c76ae3640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c701082a0 0x7f5c7019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 -- 172.21.15.28:0/498853453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c701082a0 msgr2=0x7f5c7019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c701082a0 0x7f5c7019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 -- 172.21.15.28:0/498853453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c7019d580 con 0x7f5c701036e0 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c75ae1640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c701036e0 0x7f5c70198b10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c64006fd0 tx=0x7f5c6400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c6f7fe640 1 -- 172.21.15.28:0/498853453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c64011830 con 0x7f5c701036e0 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c6f7fe640 1 -- 172.21.15.28:0/498853453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c64016070 con 0x7f5c701036e0 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c7019d870 con 0x7f5c701036e0 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c6f7fe640 1 -- 172.21.15.28:0/498853453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400f2e0 con 0x7f5c701036e0 2024-09-15T21:54:09.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.699+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c701a5150 con 0x7f5c701036e0 2024-09-15T21:54:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.701+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c7006a850 con 0x7f5c701036e0 2024-09-15T21:54:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.702+0000 7f5c6f7fe640 1 -- 172.21.15.28:0/498853453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5c6401b460 con 0x7f5c701036e0 2024-09-15T21:54:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.702+0000 7f5c6f7fe640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5c580a3240 0x7f5c580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.702+0000 7f5c762e2640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5c580a3240 0x7f5c580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.702+0000 7f5c6f7fe640 1 -- 172.21.15.28:0/498853453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f5c640ceb10 con 0x7f5c701036e0 2024-09-15T21:54:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.703+0000 7f5c762e2640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5c580a3240 0x7f5c580a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c68006010 tx=0x7f5c68000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.707+0000 7f5c6f7fe640 1 -- 172.21.15.28:0/498853453 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5c6409bcd0 con 0x7f5c701036e0 2024-09-15T21:54:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.882+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c7010a2b0 con 0x7f5c701036e0 2024-09-15T21:54:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.884+0000 7f5c6f7fe640 1 -- 172.21.15.28:0/498853453 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+18443 (secure 0 0 0) 0x7f5c6409b5d0 con 0x7f5c701036e0 2024-09-15T21:54:09.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:09.886 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:54:07.171670+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5c580a3240 msgr2=0x7f5c580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5c580a3240 0x7f5c580a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5c68006010 tx=0x7f5c68000bc0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c701036e0 msgr2=0x7f5c70198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c701036e0 0x7f5c70198b10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c64006fd0 tx=0x7f5c6400b8e0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 shutdown_connections 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5c580a3240 0x7f5c580a5700 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c701082a0 0x7f5c7019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c701036e0 0x7f5c70198b10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 --2- 172.21.15.28:0/498853453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c70102d30 0x7f5c701985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 >> 172.21.15.28:0/498853453 conn(0x7f5c700fc7d0 msgr2=0x7f5c70106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 shutdown_connections 2024-09-15T21:54:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:09.886+0000 7f5c78546640 1 -- 172.21.15.28:0/498853453 wait complete. 2024-09-15T21:54:10.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:10 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/498853453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:54:10.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:10 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/498853453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:54:10.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:10 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/498853453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:54:10.672 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T21:50:13.957276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-15T21:54:10.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-15T21:54:10.931 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:11.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:11 smithi028 ceph-mon[30969]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 -- 172.21.15.28:0/3817095072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904101420 msgr2=0x7f190410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 --2- 172.21.15.28:0/3817095072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904101420 0x7f190410d950 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009cd0 tx=0x7f18fc02a8e0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 -- 172.21.15.28:0/3817095072 shutdown_connections 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 --2- 172.21.15.28:0/3817095072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904101420 0x7f190410d950 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 --2- 172.21.15.28:0/3817095072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1904100a80 0x7f1904100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 --2- 172.21.15.28:0/3817095072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1904106a80 0x7f1904106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 -- 172.21.15.28:0/3817095072 >> 172.21.15.28:0/3817095072 conn(0x7f19040fc7a0 msgr2=0x7f19040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 -- 172.21.15.28:0/3817095072 shutdown_connections 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.442+0000 7f1909172640 1 -- 172.21.15.28:0/3817095072 wait complete. 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.443+0000 7f1909172640 1 Processor -- start 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.443+0000 7f1909172640 1 -- start start 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.443+0000 7f1909172640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904100a80 0x7f1904198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1909172640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1904101420 0x7f1904198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1909172640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1904106a80 0x7f190419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1909172640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1904110e80 con 0x7f1904100a80 2024-09-15T21:54:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1909172640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1904110d00 con 0x7f1904106a80 2024-09-15T21:54:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1909172640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1904111000 con 0x7f1904101420 2024-09-15T21:54:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904100a80 0x7f1904198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904100a80 0x7f1904198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38522/0 (socket says 172.21.15.28:38522) 2024-09-15T21:54:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 -- 172.21.15.28:0/1466282463 learned_addr learned my addr 172.21.15.28:0/1466282463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f190259c640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1904101420 0x7f1904198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f190359e640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1904106a80 0x7f190419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 -- 172.21.15.28:0/1466282463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1904101420 msgr2=0x7f1904198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1904101420 0x7f1904198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 -- 172.21.15.28:0/1466282463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1904106a80 msgr2=0x7f190419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1904106a80 0x7f190419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 -- 172.21.15.28:0/1466282463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190419d3d0 con 0x7f1904100a80 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f190359e640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1904106a80 0x7f190419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1902d9d640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904100a80 0x7f1904198420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d950 tx=0x7f18f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f18ebfff640 1 -- 172.21.15.28:0/1466282463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4017070 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190419d6c0 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f18ebfff640 1 -- 172.21.15.28:0/1466282463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18f4009c30 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.444+0000 7f18ebfff640 1 -- 172.21.15.28:0/1466282463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f400f6b0 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.445+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19041a4fa0 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.446+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1904100ee0 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.448+0000 7f18ebfff640 1 -- 172.21.15.28:0/1466282463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f18f40109d0 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.448+0000 7f18ebfff640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f18ec0a3240 0x7f18ec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.448+0000 7f18ebfff640 1 -- 172.21.15.28:0/1466282463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f18f40cf020 con 0x7f1904100a80 2024-09-15T21:54:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.448+0000 7f190259c640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f18ec0a3240 0x7f18ec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.449+0000 7f190259c640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f18ec0a3240 0x7f18ec0a5700 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f190406b7e0 tx=0x7f18f800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.451+0000 7f18ebfff640 1 -- 172.21.15.28:0/1466282463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f18f4009de0 con 0x7f1904100a80 2024-09-15T21:54:11.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:11 smithi203 ceph-mon[35546]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:11.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:11 smithi174 ceph-mon[35244]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:11.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.628+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f19041a51a0 con 0x7f1904100a80 2024-09-15T21:54:11.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.628+0000 7f18ebfff640 1 -- 172.21.15.28:0/1466282463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7f18f409c1e0 con 0x7f1904100a80 2024-09-15T21:54:11.630 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-15T21:54:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.630+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f18ec0a3240 msgr2=0x7f18ec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.630+0000 7f1909172640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f18ec0a3240 0x7f18ec0a5700 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f190406b7e0 tx=0x7f18f800b390 comp rx=0 tx=0).stop 2024-09-15T21:54:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.630+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904100a80 msgr2=0x7f1904198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.630+0000 7f1909172640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904100a80 0x7f1904198420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d950 tx=0x7f18f400de10 comp rx=0 tx=0).stop 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 shutdown_connections 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f18ec0a3240 0x7f18ec0a5700 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1904106a80 0x7f190419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1904101420 0x7f1904198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 --2- 172.21.15.28:0/1466282463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1904100a80 0x7f1904198420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 >> 172.21.15.28:0/1466282463 conn(0x7f19040fc7a0 msgr2=0x7f1904104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 shutdown_connections 2024-09-15T21:54:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:11.631+0000 7f1909172640 1 -- 172.21.15.28:0/1466282463 wait complete. 2024-09-15T21:54:12.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:12 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1466282463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T21:54:12.382 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.b on smithi174 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.e on smithi174 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.h on smithi174 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.k on smithi174 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.c on smithi203 2024-09-15T21:54:12.383 INFO:tasks.cephadm:Adding mds.f on smithi203 2024-09-15T21:54:12.384 INFO:tasks.cephadm:Adding mds.i on smithi203 2024-09-15T21:54:12.384 INFO:tasks.cephadm:Adding mds.l on smithi203 2024-09-15T21:54:12.384 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l' 2024-09-15T21:54:12.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:12 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1466282463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T21:54:12.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:12 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1466282463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T21:54:12.642 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T21:54:13.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 -- 172.21.15.203:0/827809729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1730101c00 msgr2=0x7f1730102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:13.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 --2- 172.21.15.203:0/827809729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1730101c00 0x7f1730102000 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f172c009900 tx=0x7f172c02a410 comp rx=0 tx=0).stop 2024-09-15T21:54:13.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 -- 172.21.15.203:0/827809729 shutdown_connections 2024-09-15T21:54:13.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 --2- 172.21.15.203:0/827809729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17301037c0 0x7f173010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 --2- 172.21.15.203:0/827809729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1730102e00 0x7f1730103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 --2- 172.21.15.203:0/827809729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1730101c00 0x7f1730102000 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 -- 172.21.15.203:0/827809729 >> 172.21.15.203:0/827809729 conn(0x7f17300fd390 msgr2=0x7f17300ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 -- 172.21.15.203:0/827809729 shutdown_connections 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.158+0000 7f17373f2640 1 -- 172.21.15.203:0/827809729 wait complete. 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 Processor -- start 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 -- start start 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1730101c00 0x7f1730197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1730102e00 0x7f1730198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17301037c0 0x7f173019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f173010d930 con 0x7f1730101c00 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f173010d7b0 con 0x7f1730102e00 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f17373f2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f173010dab0 con 0x7f17301037c0 2024-09-15T21:54:13.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f173598f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17301037c0 0x7f173019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f173598f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17301037c0 0x7f173019f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:44326/0 (socket says 172.21.15.203:44326) 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f173598f640 1 -- 172.21.15.203:0/3667882498 learned_addr learned my addr 172.21.15.203:0/3667882498 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.159+0000 7f173518e640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1730101c00 0x7f1730197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f173498d640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1730102e00 0x7f1730198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f173598f640 1 -- 172.21.15.203:0/3667882498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1730102e00 msgr2=0x7f1730198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f173598f640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1730102e00 0x7f1730198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f173598f640 1 -- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1730101c00 msgr2=0x7f1730197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f173598f640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1730101c00 0x7f1730197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f173598f640 1 -- 172.21.15.203:0/3667882498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173019fb60 con 0x7f17301037c0 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f173598f640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17301037c0 0x7f173019f4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f172800cce0 tx=0x7f172800a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f17267fc640 1 -- 172.21.15.203:0/3667882498 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172800f450 con 0x7f17301037c0 2024-09-15T21:54:13.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f17267fc640 1 -- 172.21.15.203:0/3667882498 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f172800fad0 con 0x7f17301037c0 2024-09-15T21:54:13.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173019fe50 con 0x7f17301037c0 2024-09-15T21:54:13.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f17267fc640 1 -- 172.21.15.203:0/3667882498 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728018c10 con 0x7f17301037c0 2024-09-15T21:54:13.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.160+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17301a0330 con 0x7f17301037c0 2024-09-15T21:54:13.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.161+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1730109000 con 0x7f17301037c0 2024-09-15T21:54:13.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.162+0000 7f17267fc640 1 -- 172.21.15.203:0/3667882498 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f172800f5f0 con 0x7f17301037c0 2024-09-15T21:54:13.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.163+0000 7f17267fc640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f171c0a3270 0x7f171c0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:13.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.163+0000 7f17267fc640 1 -- 172.21.15.203:0/3667882498 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f17280cfb30 con 0x7f17301037c0 2024-09-15T21:54:13.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.163+0000 7f173518e640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f171c0a3270 0x7f171c0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:13.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.164+0000 7f173518e640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f171c0a3270 0x7f171c0a5730 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f172c02aa00 tx=0x7f172c0315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:13.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.167+0000 7f17267fc640 1 -- 172.21.15.203:0/3667882498 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1728014030 con 0x7f17301037c0 2024-09-15T21:54:13.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.343+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}) -- 0x7f1730062730 con 0x7f171c0a3270 2024-09-15T21:54:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.351+0000 7f17267fc640 1 -- 172.21.15.203:0/3667882498 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f1730062730 con 0x7f171c0a3270 2024-09-15T21:54:13.353 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mds.all update... 2024-09-15T21:54:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f171c0a3270 msgr2=0x7f171c0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f171c0a3270 0x7f171c0a5730 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f172c02aa00 tx=0x7f172c0315b0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17301037c0 msgr2=0x7f173019f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17301037c0 0x7f173019f4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f172800cce0 tx=0x7f172800a620 comp rx=0 tx=0).stop 2024-09-15T21:54:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 shutdown_connections 2024-09-15T21:54:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f171c0a3270 0x7f171c0a5730 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17301037c0 0x7f173019f4f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1730102e00 0x7f1730198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 --2- 172.21.15.203:0/3667882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1730101c00 0x7f1730197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:13.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 >> 172.21.15.203:0/3667882498 conn(0x7f17300fd390 msgr2=0x7f17301061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:13.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 shutdown_connections 2024-09-15T21:54:13.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T21:54:13.353+0000 7f17373f2640 1 -- 172.21.15.203:0/3667882498 wait complete. 2024-09-15T21:54:13.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:13 smithi203 ceph-mon[35546]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:13.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:13 smithi174 ceph-mon[35244]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:13.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:13 smithi028 ceph-mon[30969]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:14.064 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.a.service 2024-09-15T21:54:14.067 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.d.service 2024-09-15T21:54:14.110 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.g.service 2024-09-15T21:54:14.153 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.j.service 2024-09-15T21:54:14.198 DEBUG:teuthology.orchestra.run.smithi174:mds.b> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.b.service 2024-09-15T21:54:14.201 DEBUG:teuthology.orchestra.run.smithi174:mds.e> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.e.service 2024-09-15T21:54:14.244 DEBUG:teuthology.orchestra.run.smithi174:mds.h> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.h.service 2024-09-15T21:54:14.287 DEBUG:teuthology.orchestra.run.smithi174:mds.k> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.k.service 2024-09-15T21:54:14.331 DEBUG:teuthology.orchestra.run.smithi203:mds.c> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.c.service 2024-09-15T21:54:14.334 DEBUG:teuthology.orchestra.run.smithi203:mds.f> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.f.service 2024-09-15T21:54:14.379 DEBUG:teuthology.orchestra.run.smithi203:mds.i> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.i.service 2024-09-15T21:54:14.421 DEBUG:teuthology.orchestra.run.smithi203:mds.l> sudo journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.l.service 2024-09-15T21:54:14.464 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-15T21:54:14.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:14.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:14.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:14.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:14 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:14.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:54:14.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:14.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:14 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:14.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:54:14.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-09-15T21:54:14.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:14.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:14.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:14.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:14.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:14.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:14.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:14.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:14 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:14.725 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 -- 172.21.15.28:0/3835612920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 msgr2=0x7fbdb41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/3835612920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb41067e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009900 tx=0x7fbd9c02a410 comp rx=0 tx=0).stop 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 -- 172.21.15.28:0/3835612920 shutdown_connections 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/3835612920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdb4106f10 0x7fbdb410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/3835612920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb41067e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/3835612920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbdb4101de0 0x7fbdb41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 -- 172.21.15.28:0/3835612920 >> 172.21.15.28:0/3835612920 conn(0x7fbdb40fd8c0 msgr2=0x7fbdb40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 -- 172.21.15.28:0/3835612920 shutdown_connections 2024-09-15T21:54:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.197+0000 7fbdb89a3640 1 -- 172.21.15.28:0/3835612920 wait complete. 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 Processor -- start 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 -- start start 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbdb4101de0 0x7fbdb4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb4198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdb4106f10 0x7fbdb419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb4111f70 con 0x7fbdb4102790 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb4111df0 con 0x7fbdb4101de0 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.198+0000 7fbdb89a3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb41120f0 con 0x7fbdb4106f10 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb4198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb4198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38544/0 (socket says 172.21.15.28:38544) 2024-09-15T21:54:15.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 -- 172.21.15.28:0/353837324 learned_addr learned my addr 172.21.15.28:0/353837324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdb2d9d640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdb4106f10 0x7fbdb419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdb259c640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbdb4101de0 0x7fbdb4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 -- 172.21.15.28:0/353837324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdb4106f10 msgr2=0x7fbdb419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdb4106f10 0x7fbdb419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 -- 172.21.15.28:0/353837324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbdb4101de0 msgr2=0x7fbdb4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbdb4101de0 0x7fbdb4198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 -- 172.21.15.28:0/353837324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb419d3c0 con 0x7fbdb4102790 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdabfff640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb4198950 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006010 tx=0x7fbd9c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdab7fe640 1 -- 172.21.15.28:0/353837324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c03f030 con 0x7fbdb4102790 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdab7fe640 1 -- 172.21.15.28:0/353837324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd9c003ce0 con 0x7fbdb4102790 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb419d650 con 0x7fbdb4102790 2024-09-15T21:54:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdab7fe640 1 -- 172.21.15.28:0/353837324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c035070 con 0x7fbdb4102790 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.199+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb41a4f90 con 0x7fbdb4102790 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.200+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdb41032c0 con 0x7fbdb4102790 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.201+0000 7fbdab7fe640 1 -- 172.21.15.28:0/353837324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbd9c042030 con 0x7fbdb4102790 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.202+0000 7fbdab7fe640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbda00a32c0 0x7fbda00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.202+0000 7fbdb259c640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbda00a32c0 0x7fbda00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.202+0000 7fbdab7fe640 1 -- 172.21.15.28:0/353837324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7fbd9c0efc90 con 0x7fbdb4102790 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.203+0000 7fbdb259c640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbda00a32c0 0x7fbda00a5780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbda400bbe0 tx=0x7fbda400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.205+0000 7fbdab7fe640 1 -- 172.21.15.28:0/353837324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbd9c0bc8c0 con 0x7fbdb4102790 2024-09-15T21:54:15.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.419+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdb41021c0 con 0x7fbdb4102790 2024-09-15T21:54:15.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:15 smithi028 ceph-mon[30969]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:15.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:15 smithi028 ceph-mon[30969]: Deploying daemon mds.l on smithi203 2024-09-15T21:54:15.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.419+0000 7fbdab7fe640 1 -- 172.21.15.28:0/353837324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7fbdb41021c0 con 0x7fbdb4102790 2024-09-15T21:54:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:15.422 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"btime":"2024-09-15T21:47:10:448594+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-15T21:54:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-09-15T21:54:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.421+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbda00a32c0 msgr2=0x7fbda00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.421+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbda00a32c0 0x7fbda00a5780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbda400bbe0 tx=0x7fbda400b390 comp rx=0 tx=0).stop 2024-09-15T21:54:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.421+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 msgr2=0x7fbdb4198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.421+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb4198950 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006010 tx=0x7fbd9c0042f0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.421+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 shutdown_connections 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.422+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbda00a32c0 0x7fbda00a5780 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.422+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdb4106f10 0x7fbdb419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.422+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdb4102790 0x7fbdb4198950 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.422+0000 7fbdb89a3640 1 --2- 172.21.15.28:0/353837324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbdb4101de0 0x7fbdb4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.422+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 >> 172.21.15.28:0/353837324 conn(0x7fbdb40fd8c0 msgr2=0x7fbdb40fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.422+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 shutdown_connections 2024-09-15T21:54:15.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:15.422+0000 7fbdb89a3640 1 -- 172.21.15.28:0/353837324 wait complete. 2024-09-15T21:54:15.623 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:15 smithi203 ceph-mon[35546]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:15.624 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:15 smithi203 ceph-mon[35546]: Deploying daemon mds.l on smithi203 2024-09-15T21:54:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:15 smithi174 ceph-mon[35244]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:15 smithi174 ceph-mon[35244]: Deploying daemon mds.l on smithi203 2024-09-15T21:54:15.968 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:15 smithi203 systemd[1]: Starting Ceph mds.l for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:16.294 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 podman[65107]: 2024-09-15 21:54:15.920300222 +0000 UTC m=+0.015947418 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:16.294 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 podman[65107]: 2024-09-15 21:54:16.048987434 +0000 UTC m=+0.144634630 container create 7678aedb8bc26dbf6ad8078c4405550c1d7285855a397fac3c36ef0620397917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l, RELEASE=squid-8921c8e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T21:54:16.294 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 podman[65107]: 2024-09-15 21:54:16.293120354 +0000 UTC m=+0.388767551 container init 7678aedb8bc26dbf6ad8078c4405550c1d7285855a397fac3c36ef0620397917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-15T21:54:16.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:16 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/353837324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:16.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:16 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6832/2800102614,v1:172.21.15.203:6833/2800102614] up:boot 2024-09-15T21:54:16.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:16 smithi028 ceph-mon[30969]: fsmap 1 up:standby 2024-09-15T21:54:16.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:16 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T21:54:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:16 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/353837324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:16 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6832/2800102614,v1:172.21.15.203:6833/2800102614] up:boot 2024-09-15T21:54:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:16 smithi203 ceph-mon[35546]: fsmap 1 up:standby 2024-09-15T21:54:16.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:16 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T21:54:16.718 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 podman[65107]: 2024-09-15 21:54:16.295942507 +0000 UTC m=+0.391589704 container start 7678aedb8bc26dbf6ad8078c4405550c1d7285855a397fac3c36ef0620397917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-15T21:54:16.719 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 ceph-mds[65126]: -- 172.21.15.203:0/1949079399 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb69967860 con 0x55cb69945000 2024-09-15T21:54:16.719 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l[65122]: starting mds.l at 2024-09-15T21:54:16.719 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 bash[65107]: 7678aedb8bc26dbf6ad8078c4405550c1d7285855a397fac3c36ef0620397917 2024-09-15T21:54:16.719 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 21:54:16 smithi203 systemd[1]: Started Ceph mds.l for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:16 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/353837324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:16 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6832/2800102614,v1:172.21.15.203:6833/2800102614] up:boot 2024-09-15T21:54:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:16 smithi174 ceph-mon[35244]: fsmap 1 up:standby 2024-09-15T21:54:16.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:16 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T21:54:17.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:17.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:17.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:17.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:17.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:17 smithi203 ceph-mon[35546]: Deploying daemon mds.k on smithi174 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:17 smithi174 ceph-mon[35244]: Deploying daemon mds.k on smithi174 2024-09-15T21:54:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:17.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:17 smithi028 ceph-mon[30969]: Deploying daemon mds.k on smithi174 2024-09-15T21:54:18.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:18.492 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:18.704 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:18 smithi174 systemd[1]: Starting Ceph mds.k for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 -- 172.21.15.28:0/2520263484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990102d40 msgr2=0x7f2990103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 --2- 172.21.15.28:0/2520263484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990102d40 0x7f2990103120 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2980009ca0 tx=0x7f298002a950 comp rx=0 tx=0).stop 2024-09-15T21:54:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 -- 172.21.15.28:0/2520263484 shutdown_connections 2024-09-15T21:54:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 --2- 172.21.15.28:0/2520263484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f299010c170 0x7f299010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 --2- 172.21.15.28:0/2520263484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2990103660 0x7f299010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 --2- 172.21.15.28:0/2520263484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990102d40 0x7f2990103120 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 -- 172.21.15.28:0/2520263484 >> 172.21.15.28:0/2520263484 conn(0x7f29900fc7a0 msgr2=0x7f29900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 -- 172.21.15.28:0/2520263484 shutdown_connections 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.957+0000 7f2996162640 1 -- 172.21.15.28:0/2520263484 wait complete. 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 Processor -- start 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 -- start start 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2990102d40 0x7f29901009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103660 0x7f2990100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299010c170 0x7f2990101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2990111f70 con 0x7f2990103660 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2990111df0 con 0x7f2990102d40 2024-09-15T21:54:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f2996162640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f29901120f0 con 0x7f299010c170 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f298effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103660 0x7f2990100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f298effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103660 0x7f2990100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38558/0 (socket says 172.21.15.28:38558) 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f298effd640 1 -- 172.21.15.28:0/1143617564 learned_addr learned my addr 172.21.15.28:0/1143617564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.958+0000 7f298f7fe640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2990102d40 0x7f29901009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298ffff640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299010c170 0x7f2990101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298f7fe640 1 -- 172.21.15.28:0/1143617564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299010c170 msgr2=0x7f2990101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298f7fe640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299010c170 0x7f2990101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298f7fe640 1 -- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103660 msgr2=0x7f2990100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298f7fe640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103660 0x7f2990100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298f7fe640 1 -- 172.21.15.28:0/1143617564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29901a69a0 con 0x7f2990102d40 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298effd640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103660 0x7f2990100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:54:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298ffff640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299010c170 0x7f2990101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298f7fe640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2990102d40 0x7f29901009a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2980006010 tx=0x7f298002f5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298cff9640 1 -- 172.21.15.28:0/1143617564 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980034040 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29901a6c30 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298cff9640 1 -- 172.21.15.28:0/1143617564 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2980030920 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f298cff9640 1 -- 172.21.15.28:0/1143617564 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980004730 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.959+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29901a7110 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.960+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2990103120 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.961+0000 7f298cff9640 1 -- 172.21.15.28:0/1143617564 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2980037020 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.962+0000 7f298cff9640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f29780a3240 0x7f29780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.962+0000 7f298effd640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f29780a3240 0x7f29780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.962+0000 7f298cff9640 1 -- 172.21.15.28:0/1143617564 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f29800f0670 con 0x7f2990102d40 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.963+0000 7f298effd640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f29780a3240 0x7f29780a5700 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2990102430 tx=0x7f2984005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:18.965+0000 7f298cff9640 1 -- 172.21.15.28:0/1143617564 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f29800bd790 con 0x7f2990102d40 2024-09-15T21:54:19.007 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:18 smithi174 podman[65023]: 2024-09-15 21:54:18.605823953 +0000 UTC m=+0.015937577 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:19.008 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:18 smithi174 podman[65023]: 2024-09-15 21:54:18.737009973 +0000 UTC m=+0.147123593 container create 6eb2bd6a32e1cbaa70b209b8ded4ee6e90e4a34b439da74418349e4200a895e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-15T21:54:19.008 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:19 smithi174 podman[65023]: 2024-09-15 21:54:19.006563821 +0000 UTC m=+0.416677441 container init 6eb2bd6a32e1cbaa70b209b8ded4ee6e90e4a34b439da74418349e4200a895e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T21:54:19.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.182+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29901071e0 con 0x7f2990102d40 2024-09-15T21:54:19.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.182+0000 7f298cff9640 1 -- 172.21.15.28:0/1143617564 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7f29800bd090 con 0x7f2990102d40 2024-09-15T21:54:19.184 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:19.185 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"btime":"2024-09-15T21:54:16:324016+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-15T21:54:19.185 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f29780a3240 msgr2=0x7f29780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f29780a3240 0x7f29780a5700 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2990102430 tx=0x7f2984005f90 comp rx=0 tx=0).stop 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2990102d40 msgr2=0x7f29901009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2990102d40 0x7f29901009a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2980006010 tx=0x7f298002f5d0 comp rx=0 tx=0).stop 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 shutdown_connections 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f29780a3240 0x7f29780a5700 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299010c170 0x7f2990101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103660 0x7f2990100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:19.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 --2- 172.21.15.28:0/1143617564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2990102d40 0x7f29901009a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:19.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 >> 172.21.15.28:0/1143617564 conn(0x7f29900fc7a0 msgr2=0x7f29900fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:19.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 shutdown_connections 2024-09-15T21:54:19.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:19.185+0000 7f2996162640 1 -- 172.21.15.28:0/1143617564 wait complete. 2024-09-15T21:54:19.260 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:19 smithi174 podman[65023]: 2024-09-15 21:54:19.010052018 +0000 UTC m=+0.420165639 container start 6eb2bd6a32e1cbaa70b209b8ded4ee6e90e4a34b439da74418349e4200a895e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-15T21:54:19.260 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mds[65042]: -- 172.21.15.174:0/3663661157 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc85325860 con 0x55cc85303400 2024-09-15T21:54:19.261 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k[65038]: starting mds.k at 2024-09-15T21:54:19.261 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:19 smithi174 bash[65023]: 6eb2bd6a32e1cbaa70b209b8ded4ee6e90e4a34b439da74418349e4200a895e5 2024-09-15T21:54:19.261 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 21:54:19 smithi174 systemd[1]: Started Ceph mds.k for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:19.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:19.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:19 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1143617564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:19.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:19.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:19.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:19 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1143617564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:19.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:19.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:19.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:19.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:19.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:19.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:19 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1143617564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:20.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:20 smithi028 ceph-mon[30969]: Deploying daemon mds.j on smithi028 2024-09-15T21:54:20.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:20 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6832/1988552520,v1:172.21.15.174:6833/1988552520] up:boot 2024-09-15T21:54:20.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:20 smithi028 ceph-mon[30969]: fsmap 2 up:standby 2024-09-15T21:54:20.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:20 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T21:54:20.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:20 smithi203 ceph-mon[35546]: Deploying daemon mds.j on smithi028 2024-09-15T21:54:20.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:20 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6832/1988552520,v1:172.21.15.174:6833/1988552520] up:boot 2024-09-15T21:54:20.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:20 smithi203 ceph-mon[35546]: fsmap 2 up:standby 2024-09-15T21:54:20.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:20 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T21:54:20.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:20 smithi174 ceph-mon[35244]: Deploying daemon mds.j on smithi028 2024-09-15T21:54:20.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:20 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6832/1988552520,v1:172.21.15.174:6833/1988552520] up:boot 2024-09-15T21:54:20.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:20 smithi174 ceph-mon[35244]: fsmap 2 up:standby 2024-09-15T21:54:20.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:20 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T21:54:21.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:21 smithi028 ceph-mon[30969]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:21.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:21 smithi203 ceph-mon[35546]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:21.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:21 smithi174 ceph-mon[35244]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:21.921 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:21 smithi028 systemd[1]: Starting Ceph mds.j for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:22.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:22.238 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:21 smithi028 podman[69998]: 2024-09-15 21:54:21.870450481 +0000 UTC m=+0.016219066 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:22.238 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:21 smithi028 podman[69998]: 2024-09-15 21:54:21.99262064 +0000 UTC m=+0.138389201 container create 1d6c8bef2d45e40cf905199b3901d0d48a0e38e99e2d42a7c2fc9dd9f05eca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-15T21:54:22.239 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:22 smithi028 podman[69998]: 2024-09-15 21:54:22.237238395 +0000 UTC m=+0.383006959 container init 1d6c8bef2d45e40cf905199b3901d0d48a0e38e99e2d42a7c2fc9dd9f05eca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T21:54:22.418 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:22.541 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:22 smithi028 podman[69998]: 2024-09-15 21:54:22.240719843 +0000 UTC m=+0.386488406 container start 1d6c8bef2d45e40cf905199b3901d0d48a0e38e99e2d42a7c2fc9dd9f05eca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-15T21:54:22.541 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:22 smithi028 ceph-mds[70046]: -- 172.21.15.28:0/640505467 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607624ed860 con 0x5607624cb400 2024-09-15T21:54:22.541 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j[70031]: starting mds.j at 2024-09-15T21:54:22.541 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:22 smithi028 bash[69998]: 1d6c8bef2d45e40cf905199b3901d0d48a0e38e99e2d42a7c2fc9dd9f05eca6d 2024-09-15T21:54:22.541 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 21:54:22 smithi028 systemd[1]: Started Ceph mds.j for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.898+0000 7f61fc857640 1 -- 172.21.15.28:0/758638170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f8103660 msgr2=0x7f61f810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.898+0000 7f61fc857640 1 --2- 172.21.15.28:0/758638170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f8103660 0x7f61f810bb50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009900 tx=0x7f61ec02a480 comp rx=0 tx=0).stop 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 -- 172.21.15.28:0/758638170 shutdown_connections 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 --2- 172.21.15.28:0/758638170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61f810c170 0x7f61f810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 --2- 172.21.15.28:0/758638170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f8103660 0x7f61f810bb50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 --2- 172.21.15.28:0/758638170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f8102d40 0x7f61f8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 -- 172.21.15.28:0/758638170 >> 172.21.15.28:0/758638170 conn(0x7f61f80fc7a0 msgr2=0x7f61f80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 -- 172.21.15.28:0/758638170 shutdown_connections 2024-09-15T21:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 -- 172.21.15.28:0/758638170 wait complete. 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 Processor -- start 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.899+0000 7f61fc857640 1 -- start start 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61fc857640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f8102d40 0x7f61f8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61fc857640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61f8103660 0x7f61f8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61fc857640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f810c170 0x7f61f819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61fc857640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61f8112030 con 0x7f61f810c170 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61fc857640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f61f8111eb0 con 0x7f61f8103660 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61fc857640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61f81121b0 con 0x7f61f8102d40 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f810c170 0x7f61f819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f810c170 0x7f61f819cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35632/0 (socket says 172.21.15.28:35632) 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 -- 172.21.15.28:0/2922917983 learned_addr learned my addr 172.21.15.28:0/2922917983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f5d9b640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61f8103660 0x7f61f8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f659c640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f8102d40 0x7f61f8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 -- 172.21.15.28:0/2922917983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f8102d40 msgr2=0x7f61f8198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f8102d40 0x7f61f8198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 -- 172.21.15.28:0/2922917983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61f8103660 msgr2=0x7f61f8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61f8103660 0x7f61f8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 -- 172.21.15.28:0/2922917983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f819d640 con 0x7f61f810c170 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61f6d9d640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f810c170 0x7f61f819cf60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f61f000b970 tx=0x7f61f000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61e77fe640 1 -- 172.21.15.28:0/2922917983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0019070 con 0x7f61f810c170 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f819d930 con 0x7f61f810c170 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61e77fe640 1 -- 172.21.15.28:0/2922917983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f000cb40 con 0x7f61f810c170 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.900+0000 7f61e77fe640 1 -- 172.21.15.28:0/2922917983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0007650 con 0x7f61f810c170 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.901+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f81a5210 con 0x7f61f810c170 2024-09-15T21:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.902+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f806b070 con 0x7f61f810c170 2024-09-15T21:54:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.903+0000 7f61e77fe640 1 -- 172.21.15.28:0/2922917983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f61f001c070 con 0x7f61f810c170 2024-09-15T21:54:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.903+0000 7f61e77fe640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f61e00a3240 0x7f61e00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.903+0000 7f61f659c640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f61e00a3240 0x7f61e00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.903+0000 7f61e77fe640 1 -- 172.21.15.28:0/2922917983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f61f00cea70 con 0x7f61f810c170 2024-09-15T21:54:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.904+0000 7f61f659c640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f61e00a3240 0x7f61e00a5700 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009890 tx=0x7f61e8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:22.908+0000 7f61e77fe640 1 -- 172.21.15.28:0/2922917983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f61f009b5e0 con 0x7f61f810c170 2024-09-15T21:54:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.132+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61f81071e0 con 0x7f61f810c170 2024-09-15T21:54:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.132+0000 7f61e77fe640 1 -- 172.21.15.28:0/2922917983 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3091 (secure 0 0 0) 0x7f61f009f8e0 con 0x7f61f810c170 2024-09-15T21:54:23.134 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:23.134 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"btime":"2024-09-15T21:54:22:454857+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-15T21:54:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-09-15T21:54:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f61e00a3240 msgr2=0x7f61e00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f61e00a3240 0x7f61e00a5700 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009890 tx=0x7f61e8008040 comp rx=0 tx=0).stop 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f810c170 msgr2=0x7f61f819cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f810c170 0x7f61f819cf60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f61f000b970 tx=0x7f61f000be30 comp rx=0 tx=0).stop 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 shutdown_connections 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f61e00a3240 0x7f61e00a5700 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f810c170 0x7f61f819cf60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61f8103660 0x7f61f8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 --2- 172.21.15.28:0/2922917983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f8102d40 0x7f61f8198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 >> 172.21.15.28:0/2922917983 conn(0x7f61f80fc7a0 msgr2=0x7f61f80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 shutdown_connections 2024-09-15T21:54:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:23.135+0000 7f61fc857640 1 -- 172.21.15.28:0/2922917983 wait complete. 2024-09-15T21:54:23.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:23.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: Deploying daemon mds.d on smithi028 2024-09-15T21:54:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.28:6834/2109233987,v1:172.21.15.28:6835/2109233987] up:boot 2024-09-15T21:54:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: fsmap 3 up:standby 2024-09-15T21:54:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T21:54:23.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:23 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2922917983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:23.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:23.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: Deploying daemon mds.d on smithi028 2024-09-15T21:54:23.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.28:6834/2109233987,v1:172.21.15.28:6835/2109233987] up:boot 2024-09-15T21:54:23.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: fsmap 3 up:standby 2024-09-15T21:54:23.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T21:54:23.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:23 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2922917983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:23.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:23.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: Deploying daemon mds.d on smithi028 2024-09-15T21:54:23.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.28:6834/2109233987,v1:172.21.15.28:6835/2109233987] up:boot 2024-09-15T21:54:23.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: fsmap 3 up:standby 2024-09-15T21:54:23.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T21:54:23.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:23 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2922917983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:25.387 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 systemd[1]: Starting Ceph mds.d for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:25.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:25 smithi028 ceph-mon[30969]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:25.665 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 podman[70408]: 2024-09-15 21:54:25.437183674 +0000 UTC m=+0.016768207 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:25.665 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 podman[70408]: 2024-09-15 21:54:25.540125868 +0000 UTC m=+0.119710389 container create 8a8c18c74584c311d8862283477e8fd46cc49349160f12eb181c937b8c20bea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T21:54:25.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:25 smithi203 ceph-mon[35546]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:25 smithi174 ceph-mon[35244]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:25.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:26.171 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 podman[70408]: 2024-09-15 21:54:25.784661878 +0000 UTC m=+0.364246400 container init 8a8c18c74584c311d8862283477e8fd46cc49349160f12eb181c937b8c20bea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-15T21:54:26.171 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 podman[70408]: 2024-09-15 21:54:25.787688125 +0000 UTC m=+0.367272655 container start 8a8c18c74584c311d8862283477e8fd46cc49349160f12eb181c937b8c20bea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-15T21:54:26.171 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 ceph-mds[70427]: -- 172.21.15.28:0/592525618 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed80ea3860 con 0x55ed80e80800 2024-09-15T21:54:26.171 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d[70423]: starting mds.d at 2024-09-15T21:54:26.172 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 bash[70408]: 8a8c18c74584c311d8862283477e8fd46cc49349160f12eb181c937b8c20bea4 2024-09-15T21:54:26.172 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 21:54:25 smithi028 systemd[1]: Started Ceph mds.d for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:26.183 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 -- 172.21.15.28:0/3412871295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c108300 msgr2=0x7f0e4c10ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 --2- 172.21.15.28:0/3412871295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c108300 0x7f0e4c10ec70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0e44009930 tx=0x7f0e4402a420 comp rx=0 tx=0).stop 2024-09-15T21:54:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 -- 172.21.15.28:0/3412871295 shutdown_connections 2024-09-15T21:54:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 --2- 172.21.15.28:0/3412871295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c108300 0x7f0e4c10ec70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 --2- 172.21.15.28:0/3412871295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e4c103780 0x7f0e4c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 --2- 172.21.15.28:0/3412871295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e4c102dd0 0x7f0e4c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 -- 172.21.15.28:0/3412871295 >> 172.21.15.28:0/3412871295 conn(0x7f0e4c0fc7d0 msgr2=0x7f0e4c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 -- 172.21.15.28:0/3412871295 shutdown_connections 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.654+0000 7f0e540fd640 1 -- 172.21.15.28:0/3412871295 wait complete. 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 Processor -- start 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 -- start start 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c102dd0 0x7f0e4c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e4c103780 0x7f0e4c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e4c108300 0x7f0e4c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c1120d0 con 0x7f0e4c102dd0 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c111f50 con 0x7f0e4c108300 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e540fd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c112250 con 0x7f0e4c103780 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c102dd0 0x7f0e4c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:26.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c102dd0 0x7f0e4c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35666/0 (socket says 172.21.15.28:35666) 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 -- 172.21.15.28:0/2897757466 learned_addr learned my addr 172.21.15.28:0/2897757466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e5269a640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e4c108300 0x7f0e4c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51698640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e4c103780 0x7f0e4c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 -- 172.21.15.28:0/2897757466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e4c103780 msgr2=0x7f0e4c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e4c103780 0x7f0e4c198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 -- 172.21.15.28:0/2897757466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e4c108300 msgr2=0x7f0e4c19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e4c108300 0x7f0e4c19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51e99640 1 -- 172.21.15.28:0/2897757466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4c19d4c0 con 0x7f0e4c102dd0 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e51698640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e4c103780 0x7f0e4c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.655+0000 7f0e5269a640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e4c108300 0x7f0e4c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.656+0000 7f0e51e99640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c102dd0 0x7f0e4c198510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00cce0 tx=0x7f0e3c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:26.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.656+0000 7f0e4affd640 1 -- 172.21.15.28:0/2897757466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c00f650 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.656+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e4c19d7b0 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.656+0000 7f0e4affd640 1 -- 172.21.15.28:0/2897757466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e3c00fcd0 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.656+0000 7f0e4affd640 1 -- 172.21.15.28:0/2897757466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c018c70 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.656+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e4c1a5090 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.657+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e4c1031b0 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.658+0000 7f0e4affd640 1 -- 172.21.15.28:0/2897757466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f0e3c00f7f0 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.659+0000 7f0e4affd640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f0e340a3240 0x7f0e340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.659+0000 7f0e51698640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f0e340a3240 0x7f0e340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.659+0000 7f0e4affd640 1 -- 172.21.15.28:0/2897757466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f0e3c0cfa80 con 0x7f0e4c102dd0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.659+0000 7f0e51698640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f0e340a3240 0x7f0e340a5700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0fedd0 tx=0x7f0e40005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.663+0000 7f0e4affd640 1 -- 172.21.15.28:0/2897757466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0e3c0d1e20 con 0x7f0e4c102dd0 2024-09-15T21:54:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.880+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e4c1a52c0 con 0x7f0e4c102dd0 2024-09-15T21:54:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.880+0000 7f0e4affd640 1 -- 172.21.15.28:0/2897757466 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3916 (secure 0 0 0) 0x7f0e3c014030 con 0x7f0e4c102dd0 2024-09-15T21:54:26.883 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:26.884 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"btime":"2024-09-15T21:54:26:385939+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-15T21:54:26.884 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f0e340a3240 msgr2=0x7f0e340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f0e340a3240 0x7f0e340a5700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c0fedd0 tx=0x7f0e40005f90 comp rx=0 tx=0).stop 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c102dd0 msgr2=0x7f0e4c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c102dd0 0x7f0e4c198510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00cce0 tx=0x7f0e3c00a620 comp rx=0 tx=0).stop 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 shutdown_connections 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f0e340a3240 0x7f0e340a5700 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e4c108300 0x7f0e4c19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e4c103780 0x7f0e4c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 --2- 172.21.15.28:0/2897757466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e4c102dd0 0x7f0e4c198510 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.883+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 >> 172.21.15.28:0/2897757466 conn(0x7f0e4c0fc7d0 msgr2=0x7f0e4c105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.884+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 shutdown_connections 2024-09-15T21:54:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:26.884+0000 7f0e540fd640 1 -- 172.21.15.28:0/2897757466 wait complete. 2024-09-15T21:54:27.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:27.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:27.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:27.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:27.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: Deploying daemon mds.e on smithi174 2024-09-15T21:54:27.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.28:6836/2070322956,v1:172.21.15.28:6837/2070322956] up:boot 2024-09-15T21:54:27.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: fsmap 4 up:standby 2024-09-15T21:54:27.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T21:54:27.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:26 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2897757466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:27.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:27.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:27.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: Deploying daemon mds.e on smithi174 2024-09-15T21:54:27.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.28:6836/2070322956,v1:172.21.15.28:6837/2070322956] up:boot 2024-09-15T21:54:27.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: fsmap 4 up:standby 2024-09-15T21:54:27.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T21:54:27.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:26 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2897757466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: Deploying daemon mds.e on smithi174 2024-09-15T21:54:27.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.28:6836/2070322956,v1:172.21.15.28:6837/2070322956] up:boot 2024-09-15T21:54:27.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: fsmap 4 up:standby 2024-09-15T21:54:27.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T21:54:27.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:26 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2897757466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:28.238 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 systemd[1]: Starting Ceph mds.e for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:28.510 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 podman[65335]: 2024-09-15 21:54:28.139049196 +0000 UTC m=+0.016006618 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:28.510 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 podman[65335]: 2024-09-15 21:54:28.276356894 +0000 UTC m=+0.153314315 container create 5a42363735971110ab1ff881b1d2c804e1fa3adf9518d9000e554d3959fccba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e) 2024-09-15T21:54:28.941 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 podman[65335]: 2024-09-15 21:54:28.545530837 +0000 UTC m=+0.422488248 container init 5a42363735971110ab1ff881b1d2c804e1fa3adf9518d9000e554d3959fccba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T21:54:28.941 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 podman[65335]: 2024-09-15 21:54:28.548501386 +0000 UTC m=+0.425458805 container start 5a42363735971110ab1ff881b1d2c804e1fa3adf9518d9000e554d3959fccba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-15T21:54:28.941 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mds[65355]: -- 172.21.15.174:0/3841098285 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56055ead1860 con 0x56055eaaf400 2024-09-15T21:54:28.941 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e[65351]: starting mds.e at 2024-09-15T21:54:28.941 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 bash[65335]: 5a42363735971110ab1ff881b1d2c804e1fa3adf9518d9000e554d3959fccba7 2024-09-15T21:54:28.941 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 21:54:28 smithi174 systemd[1]: Started Ceph mds.e for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:28 smithi203 ceph-mon[35546]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:28 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:28 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:28 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:28 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:29.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:28 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:29.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:28 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mon[35244]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:29.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:29.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:28 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:29.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:28 smithi028 ceph-mon[30969]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:29.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:29.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:29.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:29.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:28 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:29.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:29.830 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:29 smithi203 ceph-mon[35546]: Deploying daemon mds.f on smithi203 2024-09-15T21:54:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:29 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6834/280309786,v1:172.21.15.174:6835/280309786] up:boot 2024-09-15T21:54:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:29 smithi203 ceph-mon[35546]: fsmap 5 up:standby 2024-09-15T21:54:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:29 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T21:54:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:29 smithi028 ceph-mon[30969]: Deploying daemon mds.f on smithi203 2024-09-15T21:54:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:29 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6834/280309786,v1:172.21.15.174:6835/280309786] up:boot 2024-09-15T21:54:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:29 smithi028 ceph-mon[30969]: fsmap 5 up:standby 2024-09-15T21:54:30.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:29 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T21:54:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:29 smithi174 ceph-mon[35244]: Deploying daemon mds.f on smithi203 2024-09-15T21:54:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:29 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6834/280309786,v1:172.21.15.174:6835/280309786] up:boot 2024-09-15T21:54:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:29 smithi174 ceph-mon[35244]: fsmap 5 up:standby 2024-09-15T21:54:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:29 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T21:54:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 -- 172.21.15.28:0/1098707930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 msgr2=0x7fa01810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 --2- 172.21.15.28:0/1098707930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa01810ea90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa010009ca0 tx=0x7fa01002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 -- 172.21.15.28:0/1098707930 shutdown_connections 2024-09-15T21:54:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 --2- 172.21.15.28:0/1098707930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa01810ea90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 --2- 172.21.15.28:0/1098707930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa018102790 0x7fa0181067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 --2- 172.21.15.28:0/1098707930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa018101de0 0x7fa0181021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 -- 172.21.15.28:0/1098707930 >> 172.21.15.28:0/1098707930 conn(0x7fa0180fd8c0 msgr2=0x7fa0180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 -- 172.21.15.28:0/1098707930 shutdown_connections 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 -- 172.21.15.28:0/1098707930 wait complete. 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 Processor -- start 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.305+0000 7fa01f310640 1 -- start start 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01f310640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa018101de0 0x7fa0180744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01f310640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa018102790 0x7fa018074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01f310640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa018074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01f310640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa018111f70 con 0x7fa018106f10 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01f310640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa018111df0 con 0x7fa018101de0 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01f310640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0181120f0 con 0x7fa018102790 2024-09-15T21:54:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa018074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa018074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35682/0 (socket says 172.21.15.28:35682) 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 -- 172.21.15.28:0/1910905669 learned_addr learned my addr 172.21.15.28:0/1910905669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01c8ab640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa018102790 0x7fa018074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d0ac640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa018101de0 0x7fa0180744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 -- 172.21.15.28:0/1910905669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa018102790 msgr2=0x7fa018074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa018102790 0x7fa018074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 -- 172.21.15.28:0/1910905669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa018101de0 msgr2=0x7fa0180744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa018101de0 0x7fa0180744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 -- 172.21.15.28:0/1910905669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0181a67f0 con 0x7fa018106f10 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d0ac640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa018101de0 0x7fa0180744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01d8ad640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa018074f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa010031a20 tx=0x7fa010031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa00e7fc640 1 -- 172.21.15.28:0/1910905669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa010003c30 con 0x7fa018106f10 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.306+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0181a6a80 con 0x7fa018106f10 2024-09-15T21:54:30.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.307+0000 7fa00e7fc640 1 -- 172.21.15.28:0/1910905669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa010003dd0 con 0x7fa018106f10 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.307+0000 7fa00e7fc640 1 -- 172.21.15.28:0/1910905669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01002fcf0 con 0x7fa018106f10 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.307+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0181a6fc0 con 0x7fa018106f10 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.308+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0181021c0 con 0x7fa018106f10 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.309+0000 7fa00e7fc640 1 -- 172.21.15.28:0/1910905669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa010004b40 con 0x7fa018106f10 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.309+0000 7fa00e7fc640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa0040a3240 0x7fa0040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.310+0000 7fa00e7fc640 1 -- 172.21.15.28:0/1910905669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7fa0100f0930 con 0x7fa018106f10 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.310+0000 7fa01d0ac640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa0040a3240 0x7fa0040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.310+0000 7fa01d0ac640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa0040a3240 0x7fa0040a5700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa01400abe0 tx=0x7fa01400a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:30.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.313+0000 7fa00e7fc640 1 -- 172.21.15.28:0/1910905669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa0100bdb80 con 0x7fa018106f10 2024-09-15T21:54:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.522+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa01810a0f0 con 0x7fa018106f10 2024-09-15T21:54:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.523+0000 7fa00e7fc640 1 -- 172.21.15.28:0/1910905669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4741 (secure 0 0 0) 0x7fa01002fe90 con 0x7fa018106f10 2024-09-15T21:54:30.526 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:30.526 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"btime":"2024-09-15T21:54:28:938933+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-15T21:54:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-09-15T21:54:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.525+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa0040a3240 msgr2=0x7fa0040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.525+0000 7fa01f310640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa0040a3240 0x7fa0040a5700 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa01400abe0 tx=0x7fa01400a5c0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.525+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 msgr2=0x7fa018074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.525+0000 7fa01f310640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa018074f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa010031a20 tx=0x7fa010031a50 comp rx=0 tx=0).stop 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.525+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 shutdown_connections 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.526+0000 7fa01f310640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa0040a3240 0x7fa0040a5700 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.526+0000 7fa01f310640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa018106f10 0x7fa018074f70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.526+0000 7fa01f310640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa018102790 0x7fa018074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.526+0000 7fa01f310640 1 --2- 172.21.15.28:0/1910905669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa018101de0 0x7fa0180744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.526+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 >> 172.21.15.28:0/1910905669 conn(0x7fa0180fd8c0 msgr2=0x7fa0180febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.526+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 shutdown_connections 2024-09-15T21:54:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:30.526+0000 7fa01f310640 1 -- 172.21.15.28:0/1910905669 wait complete. 2024-09-15T21:54:30.952 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 21:54:30 smithi203 podman[65407]: 2024-09-15 21:54:30.911439152 +0000 UTC m=+0.136708106 container create ca2aca01a6851e360c01779d61cf4f1c32f97196b6bca9b38ed2ac0f9ca2874b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-15T21:54:31.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:30 smithi028 ceph-mon[30969]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:31.207 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:30 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1910905669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:30 smithi203 ceph-mon[35546]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:30 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1910905669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:31.218 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 21:54:31 smithi203 podman[65407]: 2024-09-15 21:54:31.164425749 +0000 UTC m=+0.389694663 container init ca2aca01a6851e360c01779d61cf4f1c32f97196b6bca9b38ed2ac0f9ca2874b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-15T21:54:31.218 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 21:54:31 smithi203 podman[65407]: 2024-09-15 21:54:31.167547175 +0000 UTC m=+0.392816087 container start ca2aca01a6851e360c01779d61cf4f1c32f97196b6bca9b38ed2ac0f9ca2874b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-15T21:54:31.218 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 21:54:31 smithi203 ceph-mds[65426]: -- 172.21.15.203:0/3816815627 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d064e3860 con 0x555d064c0800 2024-09-15T21:54:31.219 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 21:54:31 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f[65422]: starting mds.f at 2024-09-15T21:54:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:30 smithi174 ceph-mon[35244]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:30 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1910905669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:31.718 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 21:54:31 smithi203 bash[65407]: ca2aca01a6851e360c01779d61cf4f1c32f97196b6bca9b38ed2ac0f9ca2874b 2024-09-15T21:54:31.718 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 21:54:31 smithi203 systemd[1]: Started Ceph mds.f for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:32.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:32.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:32.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:32.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: Deploying daemon mds.c on smithi203 2024-09-15T21:54:32.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] up:boot 2024-09-15T21:54:32.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: fsmap 6 up:standby 2024-09-15T21:54:32.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:32 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T21:54:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:32.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: Deploying daemon mds.c on smithi203 2024-09-15T21:54:32.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] up:boot 2024-09-15T21:54:32.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: fsmap 6 up:standby 2024-09-15T21:54:32.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:32 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T21:54:32.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:32.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:32.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:32.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:32.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: Deploying daemon mds.c on smithi203 2024-09-15T21:54:32.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] up:boot 2024-09-15T21:54:32.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: fsmap 6 up:standby 2024-09-15T21:54:32.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:32 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T21:54:33.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:33.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:33 smithi203 ceph-mon[35546]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:33.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:33 smithi028 ceph-mon[30969]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:33.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:33 smithi174 ceph-mon[35244]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.028+0000 7f24258a0640 1 -- 172.21.15.28:0/3373075230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100500 msgr2=0x7f24201008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.028+0000 7f24258a0640 1 --2- 172.21.15.28:0/3373075230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100500 0x7f24201008e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2408009ca0 tx=0x7f240802a8c0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.028+0000 7f24258a0640 1 -- 172.21.15.28:0/3373075230 shutdown_connections 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.028+0000 7f24258a0640 1 --2- 172.21.15.28:0/3373075230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2420106f70 0x7f242010eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.029+0000 7f24258a0640 1 --2- 172.21.15.28:0/3373075230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2420100eb0 0x7f2420106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.029+0000 7f24258a0640 1 --2- 172.21.15.28:0/3373075230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100500 0x7f24201008e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.029+0000 7f24258a0640 1 -- 172.21.15.28:0/3373075230 >> 172.21.15.28:0/3373075230 conn(0x7f24200fc5c0 msgr2=0x7f24200fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.029+0000 7f24258a0640 1 -- 172.21.15.28:0/3373075230 shutdown_connections 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.029+0000 7f24258a0640 1 -- 172.21.15.28:0/3373075230 wait complete. 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.029+0000 7f24258a0640 1 Processor -- start 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.029+0000 7f24258a0640 1 -- start start 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f24258a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2420100500 0x7f2420198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f24258a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100eb0 0x7f24201989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f24258a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2420106f70 0x7f242019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f24258a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2420111f90 con 0x7f2420100eb0 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f24258a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2420111e10 con 0x7f2420100500 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f24258a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2420112110 con 0x7f2420106f70 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100eb0 0x7f24201989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100eb0 0x7f24201989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49502/0 (socket says 172.21.15.28:49502) 2024-09-15T21:54:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 -- 172.21.15.28:0/3619411559 learned_addr learned my addr 172.21.15.28:0/3619411559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2420100500 0x7f2420198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241f7fe640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2420106f70 0x7f242019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 -- 172.21.15.28:0/3619411559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2420106f70 msgr2=0x7f242019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2420106f70 0x7f242019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 -- 172.21.15.28:0/3619411559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2420100500 msgr2=0x7f2420198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2420100500 0x7f2420198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 -- 172.21.15.28:0/3619411559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242019d440 con 0x7f2420100eb0 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.030+0000 7f241e7fc640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100eb0 0x7f24201989d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f241000d740 tx=0x7f241000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.031+0000 7f242489e640 1 -- 172.21.15.28:0/3619411559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410017070 con 0x7f2420100eb0 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.031+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242019d730 con 0x7f2420100eb0 2024-09-15T21:54:34.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.031+0000 7f242489e640 1 -- 172.21.15.28:0/3619411559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2410010460 con 0x7f2420100eb0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.031+0000 7f242489e640 1 -- 172.21.15.28:0/3619411559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241000f620 con 0x7f2420100eb0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.031+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24201a5010 con 0x7f2420100eb0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.032+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24201008e0 con 0x7f2420100eb0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.034+0000 7f242489e640 1 -- 172.21.15.28:0/3619411559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f241001a070 con 0x7f2420100eb0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.034+0000 7f242489e640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f240c0a3240 0x7f240c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.035+0000 7f241effd640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f240c0a3240 0x7f240c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.035+0000 7f242489e640 1 -- 172.21.15.28:0/3619411559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f24100cee60 con 0x7f2420100eb0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.035+0000 7f241effd640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f240c0a3240 0x7f240c0a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2408005e30 tx=0x7f2408009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:34.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.038+0000 7f242489e640 1 -- 172.21.15.28:0/3619411559 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f241009bf80 con 0x7f2420100eb0 2024-09-15T21:54:34.218 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 21:54:33 smithi203 systemd[1]: Started Ceph mds.c for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.250+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24201997f0 con 0x7f2420100eb0 2024-09-15T21:54:34.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.250+0000 7f242489e640 1 -- 172.21.15.28:0/3619411559 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5569 (secure 0 0 0) 0x7f241009b880 con 0x7f2420100eb0 2024-09-15T21:54:34.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:34.252 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"btime":"2024-09-15T21:54:31:950706+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-09-15T21:54:34.253 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f240c0a3240 msgr2=0x7f240c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f240c0a3240 0x7f240c0a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2408005e30 tx=0x7f2408009360 comp rx=0 tx=0).stop 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100eb0 msgr2=0x7f24201989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100eb0 0x7f24201989d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f241000d740 tx=0x7f241000db40 comp rx=0 tx=0).stop 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 shutdown_connections 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f240c0a3240 0x7f240c0a5700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2420106f70 0x7f242019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420100eb0 0x7f24201989d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 --2- 172.21.15.28:0/3619411559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2420100500 0x7f2420198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 >> 172.21.15.28:0/3619411559 conn(0x7f24200fc5c0 msgr2=0x7f2420105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 shutdown_connections 2024-09-15T21:54:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:34.253+0000 7f24258a0640 1 -- 172.21.15.28:0/3619411559 wait complete. 2024-09-15T21:54:35.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: Deploying daemon mds.h on smithi174 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3619411559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] up:boot 2024-09-15T21:54:35.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: fsmap 7 up:standby 2024-09-15T21:54:35.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:34 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T21:54:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:35.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:35.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:35.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: Deploying daemon mds.h on smithi174 2024-09-15T21:54:35.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3619411559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:35.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] up:boot 2024-09-15T21:54:35.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: fsmap 7 up:standby 2024-09-15T21:54:35.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:34 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T21:54:35.238 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:35.238 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.238 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.238 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: Deploying daemon mds.h on smithi174 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3619411559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] up:boot 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: fsmap 7 up:standby 2024-09-15T21:54:35.239 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:34 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T21:54:36.260 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 systemd[1]: Starting Ceph mds.h for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:36.667 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 podman[65639]: 2024-09-15 21:54:36.268310924 +0000 UTC m=+0.016463530 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:36.667 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 podman[65639]: 2024-09-15 21:54:36.404992225 +0000 UTC m=+0.153144865 container create 3fe66e52248602e426bee1abcf1f9d7019ceb8a47aafae54403795dfbbd09aa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-15T21:54:36.667 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 podman[65639]: 2024-09-15 21:54:36.666339598 +0000 UTC m=+0.414492196 container init 3fe66e52248602e426bee1abcf1f9d7019ceb8a47aafae54403795dfbbd09aa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-15T21:54:36.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:36.957 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 podman[65639]: 2024-09-15 21:54:36.669222383 +0000 UTC m=+0.417374981 container start 3fe66e52248602e426bee1abcf1f9d7019ceb8a47aafae54403795dfbbd09aa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-15T21:54:36.957 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mds[65659]: -- 172.21.15.174:0/3952164139 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558345957860 con 0x558345935400 2024-09-15T21:54:36.957 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h[65655]: starting mds.h at 2024-09-15T21:54:36.957 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 bash[65639]: 3fe66e52248602e426bee1abcf1f9d7019ceb8a47aafae54403795dfbbd09aa7 2024-09-15T21:54:36.957 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 21:54:36 smithi174 systemd[1]: Started Ceph mds.h for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:36 smithi203 ceph-mon[35546]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:36 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mon[35244]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:36 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:37.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:36 smithi028 ceph-mon[30969]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:37.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:37.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:37.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:37.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:36 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3635840399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a241036e0 msgr2=0x7f6a24107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3635840399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a241036e0 0x7f6a24107b90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009cd0 tx=0x7f6a1802a8e0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3635840399 shutdown_connections 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3635840399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a241082a0 0x7f6a2410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3635840399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a241036e0 0x7f6a24107b90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3635840399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a24102d30 0x7f6a24103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3635840399 >> 172.21.15.28:0/3635840399 conn(0x7f6a240fc7d0 msgr2=0x7f6a240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3635840399 shutdown_connections 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.690+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3635840399 wait complete. 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 Processor -- start 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 -- start start 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a24102d30 0x7f6a24198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a241036e0 0x7f6a24198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a241082a0 0x7f6a2419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a241120a0 con 0x7f6a24102d30 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a24111f20 con 0x7f6a241036e0 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a28b1c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6a24112220 con 0x7f6a241082a0 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a24102d30 0x7f6a24198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a24102d30 0x7f6a24198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49530/0 (socket says 172.21.15.28:49530) 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 -- 172.21.15.28:0/3489451597 learned_addr learned my addr 172.21.15.28:0/3489451597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a22d9d640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a241082a0 0x7f6a2419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a21d9b640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a241036e0 0x7f6a24198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 -- 172.21.15.28:0/3489451597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a241082a0 msgr2=0x7f6a2419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a241082a0 0x7f6a2419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 -- 172.21.15.28:0/3489451597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a241036e0 msgr2=0x7f6a24198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a241036e0 0x7f6a24198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a2259c640 1 -- 172.21.15.28:0/3489451597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2419d5f0 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.691+0000 7f6a21d9b640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a241036e0 0x7f6a24198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.692+0000 7f6a2259c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a24102d30 0x7f6a24198640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400c890 tx=0x7f6a1400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.692+0000 7f6a137fe640 1 -- 172.21.15.28:0/3489451597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14004220 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.692+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a2419d8e0 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.692+0000 7f6a137fe640 1 -- 172.21.15.28:0/3489451597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a140043c0 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.692+0000 7f6a137fe640 1 -- 172.21.15.28:0/3489451597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14004a00 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.692+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a241a51c0 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.693+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a24103110 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.694+0000 7f6a137fe640 1 -- 172.21.15.28:0/3489451597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6a140079b0 con 0x7f6a24102d30 2024-09-15T21:54:37.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.695+0000 7f6a137fe640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6a0c0a3240 0x7f6a0c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.695+0000 7f6a21d9b640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6a0c0a3240 0x7f6a0c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.695+0000 7f6a137fe640 1 -- 172.21.15.28:0/3489451597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f6a140cf220 con 0x7f6a24102d30 2024-09-15T21:54:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.695+0000 7f6a21d9b640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6a0c0a3240 0x7f6a0c0a5700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006010 tx=0x7f6a180093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.699+0000 7f6a137fe640 1 -- 172.21.15.28:0/3489451597 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6a14014030 con 0x7f6a24102d30 2024-09-15T21:54:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.913+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a241999a0 con 0x7f6a24102d30 2024-09-15T21:54:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.914+0000 7f6a137fe640 1 -- 172.21.15.28:0/3489451597 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7222 (secure 0 0 0) 0x7f6a1409c3e0 con 0x7f6a24102d30 2024-09-15T21:54:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:37.916 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"btime":"2024-09-15T21:54:36:946637+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-15T21:54:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6a0c0a3240 msgr2=0x7f6a0c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6a0c0a3240 0x7f6a0c0a5700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006010 tx=0x7f6a180093f0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a24102d30 msgr2=0x7f6a24198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a24102d30 0x7f6a24198640 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400c890 tx=0x7f6a1400cd50 comp rx=0 tx=0).stop 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 shutdown_connections 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6a0c0a3240 0x7f6a0c0a5700 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a241082a0 0x7f6a2419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a241036e0 0x7f6a24198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 --2- 172.21.15.28:0/3489451597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a24102d30 0x7f6a24198640 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 >> 172.21.15.28:0/3489451597 conn(0x7f6a240fc7d0 msgr2=0x7f6a24106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 shutdown_connections 2024-09-15T21:54:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:37.918+0000 7f6a28b1c640 1 -- 172.21.15.28:0/3489451597 wait complete. 2024-09-15T21:54:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:37 smithi028 ceph-mon[30969]: Deploying daemon mds.i on smithi203 2024-09-15T21:54:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:37 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] up:boot 2024-09-15T21:54:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:37 smithi028 ceph-mon[30969]: fsmap 8 up:standby 2024-09-15T21:54:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:37 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T21:54:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:37 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3489451597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:37 smithi203 ceph-mon[35546]: Deploying daemon mds.i on smithi203 2024-09-15T21:54:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:37 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] up:boot 2024-09-15T21:54:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:37 smithi203 ceph-mon[35546]: fsmap 8 up:standby 2024-09-15T21:54:38.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:37 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T21:54:38.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:37 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3489451597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:37 smithi174 ceph-mon[35244]: Deploying daemon mds.i on smithi203 2024-09-15T21:54:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:37 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] up:boot 2024-09-15T21:54:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:37 smithi174 ceph-mon[35244]: fsmap 8 up:standby 2024-09-15T21:54:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:37 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T21:54:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:37 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3489451597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:38.961 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:38 smithi203 systemd[1]: Starting Ceph mds.i for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:39.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:38 smithi203 ceph-mon[35546]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:39.218 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 podman[66010]: 2024-09-15 21:54:38.913227679 +0000 UTC m=+0.016106427 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:39.218 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 podman[66010]: 2024-09-15 21:54:39.022955595 +0000 UTC m=+0.125834380 container create 5fa2b0e44dc8cd6406dd210017918e251e18098b87e11924058994aef07c95f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-15T21:54:39.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:38 smithi174 ceph-mon[35244]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:39.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:38 smithi028 ceph-mon[30969]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:39.718 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 podman[66010]: 2024-09-15 21:54:39.267260895 +0000 UTC m=+0.370139634 container init 5fa2b0e44dc8cd6406dd210017918e251e18098b87e11924058994aef07c95f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-15T21:54:39.718 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 podman[66010]: 2024-09-15 21:54:39.270120604 +0000 UTC m=+0.372999344 container start 5fa2b0e44dc8cd6406dd210017918e251e18098b87e11924058994aef07c95f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-15T21:54:39.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 ceph-mds[66029]: -- 172.21.15.203:0/3177999935 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56287ce05860 con 0x56287cde3000 2024-09-15T21:54:39.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i[66025]: starting mds.i at 2024-09-15T21:54:39.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 bash[66010]: 5fa2b0e44dc8cd6406dd210017918e251e18098b87e11924058994aef07c95f8 2024-09-15T21:54:39.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 21:54:39 smithi203 systemd[1]: Started Ceph mds.i for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:40.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:40.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:40.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:40.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: Deploying daemon mds.g on smithi028 2024-09-15T21:54:40.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] up:boot 2024-09-15T21:54:40.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: fsmap 9 up:standby 2024-09-15T21:54:40.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:40 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T21:54:40.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:40.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:40.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: Deploying daemon mds.g on smithi028 2024-09-15T21:54:40.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] up:boot 2024-09-15T21:54:40.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: fsmap 9 up:standby 2024-09-15T21:54:40.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:40 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T21:54:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:40.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: Deploying daemon mds.g on smithi028 2024-09-15T21:54:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] up:boot 2024-09-15T21:54:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: fsmap 9 up:standby 2024-09-15T21:54:40.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:40 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T21:54:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2733472396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8106050 msgr2=0x7f55f810dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2733472396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8106050 0x7f55f810dbd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009ca0 tx=0x7f55f002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2733472396 shutdown_connections 2024-09-15T21:54:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2733472396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8106050 0x7f55f810dbd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2733472396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55f8101df0 0x7f55f8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2733472396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f8101440 0x7f55f8101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2733472396 >> 172.21.15.28:0/2733472396 conn(0x7f55f8077210 msgr2=0x7f55f80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2733472396 shutdown_connections 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.609+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2733472396 wait complete. 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 Processor -- start 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 -- start start 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8101440 0x7f55f8199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55f8101df0 0x7f55f8199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f8106050 0x7f55f81a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8111050 con 0x7f55f8101440 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8110ed0 con 0x7f55f8106050 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55fe1a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81111d0 con 0x7f55f8101df0 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.610+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8101440 0x7f55f8199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8101440 0x7f55f8199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45220/0 (socket says 172.21.15.28:45220) 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 -- 172.21.15.28:0/2532207313 learned_addr learned my addr 172.21.15.28:0/2532207313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f6ffd640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55f8101df0 0x7f55f8199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f7fff640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f8106050 0x7f55f81a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 -- 172.21.15.28:0/2532207313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55f8101df0 msgr2=0x7f55f8199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55f8101df0 0x7f55f8199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 -- 172.21.15.28:0/2532207313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f8106050 msgr2=0x7f55f81a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f8106050 0x7f55f81a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 -- 172.21.15.28:0/2532207313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f81a4a10 con 0x7f55f8101440 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f77fe640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8101440 0x7f55f8199450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c890 tx=0x7f55e800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f4ff9640 1 -- 172.21.15.28:0/2532207313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8004220 con 0x7f55f8101440 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f81a4be0 con 0x7f55f8101440 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f4ff9640 1 -- 172.21.15.28:0/2532207313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e80043c0 con 0x7f55f8101440 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55f4ff9640 1 -- 172.21.15.28:0/2532207313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8004a00 con 0x7f55f8101440 2024-09-15T21:54:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.611+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f81a5120 con 0x7f55f8101440 2024-09-15T21:54:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.612+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f8101820 con 0x7f55f8101440 2024-09-15T21:54:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.614+0000 7f55f4ff9640 1 -- 172.21.15.28:0/2532207313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f55e80079b0 con 0x7f55f8101440 2024-09-15T21:54:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.614+0000 7f55f4ff9640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f55e40a3240 0x7f55e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.614+0000 7f55f6ffd640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f55e40a3240 0x7f55e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.614+0000 7f55f4ff9640 1 -- 172.21.15.28:0/2532207313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f55e80cf540 con 0x7f55f8101440 2024-09-15T21:54:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.615+0000 7f55f6ffd640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f55e40a3240 0x7f55e40a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f55f8100c20 tx=0x7f55e000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.618+0000 7f55f4ff9640 1 -- 172.21.15.28:0/2532207313 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f55e8014030 con 0x7f55f8101440 2024-09-15T21:54:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:41 smithi203 ceph-mon[35546]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:41 smithi174 ceph-mon[35244]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:41.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:41 smithi028 ceph-mon[30969]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.859+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55f8109290 con 0x7f55f8101440 2024-09-15T21:54:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.860+0000 7f55f4ff9640 1 -- 172.21.15.28:0/2532207313 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8052 (secure 0 0 0) 0x7f55e809c700 con 0x7f55f8101440 2024-09-15T21:54:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:41.862 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"btime":"2024-09-15T21:54:39:959294+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-15T21:54:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-09-15T21:54:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f55e40a3240 msgr2=0x7f55e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f55e40a3240 0x7f55e40a5700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f55f8100c20 tx=0x7f55e000a2d0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8101440 msgr2=0x7f55f8199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8101440 0x7f55f8199450 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c890 tx=0x7f55e800cc90 comp rx=0 tx=0).stop 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 shutdown_connections 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f55e40a3240 0x7f55e40a5700 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55f8106050 0x7f55f81a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55f8101df0 0x7f55f8199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 --2- 172.21.15.28:0/2532207313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8101440 0x7f55f8199450 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 >> 172.21.15.28:0/2532207313 conn(0x7f55f8077210 msgr2=0x7f55f8104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 shutdown_connections 2024-09-15T21:54:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:41.862+0000 7f55fe1a0640 1 -- 172.21.15.28:0/2532207313 wait complete. 2024-09-15T21:54:42.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:42 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2532207313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:42.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:42 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2532207313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:42.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:42 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2532207313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:42.921 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 21:54:42 smithi028 systemd[1]: Started Ceph mds.g for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:43.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:43 smithi028 ceph-mon[30969]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:43.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:43.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:43.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:43 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:43 smithi203 ceph-mon[35546]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:43 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:43 smithi174 ceph-mon[35244]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:43.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:43.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:43 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:44.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:44 smithi028 ceph-mon[30969]: Deploying daemon mds.a on smithi028 2024-09-15T21:54:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:44 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.28:6838/3176985638,v1:172.21.15.28:6839/3176985638] up:boot 2024-09-15T21:54:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:44 smithi028 ceph-mon[30969]: fsmap 10 up:standby 2024-09-15T21:54:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:44 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T21:54:44.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:44 smithi174 ceph-mon[35244]: Deploying daemon mds.a on smithi028 2024-09-15T21:54:44.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:44 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.28:6838/3176985638,v1:172.21.15.28:6839/3176985638] up:boot 2024-09-15T21:54:44.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:44 smithi174 ceph-mon[35244]: fsmap 10 up:standby 2024-09-15T21:54:44.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:44 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T21:54:44.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:44 smithi203 ceph-mon[35546]: Deploying daemon mds.a on smithi028 2024-09-15T21:54:44.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:44 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.28:6838/3176985638,v1:172.21.15.28:6839/3176985638] up:boot 2024-09-15T21:54:44.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:44 smithi203 ceph-mon[35546]: fsmap 10 up:standby 2024-09-15T21:54:44.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:44 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T21:54:45.074 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:44 smithi028 systemd[1]: Starting Ceph mds.a for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:45.391 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 podman[71611]: 2024-09-15 21:54:44.975284647 +0000 UTC m=+0.020895721 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:45.391 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 podman[71611]: 2024-09-15 21:54:45.112164778 +0000 UTC m=+0.157775853 container create ea2f62eab5bd5c34bf4a032d1df55f86ac3fa6e9d55a41e79d268d02fe8efb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-15T21:54:45.391 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 podman[71611]: 2024-09-15 21:54:45.389981403 +0000 UTC m=+0.435592486 container init ea2f62eab5bd5c34bf4a032d1df55f86ac3fa6e9d55a41e79d268d02fe8efb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-15T21:54:45.551 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:45.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:45 smithi028 ceph-mon[30969]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:45.671 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 podman[71611]: 2024-09-15 21:54:45.393127446 +0000 UTC m=+0.438738524 container start ea2f62eab5bd5c34bf4a032d1df55f86ac3fa6e9d55a41e79d268d02fe8efb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T21:54:45.671 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 ceph-mds[71656]: -- 172.21.15.28:0/2742949649 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555a8c255860 con 0x555a8c233400 2024-09-15T21:54:45.671 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a[71640]: starting mds.a at 2024-09-15T21:54:45.672 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 bash[71611]: ea2f62eab5bd5c34bf4a032d1df55f86ac3fa6e9d55a41e79d268d02fe8efb29 2024-09-15T21:54:45.672 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 21:54:45 smithi028 systemd[1]: Started Ceph mds.a for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:45.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:45 smithi174 ceph-mon[35244]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:45.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:45 smithi203 ceph-mon[35546]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 -- 172.21.15.28:0/3514787348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8104102df0 msgr2=0x7f81041031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 --2- 172.21.15.28:0/3514787348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8104102df0 0x7f81041031d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009ca0 tx=0x7f80f402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 -- 172.21.15.28:0/3514787348 shutdown_connections 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 --2- 172.21.15.28:0/3514787348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f810410c2f0 0x7f810410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 --2- 172.21.15.28:0/3514787348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81041037a0 0x7f810410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 --2- 172.21.15.28:0/3514787348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8104102df0 0x7f81041031d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 -- 172.21.15.28:0/3514787348 >> 172.21.15.28:0/3514787348 conn(0x7f81040fc7d0 msgr2=0x7f81040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 -- 172.21.15.28:0/3514787348 shutdown_connections 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 -- 172.21.15.28:0/3514787348 wait complete. 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 Processor -- start 2024-09-15T21:54:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.061+0000 7f810c10a640 1 -- start start 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810c10a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8104102df0 0x7f8104100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810c10a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81041037a0 0x7f8104100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810c10a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f810410c2f0 0x7f81041014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810c10a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8104112110 con 0x7f810410c2f0 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810c10a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8104111f90 con 0x7f8104102df0 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810c10a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8104112290 con 0x7f81041037a0 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f810410c2f0 0x7f81041014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f8109ea6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8104102df0 0x7f8104100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f810410c2f0 0x7f81041014d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45272/0 (socket says 172.21.15.28:45272) 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 -- 172.21.15.28:0/474395966 learned_addr learned my addr 172.21.15.28:0/474395966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f81096a5640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81041037a0 0x7f8104100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 -- 172.21.15.28:0/474395966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81041037a0 msgr2=0x7f8104100f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81041037a0 0x7f8104100f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 -- 172.21.15.28:0/474395966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8104102df0 msgr2=0x7f8104100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8104102df0 0x7f8104100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 -- 172.21.15.28:0/474395966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81041a6ad0 con 0x7f810410c2f0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f8109ea6640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8104102df0 0x7f8104100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f810a6a7640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f810410c2f0 0x7f81041014d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00d750 tx=0x7f80fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.062+0000 7f8102ffd640 1 -- 172.21.15.28:0/474395966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc017070 con 0x7f810410c2f0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.063+0000 7f8102ffd640 1 -- 172.21.15.28:0/474395966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80fc005500 con 0x7f810410c2f0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.063+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81041a6dc0 con 0x7f810410c2f0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.063+0000 7f8102ffd640 1 -- 172.21.15.28:0/474395966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc004d90 con 0x7f810410c2f0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.063+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81041a72d0 con 0x7f810410c2f0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.064+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f810406a790 con 0x7f810410c2f0 2024-09-15T21:54:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.065+0000 7f8102ffd640 1 -- 172.21.15.28:0/474395966 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f80fc004f30 con 0x7f810410c2f0 2024-09-15T21:54:46.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.065+0000 7f8102ffd640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80ec0a3240 0x7f80ec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:46.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.066+0000 7f8109ea6640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80ec0a3240 0x7f80ec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:46.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.066+0000 7f8102ffd640 1 -- 172.21.15.28:0/474395966 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f80fc0ced50 con 0x7f810410c2f0 2024-09-15T21:54:46.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.066+0000 7f8109ea6640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80ec0a3240 0x7f80ec0a5700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80f4006010 tx=0x7f80f4005f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:46.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.070+0000 7f8102ffd640 1 -- 172.21.15.28:0/474395966 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f80fc09b820 con 0x7f810410c2f0 2024-09-15T21:54:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.279+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81041031d0 con 0x7f810410c2f0 2024-09-15T21:54:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.280+0000 7f8102ffd640 1 -- 172.21.15.28:0/474395966 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9704 (secure 0 0 0) 0x7f80fc09fb20 con 0x7f810410c2f0 2024-09-15T21:54:46.281 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:46.282 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"btime":"2024-09-15T21:54:45:472770+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-15T21:54:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-09-15T21:54:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.282+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80ec0a3240 msgr2=0x7f80ec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.282+0000 7f810c10a640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80ec0a3240 0x7f80ec0a5700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80f4006010 tx=0x7f80f4005f30 comp rx=0 tx=0).stop 2024-09-15T21:54:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.282+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f810410c2f0 msgr2=0x7f81041014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.282+0000 7f810c10a640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f810410c2f0 0x7f81041014d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00d750 tx=0x7f80fc00dc10 comp rx=0 tx=0).stop 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 shutdown_connections 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80ec0a3240 0x7f80ec0a5700 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f810410c2f0 0x7f81041014d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81041037a0 0x7f8104100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 --2- 172.21.15.28:0/474395966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8104102df0 0x7f8104100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 >> 172.21.15.28:0/474395966 conn(0x7f81040fc7d0 msgr2=0x7f81040fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 shutdown_connections 2024-09-15T21:54:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:46.283+0000 7f810c10a640 1 -- 172.21.15.28:0/474395966 wait complete. 2024-09-15T21:54:46.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.28:6840/2641693590,v1:172.21.15.28:6841/2641693590] up:boot 2024-09-15T21:54:46.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: fsmap 11 up:standby 2024-09-15T21:54:46.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T21:54:46.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:46.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:46.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:46.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:46 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/474395966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:46.732 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.28:6840/2641693590,v1:172.21.15.28:6841/2641693590] up:boot 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: fsmap 11 up:standby 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:46.733 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:46 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/474395966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:46.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.28:6840/2641693590,v1:172.21.15.28:6841/2641693590] up:boot 2024-09-15T21:54:46.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: fsmap 11 up:standby 2024-09-15T21:54:46.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T21:54:46.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:46.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T21:54:46.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T21:54:46.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:46.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:46 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/474395966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:47.563 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:47 smithi174 ceph-mon[35244]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:47.563 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:47 smithi174 ceph-mon[35244]: Deploying daemon mds.b on smithi174 2024-09-15T21:54:47.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:47 smithi028 ceph-mon[30969]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:47.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:47 smithi028 ceph-mon[30969]: Deploying daemon mds.b on smithi174 2024-09-15T21:54:47.963 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:47 smithi174 systemd[1]: Starting Ceph mds.b for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T21:54:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:47 smithi203 ceph-mon[35546]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:47 smithi203 ceph-mon[35546]: Deploying daemon mds.b on smithi174 2024-09-15T21:54:48.248 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:47 smithi174 podman[65956]: 2024-09-15 21:54:47.864500797 +0000 UTC m=+0.016001182 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T21:54:48.248 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:47 smithi174 podman[65956]: 2024-09-15 21:54:47.985822889 +0000 UTC m=+0.137323265 container create 52e7bc84e2d02d59323bdfd87073224872f4e1b412507311a309a9f1051cf652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True) 2024-09-15T21:54:48.510 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:48 smithi174 podman[65956]: 2024-09-15 21:54:48.247101981 +0000 UTC m=+0.398602355 container init 52e7bc84e2d02d59323bdfd87073224872f4e1b412507311a309a9f1051cf652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-15T21:54:48.510 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:48 smithi174 podman[65956]: 2024-09-15 21:54:48.250434205 +0000 UTC m=+0.401934581 container start 52e7bc84e2d02d59323bdfd87073224872f4e1b412507311a309a9f1051cf652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-15T21:54:48.510 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:48 smithi174 ceph-mds[65975]: -- 172.21.15.174:0/1004952223 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8d3a1b860 con 0x55f8d39f9400 2024-09-15T21:54:48.510 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:48 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: starting mds.b at 2024-09-15T21:54:48.510 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:48 smithi174 bash[65956]: 52e7bc84e2d02d59323bdfd87073224872f4e1b412507311a309a9f1051cf652 2024-09-15T21:54:48.510 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:54:48 smithi174 systemd[1]: Started Ceph mds.b for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T21:54:48.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:54:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] up:boot 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: fsmap 12 up:standby 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T21:54:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:49 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] up:boot 2024-09-15T21:54:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: fsmap 12 up:standby 2024-09-15T21:54:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T21:54:49.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:49 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:49.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] up:boot 2024-09-15T21:54:49.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: fsmap 12 up:standby 2024-09-15T21:54:49.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T21:54:49.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:49 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 -- 172.21.15.28:0/1608941896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 msgr2=0x7f4f8c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 --2- 172.21.15.28:0/1608941896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c101200 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009900 tx=0x7f4f8802a480 comp rx=0 tx=0).stop 2024-09-15T21:54:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 -- 172.21.15.28:0/1608941896 shutdown_connections 2024-09-15T21:54:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 --2- 172.21.15.28:0/1608941896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 --2- 172.21.15.28:0/1608941896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8c102570 0x7f4f8c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 --2- 172.21.15.28:0/1608941896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c101200 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 -- 172.21.15.28:0/1608941896 >> 172.21.15.28:0/1608941896 conn(0x7f4f8c0fc7d0 msgr2=0x7f4f8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 -- 172.21.15.28:0/1608941896 shutdown_connections 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.208+0000 7f4f942f9640 1 -- 172.21.15.28:0/1608941896 wait complete. 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 Processor -- start 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 -- start start 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c110f50 con 0x7f4f8c100e20 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c110dd0 con 0x7f4f8c102ef0 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f942f9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c1110d0 con 0x7f4f8c102570 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45292/0 (socket says 172.21.15.28:45292) 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 -- 172.21.15.28:0/877266614 learned_addr learned my addr 172.21.15.28:0/877266614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92896640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f91894640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 -- 172.21.15.28:0/877266614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8c102570 msgr2=0x7f4f8c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 -- 172.21.15.28:0/877266614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8c102ef0 msgr2=0x7f4f8c19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 -- 172.21.15.28:0/877266614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c19d4d0 con 0x7f4f8c100e20 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f91894640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.209+0000 7f4f92095640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c198520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4f88006010 tx=0x7f4f88003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.210+0000 7f4f837fe640 1 -- 172.21.15.28:0/877266614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8803f070 con 0x7f4f8c100e20 2024-09-15T21:54:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.210+0000 7f4f837fe640 1 -- 172.21.15.28:0/877266614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f88031c30 con 0x7f4f8c100e20 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.210+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8c19d760 con 0x7f4f8c100e20 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.210+0000 7f4f837fe640 1 -- 172.21.15.28:0/877266614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88035030 con 0x7f4f8c100e20 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.210+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8c1a50a0 con 0x7f4f8c100e20 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.211+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8c1021b0 con 0x7f4f8c100e20 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.212+0000 7f4f837fe640 1 -- 172.21.15.28:0/877266614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4f88042070 con 0x7f4f8c100e20 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.213+0000 7f4f837fe640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f780a3240 0x7f4f780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.213+0000 7f4f837fe640 1 -- 172.21.15.28:0/877266614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f4f880efb70 con 0x7f4f8c100e20 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.213+0000 7f4f91894640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f780a3240 0x7f4f780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.213+0000 7f4f91894640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f780a3240 0x7f4f780a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0058c0 tx=0x7f4f7c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.217+0000 7f4f837fe640 1 -- 172.21.15.28:0/877266614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4f880bcd30 con 0x7f4f8c100e20 2024-09-15T21:54:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.456+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f8c10a0a0 con 0x7f4f8c100e20 2024-09-15T21:54:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.457+0000 7f4f837fe640 1 -- 172.21.15.28:0/877266614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10533 (secure 0 0 0) 0x7f4f880bc630 con 0x7f4f8c100e20 2024-09-15T21:54:50.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:50.459 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"btime":"2024-09-15T21:54:48:485607+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24485,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-15T21:54:50.459 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-09-15T21:54:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f780a3240 msgr2=0x7f4f780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f780a3240 0x7f4f780a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0058c0 tx=0x7f4f7c005850 comp rx=0 tx=0).stop 2024-09-15T21:54:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 msgr2=0x7f4f8c198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c198520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4f88006010 tx=0x7f4f88003c10 comp rx=0 tx=0).stop 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 shutdown_connections 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f780a3240 0x7f4f780a5700 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 --2- 172.21.15.28:0/877266614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f8c100e20 0x7f4f8c198520 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 >> 172.21.15.28:0/877266614 conn(0x7f4f8c0fc7d0 msgr2=0x7f4f8c06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 shutdown_connections 2024-09-15T21:54:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:50.460+0000 7f4f942f9640 1 -- 172.21.15.28:0/877266614 wait complete. 2024-09-15T21:54:51.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:51.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/877266614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:51.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:51.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:51.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:51 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:51.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-15T21:54:51.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:51.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/877266614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:51.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:51.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:51.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:51 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/877266614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:51.474 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:51 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:54:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 -- 172.21.15.28:0/1484858511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102ef0 msgr2=0x7f20c010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 --2- 172.21.15.28:0/1484858511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102ef0 0x7f20c010da40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009ca0 tx=0x7f20b802a950 comp rx=0 tx=0).stop 2024-09-15T21:54:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 -- 172.21.15.28:0/1484858511 shutdown_connections 2024-09-15T21:54:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 --2- 172.21.15.28:0/1484858511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102ef0 0x7f20c010da40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 --2- 172.21.15.28:0/1484858511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c0102570 0x7f20c01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 --2- 172.21.15.28:0/1484858511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20c0100e20 0x7f20c0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 -- 172.21.15.28:0/1484858511 >> 172.21.15.28:0/1484858511 conn(0x7f20c00fc7d0 msgr2=0x7f20c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 -- 172.21.15.28:0/1484858511 shutdown_connections 2024-09-15T21:54:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.571+0000 7f20c4d48640 1 -- 172.21.15.28:0/1484858511 wait complete. 2024-09-15T21:54:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 Processor -- start 2024-09-15T21:54:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 -- start start 2024-09-15T21:54:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20c0100e20 0x7f20c0199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102570 0x7f20c01998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c0102ef0 0x7f20c01a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20c0110f80 con 0x7f20c0102570 2024-09-15T21:54:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20c0110e00 con 0x7f20c0100e20 2024-09-15T21:54:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20c4d48640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20c0111100 con 0x7f20c0102ef0 2024-09-15T21:54:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20bdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102570 0x7f20c01998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20bdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102570 0x7f20c01998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46108/0 (socket says 172.21.15.28:46108) 2024-09-15T21:54:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20bdd9b640 1 -- 172.21.15.28:0/670792652 learned_addr learned my addr 172.21.15.28:0/670792652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.572+0000 7f20bed9d640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c0102ef0 0x7f20c01a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20bdd9b640 1 -- 172.21.15.28:0/670792652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c0102ef0 msgr2=0x7f20c01a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20bdd9b640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c0102ef0 0x7f20c01a43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20bdd9b640 1 -- 172.21.15.28:0/670792652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20c0100e20 msgr2=0x7f20c0199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20bdd9b640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20c0100e20 0x7f20c0199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20bdd9b640 1 -- 172.21.15.28:0/670792652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c01a4930 con 0x7f20c0102570 2024-09-15T21:54:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20bdd9b640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102570 0x7f20c01998b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e9f0 tx=0x7f20b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20af7fe640 1 -- 172.21.15.28:0/670792652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b400cd80 con 0x7f20c0102570 2024-09-15T21:54:52.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20af7fe640 1 -- 172.21.15.28:0/670792652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20b4005500 con 0x7f20c0102570 2024-09-15T21:54:52.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c01a4b00 con 0x7f20c0102570 2024-09-15T21:54:52.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20af7fe640 1 -- 172.21.15.28:0/670792652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4010690 con 0x7f20c0102570 2024-09-15T21:54:52.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.573+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c01a4fc0 con 0x7f20c0102570 2024-09-15T21:54:52.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.574+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c0101200 con 0x7f20c0102570 2024-09-15T21:54:52.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.575+0000 7f20af7fe640 1 -- 172.21.15.28:0/670792652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f20b40046c0 con 0x7f20c0102570 2024-09-15T21:54:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.576+0000 7f20af7fe640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20a80a3240 0x7f20a80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.576+0000 7f20be59c640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20a80a3240 0x7f20a80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.576+0000 7f20af7fe640 1 -- 172.21.15.28:0/670792652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f20b40cfb60 con 0x7f20c0102570 2024-09-15T21:54:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.577+0000 7f20be59c640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20a80a3240 0x7f20a80a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f20b0005fd0 tx=0x7f20b0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.580+0000 7f20af7fe640 1 -- 172.21.15.28:0/670792652 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f20b40541b0 con 0x7f20c0102570 2024-09-15T21:54:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.773+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f20c01a5250 con 0x7f20c0102570 2024-09-15T21:54:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.774+0000 7f20af7fe640 1 -- 172.21.15.28:0/670792652 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) ==== 84+0+65 (secure 0 0 0) 0x7f20b401f340 con 0x7f20c0102570 2024-09-15T21:54:52.776 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:54:52.776 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-15T21:54:52.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T21:54:52.776 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-15T21:54:52.776 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-15T21:54:52.776 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:54:52.776 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-15T21:54:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20a80a3240 msgr2=0x7f20a80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20a80a3240 0x7f20a80a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f20b0005fd0 tx=0x7f20b0005850 comp rx=0 tx=0).stop 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102570 msgr2=0x7f20c01998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102570 0x7f20c01998b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e9f0 tx=0x7f20b400edf0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 shutdown_connections 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20a80a3240 0x7f20a80a5700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20c0102ef0 0x7f20c01a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20c0102570 0x7f20c01998b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 --2- 172.21.15.28:0/670792652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20c0100e20 0x7f20c0199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 >> 172.21.15.28:0/670792652 conn(0x7f20c00fc7d0 msgr2=0x7f20c0106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 shutdown_connections 2024-09-15T21:54:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:52.776+0000 7f20c4d48640 1 -- 172.21.15.28:0/670792652 wait complete. 2024-09-15T21:54:53.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:53.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/670792652' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:53.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.453 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-15T21:54:53.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-15T21:54:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/670792652' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T21:54:53.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:53.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:53.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/670792652' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:53.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:54:53.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:54:53.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:54:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 -- 172.21.15.28:0/3074483694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8101de0 msgr2=0x7f63c81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3074483694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8101de0 0x7f63c81021c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009900 tx=0x7f63b802a3f0 comp rx=0 tx=0).stop 2024-09-15T21:54:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 -- 172.21.15.28:0/3074483694 shutdown_connections 2024-09-15T21:54:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3074483694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c8106f10 0x7f63c810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3074483694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c8102790 0x7f63c81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3074483694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8101de0 0x7f63c81021c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:54.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 -- 172.21.15.28:0/3074483694 >> 172.21.15.28:0/3074483694 conn(0x7f63c80fd8c0 msgr2=0x7f63c80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 -- 172.21.15.28:0/3074483694 shutdown_connections 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.189+0000 7f63cd62e640 1 -- 172.21.15.28:0/3074483694 wait complete. 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 Processor -- start 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 -- start start 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c8101de0 0x7f63c819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c8102790 0x7f63c819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8106f10 0x7f63c819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8111f90 con 0x7f63c8106f10 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8111e10 con 0x7f63c8102790 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63cd62e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8112110 con 0x7f63c8101de0 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8106f10 0x7f63c819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:54.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8106f10 0x7f63c819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46130/0 (socket says 172.21.15.28:46130) 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.190+0000 7f63c77fe640 1 -- 172.21.15.28:0/3601405894 learned_addr learned my addr 172.21.15.28:0/3601405894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c6ffd640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c8101de0 0x7f63c819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63be7fc640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c8102790 0x7f63c819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c77fe640 1 -- 172.21.15.28:0/3601405894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c8101de0 msgr2=0x7f63c819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c77fe640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c8101de0 0x7f63c819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c77fe640 1 -- 172.21.15.28:0/3601405894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c8102790 msgr2=0x7f63c819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c77fe640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c8102790 0x7f63c819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c77fe640 1 -- 172.21.15.28:0/3601405894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c819f6a0 con 0x7f63c8106f10 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c6ffd640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c8101de0 0x7f63c819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c77fe640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8106f10 0x7f63c819efc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f63c000b970 tx=0x7f63c000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c4ff9640 1 -- 172.21.15.28:0/3601405894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0019070 con 0x7f63c8106f10 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c4ff9640 1 -- 172.21.15.28:0/3601405894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63c000cb40 con 0x7f63c8106f10 2024-09-15T21:54:54.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c819f990 con 0x7f63c8106f10 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63c4ff9640 1 -- 172.21.15.28:0/3601405894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0007650 con 0x7f63c8106f10 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.191+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c81a7270 con 0x7f63c8106f10 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.192+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c81021c0 con 0x7f63c8106f10 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.193+0000 7f63c4ff9640 1 -- 172.21.15.28:0/3601405894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f63c001c070 con 0x7f63c8106f10 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.194+0000 7f63c4ff9640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f63b40a32c0 0x7f63b40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.194+0000 7f63c6ffd640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f63b40a32c0 0x7f63b40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.194+0000 7f63c4ff9640 1 -- 172.21.15.28:0/3601405894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f63c00ceea0 con 0x7f63c8106f10 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.194+0000 7f63c6ffd640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f63b40a32c0 0x7f63b40a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005ab0 tx=0x7f63b8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.197+0000 7f63c4ff9640 1 -- 172.21.15.28:0/3601405894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f63c009c060 con 0x7f63c8106f10 2024-09-15T21:54:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:54.374+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f63c81a74a0 con 0x7f63c8106f10 2024-09-15T21:54:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.324+0000 7f63c4ff9640 1 -- 172.21.15.28:0/3601405894 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) ==== 138+0+0 (secure 0 0 0) 0x7f63c009b960 con 0x7f63c8106f10 2024-09-15T21:54:55.326 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-15T21:54:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.327+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f63b40a32c0 msgr2=0x7f63b40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.327+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f63b40a32c0 0x7f63b40a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005ab0 tx=0x7f63b8005a20 comp rx=0 tx=0).stop 2024-09-15T21:54:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.327+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8106f10 msgr2=0x7f63c819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.327+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8106f10 0x7f63c819efc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f63c000b970 tx=0x7f63c000be30 comp rx=0 tx=0).stop 2024-09-15T21:54:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 shutdown_connections 2024-09-15T21:54:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f63b40a32c0 0x7f63b40a5780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c8106f10 0x7f63c819efc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63c8102790 0x7f63c819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 --2- 172.21.15.28:0/3601405894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c8101de0 0x7f63c819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 >> 172.21.15.28:0/3601405894 conn(0x7f63c80fd8c0 msgr2=0x7f63c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 shutdown_connections 2024-09-15T21:54:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:55.328+0000 7f63cd62e640 1 -- 172.21.15.28:0/3601405894 wait complete. 2024-09-15T21:54:55.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:55 smithi028 ceph-mon[30969]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:55.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:55 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3601405894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T21:54:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:55 smithi203 ceph-mon[35546]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:55 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3601405894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T21:54:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:55 smithi174 ceph-mon[35244]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:55 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3601405894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T21:54:55.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-15T21:54:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:56.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:56 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3601405894' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T21:54:56.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:56 smithi028 ceph-mon[30969]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T21:54:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:56 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3601405894' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T21:54:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:56 smithi203 ceph-mon[35546]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T21:54:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:56 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3601405894' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T21:54:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:56 smithi174 ceph-mon[35244]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T21:54:56.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 -- 172.21.15.28:0/3496114956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080100a60 msgr2=0x7f8080100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:56.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 --2- 172.21.15.28:0/3496114956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080100a60 0x7f8080100ec0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8068009930 tx=0x7f806802a420 comp rx=0 tx=0).stop 2024-09-15T21:54:56.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 -- 172.21.15.28:0/3496114956 shutdown_connections 2024-09-15T21:54:56.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 --2- 172.21.15.28:0/3496114956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8080101400 0x7f808010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 --2- 172.21.15.28:0/3496114956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080100a60 0x7f8080100ec0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 --2- 172.21.15.28:0/3496114956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8080106a60 0x7f8080106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 -- 172.21.15.28:0/3496114956 >> 172.21.15.28:0/3496114956 conn(0x7f80800fc780 msgr2=0x7f80800feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 -- 172.21.15.28:0/3496114956 shutdown_connections 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 -- 172.21.15.28:0/3496114956 wait complete. 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.791+0000 7f8084a42640 1 Processor -- start 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8084a42640 1 -- start start 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8084a42640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8080100a60 0x7f8080198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8084a42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080101400 0x7f8080198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8084a42640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8080106a60 0x7f808019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8084a42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8080111f80 con 0x7f8080101400 2024-09-15T21:54:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8084a42640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8080111e00 con 0x7f8080100a60 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8084a42640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112100 con 0x7f8080106a60 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080101400 0x7f8080198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080101400 0x7f8080198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46146/0 (socket says 172.21.15.28:46146) 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 -- 172.21.15.28:0/913055471 learned_addr learned my addr 172.21.15.28:0/913055471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f807ed9d640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8080106a60 0x7f808019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f807e59c640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8080100a60 0x7f8080198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 -- 172.21.15.28:0/913055471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8080106a60 msgr2=0x7f808019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8080106a60 0x7f808019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 -- 172.21.15.28:0/913055471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8080100a60 msgr2=0x7f8080198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8080100a60 0x7f8080198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 -- 172.21.15.28:0/913055471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808019d4e0 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.792+0000 7f8077fff640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080101400 0x7f8080198a70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8068009a50 tx=0x7f8068004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.793+0000 7f80777fe640 1 -- 172.21.15.28:0/913055471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806803f030 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.793+0000 7f80777fe640 1 -- 172.21.15.28:0/913055471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8068003ce0 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.793+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808019d770 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.793+0000 7f80777fe640 1 -- 172.21.15.28:0/913055471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068035070 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.793+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80801a50b0 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.794+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8080100ec0 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.795+0000 7f80777fe640 1 -- 172.21.15.28:0/913055471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f8068042030 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.796+0000 7f80777fe640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f806c0a32c0 0x7f806c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.796+0000 7f807e59c640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f806c0a32c0 0x7f806c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.796+0000 7f80777fe640 1 -- 172.21.15.28:0/913055471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7497+0+0 (secure 0 0 0) 0x7f80680efff0 con 0x7f8080101400 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.797+0000 7f807e59c640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f806c0a32c0 0x7f806c0a5780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8070009890 tx=0x7f8070008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.800+0000 7f80777fe640 1 -- 172.21.15.28:0/913055471 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f80680bd040 con 0x7f8080101400 2024-09-15T21:54:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:56.979+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f8080108ff0 con 0x7f8080101400 2024-09-15T21:54:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.415+0000 7f80777fe640 1 -- 172.21.15.28:0/913055471 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v80) ==== 171+0+0 (secure 0 0 0) 0x7f80680bc940 con 0x7f8080101400 2024-09-15T21:54:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-15T21:54:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f806c0a32c0 msgr2=0x7f806c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f806c0a32c0 0x7f806c0a5780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8070009890 tx=0x7f8070008040 comp rx=0 tx=0).stop 2024-09-15T21:54:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080101400 msgr2=0x7f8080198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080101400 0x7f8080198a70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8068009a50 tx=0x7f8068004270 comp rx=0 tx=0).stop 2024-09-15T21:54:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 shutdown_connections 2024-09-15T21:54:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f806c0a32c0 0x7f806c0a5780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8080106a60 0x7f808019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:57.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080101400 0x7f8080198a70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:57.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 --2- 172.21.15.28:0/913055471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8080100a60 0x7f8080198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:57.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 >> 172.21.15.28:0/913055471 conn(0x7f80800fc780 msgr2=0x7f80800fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:57.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 shutdown_connections 2024-09-15T21:54:57.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:57.418+0000 7f8084a42640 1 -- 172.21.15.28:0/913055471 wait complete. 2024-09-15T21:54:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:57 smithi028 ceph-mon[30969]: pgmap v277: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:57 smithi028 ceph-mon[30969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T21:54:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:57 smithi028 ceph-mon[30969]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T21:54:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:57 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/913055471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T21:54:57.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:57 smithi203 ceph-mon[35546]: pgmap v277: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:57.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:57 smithi203 ceph-mon[35546]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T21:54:57.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:57 smithi203 ceph-mon[35546]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T21:54:57.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:57 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/913055471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T21:54:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:57 smithi174 ceph-mon[35244]: pgmap v277: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:57 smithi174 ceph-mon[35244]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T21:54:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:57 smithi174 ceph-mon[35244]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T21:54:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:57 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/913055471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T21:54:58.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-15T21:54:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:54:58.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:58 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/913055471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T21:54:58.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:58 smithi203 ceph-mon[35546]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T21:54:58.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:58 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/913055471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T21:54:58.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:58 smithi028 ceph-mon[30969]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T21:54:58.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:58 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/913055471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T21:54:58.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:58 smithi174 ceph-mon[35244]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 -- 172.21.15.28:0/3505390342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930103660 msgr2=0x7f493010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 --2- 172.21.15.28:0/3505390342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930103660 0x7f493010bb50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4924009900 tx=0x7f492402a480 comp rx=0 tx=0).stop 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 -- 172.21.15.28:0/3505390342 shutdown_connections 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 --2- 172.21.15.28:0/3505390342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f493010c170 0x7f493010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 --2- 172.21.15.28:0/3505390342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930103660 0x7f493010bb50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 --2- 172.21.15.28:0/3505390342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4930102d40 0x7f4930103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 -- 172.21.15.28:0/3505390342 >> 172.21.15.28:0/3505390342 conn(0x7f49300fc7a0 msgr2=0x7f49300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 -- 172.21.15.28:0/3505390342 shutdown_connections 2024-09-15T21:54:58.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.829+0000 7f4935b62640 1 -- 172.21.15.28:0/3505390342 wait complete. 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 Processor -- start 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 -- start start 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930102d40 0x7f493019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4930103660 0x7f493019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f493010c170 0x7f493019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111fb0 con 0x7f493010c170 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111e30 con 0x7f4930103660 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.830+0000 7f4935b62640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4930112130 con 0x7f4930102d40 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f493010c170 0x7f493019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f493010c170 0x7f493019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46156/0 (socket says 172.21.15.28:46156) 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 -- 172.21.15.28:0/3691532661 learned_addr learned my addr 172.21.15.28:0/3691532661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:54:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492effd640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4930103660 0x7f493019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492f7fe640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930102d40 0x7f493019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 -- 172.21.15.28:0/3691532661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930102d40 msgr2=0x7f493019a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930102d40 0x7f493019a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 -- 172.21.15.28:0/3691532661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4930103660 msgr2=0x7f493019abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4930103660 0x7f493019abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 -- 172.21.15.28:0/3691532661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f493019f640 con 0x7f493010c170 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492f7fe640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930102d40 0x7f493019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492ffff640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f493010c170 0x7f493019ef60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f492800b970 tx=0x7f492800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492cff9640 1 -- 172.21.15.28:0/3691532661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928019070 con 0x7f493010c170 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492cff9640 1 -- 172.21.15.28:0/3691532661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f492800cb40 con 0x7f493010c170 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f493019f930 con 0x7f493010c170 2024-09-15T21:54:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f492cff9640 1 -- 172.21.15.28:0/3691532661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928007650 con 0x7f493010c170 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.831+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49301a7210 con 0x7f493010c170 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.832+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4930103120 con 0x7f493010c170 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.834+0000 7f492cff9640 1 -- 172.21.15.28:0/3691532661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f492801c070 con 0x7f493010c170 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.834+0000 7f492cff9640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f49180a32c0 0x7f49180a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.834+0000 7f492f7fe640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f49180a32c0 0x7f49180a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.834+0000 7f492cff9640 1 -- 172.21.15.28:0/3691532661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7879+0+0 (secure 0 0 0) 0x7f49280cf190 con 0x7f493010c170 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.835+0000 7f492f7fe640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f49180a32c0 0x7f49180a5780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4920009b40 tx=0x7f4920005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:54:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:58.838+0000 7f492cff9640 1 -- 172.21.15.28:0/3691532661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f492809c060 con 0x7f493010c170 2024-09-15T21:54:59.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.047+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f493019b9f0 con 0x7f493010c170 2024-09-15T21:54:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.423+0000 7f492cff9640 1 -- 172.21.15.28:0/3691532661 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f493019b9f0 con 0x7f493010c170 2024-09-15T21:54:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-09-15T21:54:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.425+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f49180a32c0 msgr2=0x7f49180a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.425+0000 7f4935b62640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f49180a32c0 0x7f49180a5780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4920009b40 tx=0x7f4920005e50 comp rx=0 tx=0).stop 2024-09-15T21:54:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.425+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f493010c170 msgr2=0x7f493019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.425+0000 7f4935b62640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f493010c170 0x7f493019ef60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f492800b970 tx=0x7f492800be30 comp rx=0 tx=0).stop 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 shutdown_connections 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f49180a32c0 0x7f49180a5780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f493010c170 0x7f493019ef60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4930103660 0x7f493019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 --2- 172.21.15.28:0/3691532661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930102d40 0x7f493019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 >> 172.21.15.28:0/3691532661 conn(0x7f49300fc7a0 msgr2=0x7f49301008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 shutdown_connections 2024-09-15T21:54:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:54:59.426+0000 7f4935b62640 1 -- 172.21.15.28:0/3691532661 wait complete. 2024-09-15T21:54:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:59 smithi203 ceph-mon[35546]: pgmap v280: 97 pgs: 8 creating+peering, 85 unknown, 4 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:59 smithi203 ceph-mon[35546]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T21:54:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:59 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3691532661' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T21:54:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:54:59 smithi203 ceph-mon[35546]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T21:54:59.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:59 smithi028 ceph-mon[30969]: pgmap v280: 97 pgs: 8 creating+peering, 85 unknown, 4 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:59.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:59 smithi028 ceph-mon[30969]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T21:54:59.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:59 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3691532661' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T21:54:59.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:54:59 smithi028 ceph-mon[30969]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T21:54:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:59 smithi174 ceph-mon[35244]: pgmap v280: 97 pgs: 8 creating+peering, 85 unknown, 4 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:54:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:59 smithi174 ceph-mon[35244]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T21:54:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:59 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3691532661' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T21:54:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:54:59 smithi174 ceph-mon[35244]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T21:55:00.096 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-15T21:55:00.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-15T21:55:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:00.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:00 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3691532661' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T21:55:00.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:00 smithi028 ceph-mon[30969]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T21:55:00.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:00 smithi028 ceph-mon[30969]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T21:55:00.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:00 smithi028 ceph-mon[30969]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T21:55:00.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:00 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3691532661' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T21:55:00.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:00 smithi203 ceph-mon[35546]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T21:55:00.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:00 smithi203 ceph-mon[35546]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T21:55:00.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:00 smithi203 ceph-mon[35546]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T21:55:00.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:00 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3691532661' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T21:55:00.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:00 smithi174 ceph-mon[35244]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T21:55:00.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:00 smithi174 ceph-mon[35244]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T21:55:00.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:00 smithi174 ceph-mon[35244]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T21:55:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 -- 172.21.15.28:0/119006094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102ef0 msgr2=0x7f3da410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/119006094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102ef0 0x7f3da410da40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009900 tx=0x7f3d9c02a480 comp rx=0 tx=0).stop 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 -- 172.21.15.28:0/119006094 shutdown_connections 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/119006094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102ef0 0x7f3da410da40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/119006094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3da4102570 0x7f3da41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/119006094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da4100e20 0x7f3da4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 -- 172.21.15.28:0/119006094 >> 172.21.15.28:0/119006094 conn(0x7f3da40fc7d0 msgr2=0x7f3da40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 -- 172.21.15.28:0/119006094 shutdown_connections 2024-09-15T21:55:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.890+0000 7f3dab8a4640 1 -- 172.21.15.28:0/119006094 wait complete. 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 Processor -- start 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 -- start start 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3da4100e20 0x7f3da41994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102570 0x7f3da41999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da4102ef0 0x7f3da41a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4111020 con 0x7f3da4102570 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4110ea0 con 0x7f3da4100e20 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3dab8a4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3da41111a0 con 0x7f3da4102ef0 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102570 0x7f3da41999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102570 0x7f3da41999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33728/0 (socket says 172.21.15.28:33728) 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 -- 172.21.15.28:0/2168139964 learned_addr learned my addr 172.21.15.28:0/2168139964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da9640640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3da4100e20 0x7f3da41994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 -- 172.21.15.28:0/2168139964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da4102ef0 msgr2=0x7f3da41a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da4102ef0 0x7f3da41a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 -- 172.21.15.28:0/2168139964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3da4100e20 msgr2=0x7f3da41994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3da4100e20 0x7f3da41994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 -- 172.21.15.28:0/2168139964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da41a4a70 con 0x7f3da4102570 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.891+0000 7f3da8e3f640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102570 0x7f3da41999f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400b4a0 tx=0x7f3d9400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.892+0000 7f3d9a7fc640 1 -- 172.21.15.28:0/2168139964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94014070 con 0x7f3da4102570 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.892+0000 7f3d9a7fc640 1 -- 172.21.15.28:0/2168139964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d94010af0 con 0x7f3da4102570 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.892+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da41a4c40 con 0x7f3da4102570 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.892+0000 7f3d9a7fc640 1 -- 172.21.15.28:0/2168139964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94017d10 con 0x7f3da4102570 2024-09-15T21:55:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.892+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da41a5150 con 0x7f3da4102570 2024-09-15T21:55:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.893+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da4101280 con 0x7f3da4102570 2024-09-15T21:55:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.894+0000 7f3d9a7fc640 1 -- 172.21.15.28:0/2168139964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3d94017070 con 0x7f3da4102570 2024-09-15T21:55:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.895+0000 7f3d9a7fc640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3d900a3240 0x7f3d900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.895+0000 7f3da9640640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3d900a3240 0x7f3d900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.895+0000 7f3d9a7fc640 1 -- 172.21.15.28:0/2168139964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7f3d940cec80 con 0x7f3da4102570 2024-09-15T21:55:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.895+0000 7f3da9640640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3d900a3240 0x7f3d900a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3da0007dc0 tx=0x7f3da00072f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:00.899+0000 7f3d9a7fc640 1 -- 172.21.15.28:0/2168139964 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3d9409ba50 con 0x7f3da4102570 2024-09-15T21:55:01.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.081+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f3da402d7c0 con 0x7f3da4102570 2024-09-15T21:55:01.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.440+0000 7f3d9a7fc640 1 -- 172.21.15.28:0/2168139964 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v83) ==== 157+0+0 (secure 0 0 0) 0x7f3d9409b350 con 0x7f3da4102570 2024-09-15T21:55:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.442+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3d900a3240 msgr2=0x7f3d900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.442+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3d900a3240 0x7f3d900a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3da0007dc0 tx=0x7f3da00072f0 comp rx=0 tx=0).stop 2024-09-15T21:55:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.442+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102570 msgr2=0x7f3da41999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.442+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102570 0x7f3da41999f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400b4a0 tx=0x7f3d9400b960 comp rx=0 tx=0).stop 2024-09-15T21:55:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 shutdown_connections 2024-09-15T21:55:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3d900a3240 0x7f3d900a5700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3da4102ef0 0x7f3da41a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3da4102570 0x7f3da41999f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 --2- 172.21.15.28:0/2168139964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3da4100e20 0x7f3da41994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 >> 172.21.15.28:0/2168139964 conn(0x7f3da40fc7d0 msgr2=0x7f3da40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 shutdown_connections 2024-09-15T21:55:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:01.443+0000 7f3dab8a4640 1 -- 172.21.15.28:0/2168139964 wait complete. 2024-09-15T21:55:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:01 smithi028 ceph-mon[30969]: pgmap v283: 97 pgs: 20 creating+peering, 35 unknown, 42 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:01 smithi028 ceph-mon[30969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T21:55:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:01 smithi028 ceph-mon[30969]: Cluster is now healthy 2024-09-15T21:55:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:01 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] up:active 2024-09-15T21:55:01.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:01 smithi028 ceph-mon[30969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:01.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:01 smithi028 ceph-mon[30969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:01.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:01 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2168139964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T21:55:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:01 smithi203 ceph-mon[35546]: pgmap v283: 97 pgs: 20 creating+peering, 35 unknown, 42 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:01 smithi203 ceph-mon[35546]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T21:55:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:01 smithi203 ceph-mon[35546]: Cluster is now healthy 2024-09-15T21:55:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:01 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] up:active 2024-09-15T21:55:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:01 smithi203 ceph-mon[35546]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:01 smithi203 ceph-mon[35546]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:01 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2168139964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T21:55:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:01 smithi174 ceph-mon[35244]: pgmap v283: 97 pgs: 20 creating+peering, 35 unknown, 42 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:01 smithi174 ceph-mon[35244]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T21:55:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:01 smithi174 ceph-mon[35244]: Cluster is now healthy 2024-09-15T21:55:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:01 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] up:active 2024-09-15T21:55:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:01 smithi174 ceph-mon[35244]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:01.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:01 smithi174 ceph-mon[35244]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:01.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:01 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2168139964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T21:55:02.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-15T21:55:02.296 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:02.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:02 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2168139964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T21:55:02.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:02 smithi028 ceph-mon[30969]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T21:55:02.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:02 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2168139964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T21:55:02.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:02 smithi203 ceph-mon[35546]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T21:55:02.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:02 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2168139964' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T21:55:02.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:02 smithi174 ceph-mon[35244]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T21:55:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 -- 172.21.15.28:0/1170113564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 msgr2=0x7fef3c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 --2- 172.21.15.28:0/1170113564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c10e610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef34009ca0 tx=0x7fef3402a950 comp rx=0 tx=0).stop 2024-09-15T21:55:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 -- 172.21.15.28:0/1170113564 shutdown_connections 2024-09-15T21:55:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 --2- 172.21.15.28:0/1170113564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c10e610 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 --2- 172.21.15.28:0/1170113564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef3c103660 0x7fef3c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 --2- 172.21.15.28:0/1170113564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef3c102d40 0x7fef3c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:02.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 -- 172.21.15.28:0/1170113564 >> 172.21.15.28:0/1170113564 conn(0x7fef3c0fc7a0 msgr2=0x7fef3c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:02.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 -- 172.21.15.28:0/1170113564 shutdown_connections 2024-09-15T21:55:02.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 -- 172.21.15.28:0/1170113564 wait complete. 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.867+0000 7fef42b49640 1 Processor -- start 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef42b49640 1 -- start start 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef42b49640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef3c102d40 0x7fef3c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef42b49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef3c103660 0x7fef3c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef42b49640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef42b49640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c111fb0 con 0x7fef3c10c170 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef42b49640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c111e30 con 0x7fef3c103660 2024-09-15T21:55:02.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef42b49640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c112130 con 0x7fef3c102d40 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33748/0 (socket says 172.21.15.28:33748) 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 -- 172.21.15.28:0/464784689 learned_addr learned my addr 172.21.15.28:0/464784689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef3bfff640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef3c103660 0x7fef3c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef408e5640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef3c102d40 0x7fef3c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 -- 172.21.15.28:0/464784689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef3c102d40 msgr2=0x7fef3c1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef3c102d40 0x7fef3c1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 -- 172.21.15.28:0/464784689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef3c103660 msgr2=0x7fef3c100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef3c103660 0x7fef3c100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:02.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.868+0000 7fef410e6640 1 -- 172.21.15.28:0/464784689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3c1a6890 con 0x7fef3c10c170 2024-09-15T21:55:02.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.869+0000 7fef410e6640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c101420 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fef34006010 tx=0x7fef3402f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:02.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.869+0000 7fef39ffb640 1 -- 172.21.15.28:0/464784689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34034040 con 0x7fef3c10c170 2024-09-15T21:55:02.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.869+0000 7fef39ffb640 1 -- 172.21.15.28:0/464784689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef34003e00 con 0x7fef3c10c170 2024-09-15T21:55:02.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.869+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3c1a6b20 con 0x7fef3c10c170 2024-09-15T21:55:02.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.869+0000 7fef39ffb640 1 -- 172.21.15.28:0/464784689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34045480 con 0x7fef3c10c170 2024-09-15T21:55:02.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.869+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3c1a7000 con 0x7fef3c10c170 2024-09-15T21:55:02.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.870+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef3c103120 con 0x7fef3c10c170 2024-09-15T21:55:02.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.871+0000 7fef39ffb640 1 -- 172.21.15.28:0/464784689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fef34037020 con 0x7fef3c10c170 2024-09-15T21:55:02.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.872+0000 7fef39ffb640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fef2c0a3240 0x7fef2c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:02.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.872+0000 7fef408e5640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fef2c0a3240 0x7fef2c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:02.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.872+0000 7fef39ffb640 1 -- 172.21.15.28:0/464784689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fef340f0a30 con 0x7fef3c10c170 2024-09-15T21:55:02.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.872+0000 7fef408e5640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fef2c0a3240 0x7fef2c0a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef30009990 tx=0x7fef30009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:02.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:02.876+0000 7fef39ffb640 1 -- 172.21.15.28:0/464784689 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fef340bd6c0 con 0x7fef3c10c170 2024-09-15T21:55:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:03.053+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fef3c1a7230 con 0x7fef3c10c170 2024-09-15T21:55:03.760 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:55:03 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: 2024-09-15T21:55:03.495+0000 7f08b7d22640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-15T21:55:04.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:03 smithi028 ceph-mon[30969]: pgmap v285: 97 pgs: 12 creating+peering, 13 unknown, 72 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 745 B/s wr, 2 op/s 2024-09-15T21:55:04.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:03 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/464784689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T21:55:04.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:03 smithi203 ceph-mon[35546]: pgmap v285: 97 pgs: 12 creating+peering, 13 unknown, 72 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 745 B/s wr, 2 op/s 2024-09-15T21:55:04.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:03 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/464784689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T21:55:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:03 smithi174 ceph-mon[35244]: pgmap v285: 97 pgs: 12 creating+peering, 13 unknown, 72 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 745 B/s wr, 2 op/s 2024-09-15T21:55:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:03 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/464784689' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T21:55:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.451+0000 7fef39ffb640 1 -- 172.21.15.28:0/464784689 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) ==== 228+0+0 (secure 0 0 0) 0x7fef340bcfc0 con 0x7fef3c10c170 2024-09-15T21:55:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fef2c0a3240 msgr2=0x7fef2c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fef2c0a3240 0x7fef2c0a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef30009990 tx=0x7fef30009110 comp rx=0 tx=0).stop 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 msgr2=0x7fef3c101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c101420 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fef34006010 tx=0x7fef3402f710 comp rx=0 tx=0).stop 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 shutdown_connections 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fef2c0a3240 0x7fef2c0a5700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef3c10c170 0x7fef3c101420 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef3c103660 0x7fef3c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 --2- 172.21.15.28:0/464784689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef3c102d40 0x7fef3c1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 >> 172.21.15.28:0/464784689 conn(0x7fef3c0fc7a0 msgr2=0x7fef3c0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 shutdown_connections 2024-09-15T21:55:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:04.455+0000 7fef42b49640 1 -- 172.21.15.28:0/464784689 wait complete. 2024-09-15T21:55:05.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:04 smithi028 ceph-mon[30969]: pgmap v287: 97 pgs: 97 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-09-15T21:55:05.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:04 smithi028 ceph-mon[30969]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T21:55:05.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:04 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/464784689' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T21:55:05.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:04 smithi028 ceph-mon[30969]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T21:55:05.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-15T21:55:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:04 smithi203 ceph-mon[35546]: pgmap v287: 97 pgs: 97 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-09-15T21:55:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:04 smithi203 ceph-mon[35546]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T21:55:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:04 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/464784689' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T21:55:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:04 smithi203 ceph-mon[35546]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T21:55:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:04 smithi174 ceph-mon[35244]: pgmap v287: 97 pgs: 97 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-09-15T21:55:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:04 smithi174 ceph-mon[35244]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T21:55:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:04 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/464784689' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T21:55:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:04 smithi174 ceph-mon[35244]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T21:55:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 -- 172.21.15.28:0/4037006503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 msgr2=0x7fd2101067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 --2- 172.21.15.28:0/4037006503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd2101067e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009900 tx=0x7fd1f802a410 comp rx=0 tx=0).stop 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 -- 172.21.15.28:0/4037006503 shutdown_connections 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 --2- 172.21.15.28:0/4037006503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd210106f10 0x7fd21010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 --2- 172.21.15.28:0/4037006503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd2101067e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 --2- 172.21.15.28:0/4037006503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd210101de0 0x7fd2101021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 -- 172.21.15.28:0/4037006503 >> 172.21.15.28:0/4037006503 conn(0x7fd2100fd8c0 msgr2=0x7fd2100ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 -- 172.21.15.28:0/4037006503 shutdown_connections 2024-09-15T21:55:05.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.823+0000 7fd214c55640 1 -- 172.21.15.28:0/4037006503 wait complete. 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 Processor -- start 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 -- start start 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd210101de0 0x7fd21019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd21019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd210106f10 0x7fd21019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd210111f90 con 0x7fd210102790 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd210111e10 con 0x7fd210106f10 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd214c55640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd210112110 con 0x7fd210101de0 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd207fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd21019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd207fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd21019abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33764/0 (socket says 172.21.15.28:33764) 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd207fff640 1 -- 172.21.15.28:0/4193471988 learned_addr learned my addr 172.21.15.28:0/4193471988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd20e59c640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd210101de0 0x7fd21019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.824+0000 7fd20ed9d640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd210106f10 0x7fd21019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd207fff640 1 -- 172.21.15.28:0/4193471988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd210101de0 msgr2=0x7fd21019a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd207fff640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd210101de0 0x7fd21019a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd207fff640 1 -- 172.21.15.28:0/4193471988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd210106f10 msgr2=0x7fd21019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd207fff640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd210106f10 0x7fd21019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd207fff640 1 -- 172.21.15.28:0/4193471988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21019f640 con 0x7fd210102790 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd207fff640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd21019abd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8005ab0 tx=0x7fd1f80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd2077fe640 1 -- 172.21.15.28:0/4193471988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f803f030 con 0x7fd210102790 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd21019f8d0 con 0x7fd210102790 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2101a7210 con 0x7fd210102790 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd2077fe640 1 -- 172.21.15.28:0/4193471988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f8003ce0 con 0x7fd210102790 2024-09-15T21:55:05.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.825+0000 7fd2077fe640 1 -- 172.21.15.28:0/4193471988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8035070 con 0x7fd210102790 2024-09-15T21:55:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.826+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f0005290 con 0x7fd210102790 2024-09-15T21:55:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.828+0000 7fd2077fe640 1 -- 172.21.15.28:0/4193471988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd1f8042030 con 0x7fd210102790 2024-09-15T21:55:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.829+0000 7fd2077fe640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd1fc0a3340 0x7fd1fc0a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.829+0000 7fd2077fe640 1 -- 172.21.15.28:0/4193471988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8770+0+0 (secure 0 0 0) 0x7fd1f80f04d0 con 0x7fd210102790 2024-09-15T21:55:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.829+0000 7fd20e59c640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd1fc0a3340 0x7fd1fc0a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:05.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.829+0000 7fd20e59c640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd1fc0a3340 0x7fd1fc0a5800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd20000abe0 tx=0x7fd20000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:05.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:05.832+0000 7fd2077fe640 1 -- 172.21.15.28:0/4193471988 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd1f80bd090 con 0x7fd210102790 2024-09-15T21:55:06.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.006+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fd1f0005580 con 0x7fd210102790 2024-09-15T21:55:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.458+0000 7fd2077fe640 1 -- 172.21.15.28:0/4193471988 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v87) ==== 169+0+0 (secure 0 0 0) 0x7fd1f80bc990 con 0x7fd210102790 2024-09-15T21:55:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-09-15T21:55:06.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd1fc0a3340 msgr2=0x7fd1fc0a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:06.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd1fc0a3340 0x7fd1fc0a5800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd20000abe0 tx=0x7fd20000a2d0 comp rx=0 tx=0).stop 2024-09-15T21:55:06.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 msgr2=0x7fd21019abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd21019abd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8005ab0 tx=0x7fd1f80042f0 comp rx=0 tx=0).stop 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 shutdown_connections 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd1fc0a3340 0x7fd1fc0a5800 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd210106f10 0x7fd21019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd210102790 0x7fd21019abd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 --2- 172.21.15.28:0/4193471988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd210101de0 0x7fd21019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 >> 172.21.15.28:0/4193471988 conn(0x7fd2100fd8c0 msgr2=0x7fd2100febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 shutdown_connections 2024-09-15T21:55:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:06.462+0000 7fd214c55640 1 -- 172.21.15.28:0/4193471988 wait complete. 2024-09-15T21:55:06.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:06 smithi203 ceph-mon[35546]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T21:55:06.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:06 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4193471988' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T21:55:06.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:06 smithi028 ceph-mon[30969]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T21:55:06.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:06 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4193471988' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T21:55:06.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:06 smithi174 ceph-mon[35244]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T21:55:06.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:06 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4193471988' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T21:55:07.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-15T21:55:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:07 smithi203 ceph-mon[35546]: pgmap v290: 129 pgs: 2 creating+activating, 6 creating+peering, 19 unknown, 102 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-09-15T21:55:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:07 smithi203 ceph-mon[35546]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T21:55:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:07 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4193471988' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T21:55:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:07 smithi203 ceph-mon[35546]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T21:55:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:07 smithi174 ceph-mon[35244]: pgmap v290: 129 pgs: 2 creating+activating, 6 creating+peering, 19 unknown, 102 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-09-15T21:55:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:07 smithi174 ceph-mon[35244]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T21:55:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:07 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4193471988' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T21:55:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:07 smithi174 ceph-mon[35244]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T21:55:07.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:07 smithi028 ceph-mon[30969]: pgmap v290: 129 pgs: 2 creating+activating, 6 creating+peering, 19 unknown, 102 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-09-15T21:55:07.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:07 smithi028 ceph-mon[30969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T21:55:07.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:07 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4193471988' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T21:55:07.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:07 smithi028 ceph-mon[30969]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T21:55:07.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 -- 172.21.15.28:0/3556670755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681036e0 msgr2=0x7f8268107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 --2- 172.21.15.28:0/3556670755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681036e0 0x7f8268107b90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8258009930 tx=0x7f825802a420 comp rx=0 tx=0).stop 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 -- 172.21.15.28:0/3556670755 shutdown_connections 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 --2- 172.21.15.28:0/3556670755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82681082a0 0x7f826810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 --2- 172.21.15.28:0/3556670755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681036e0 0x7f8268107b90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 --2- 172.21.15.28:0/3556670755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8268102d30 0x7f8268103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 -- 172.21.15.28:0/3556670755 >> 172.21.15.28:0/3556670755 conn(0x7f82680fc7d0 msgr2=0x7f82680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 -- 172.21.15.28:0/3556670755 shutdown_connections 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.888+0000 7f827070e640 1 -- 172.21.15.28:0/3556670755 wait complete. 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 Processor -- start 2024-09-15T21:55:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 -- start start 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102d30 0x7f8268198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82681036e0 0x7f8268198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681082a0 0x7f826819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8268112130 con 0x7f82681082a0 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8268111fb0 con 0x7f8268102d30 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f827070e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82681122b0 con 0x7f82681036e0 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681082a0 0x7f826819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681082a0 0x7f826819cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33792/0 (socket says 172.21.15.28:33792) 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 -- 172.21.15.28:0/848116894 learned_addr learned my addr 172.21.15.28:0/848116894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826e4aa640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102d30 0x7f8268198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826dca9640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82681036e0 0x7f8268198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 -- 172.21.15.28:0/848116894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82681036e0 msgr2=0x7f8268198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82681036e0 0x7f8268198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 -- 172.21.15.28:0/848116894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102d30 msgr2=0x7f8268198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102d30 0x7f8268198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.889+0000 7f826ecab640 1 -- 172.21.15.28:0/848116894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826819d5d0 con 0x7f82681082a0 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.890+0000 7f826ecab640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681082a0 0x7f826819cef0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f826000b970 tx=0x7f826000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.890+0000 7f825f7fe640 1 -- 172.21.15.28:0/848116894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260019070 con 0x7f82681082a0 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.890+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826819d8c0 con 0x7f82681082a0 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.890+0000 7f825f7fe640 1 -- 172.21.15.28:0/848116894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826000cb40 con 0x7f82681082a0 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.890+0000 7f825f7fe640 1 -- 172.21.15.28:0/848116894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260007650 con 0x7f82681082a0 2024-09-15T21:55:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.890+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82681a51a0 con 0x7f82681082a0 2024-09-15T21:55:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.891+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8268103110 con 0x7f82681082a0 2024-09-15T21:55:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.892+0000 7f825f7fe640 1 -- 172.21.15.28:0/848116894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f826001c070 con 0x7f82681082a0 2024-09-15T21:55:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.893+0000 7f825f7fe640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f82540a3240 0x7f82540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.893+0000 7f826e4aa640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f82540a3240 0x7f82540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.893+0000 7f825f7fe640 1 -- 172.21.15.28:0/848116894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f82600cf1d0 con 0x7f82681082a0 2024-09-15T21:55:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.893+0000 7f826e4aa640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f82540a3240 0x7f82540a5700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8264005fd0 tx=0x7f8264005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:07.897+0000 7f825f7fe640 1 -- 172.21.15.28:0/848116894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f826009bcd0 con 0x7f82681082a0 2024-09-15T21:55:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.108+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f826810a2b0 con 0x7f82681082a0 2024-09-15T21:55:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.466+0000 7f825f7fe640 1 -- 172.21.15.28:0/848116894 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f826009b5d0 con 0x7f82681082a0 2024-09-15T21:55:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f82540a3240 msgr2=0x7f82540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f82540a3240 0x7f82540a5700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8264005fd0 tx=0x7f8264005850 comp rx=0 tx=0).stop 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681082a0 msgr2=0x7f826819cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681082a0 0x7f826819cef0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f826000b970 tx=0x7f826000be30 comp rx=0 tx=0).stop 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 shutdown_connections 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f82540a3240 0x7f82540a5700 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82681082a0 0x7f826819cef0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82681036e0 0x7f8268198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 --2- 172.21.15.28:0/848116894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102d30 0x7f8268198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 >> 172.21.15.28:0/848116894 conn(0x7f82680fc7d0 msgr2=0x7f8268105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:08.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 shutdown_connections 2024-09-15T21:55:08.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:08.469+0000 7f827070e640 1 -- 172.21.15.28:0/848116894 wait complete. 2024-09-15T21:55:08.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:08 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/848116894' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T21:55:08.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:08 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/848116894' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T21:55:08.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:08 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/848116894' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T21:55:09.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:55:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:09.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:09 smithi028 ceph-mon[30969]: pgmap v292: 129 pgs: 2 creating+activating, 6 creating+peering, 9 unknown, 112 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 B/s wr, 6 op/s 2024-09-15T21:55:09.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:09 smithi028 ceph-mon[30969]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T21:55:09.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:09 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/848116894' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T21:55:09.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:09 smithi028 ceph-mon[30969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:09 smithi203 ceph-mon[35546]: pgmap v292: 129 pgs: 2 creating+activating, 6 creating+peering, 9 unknown, 112 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 B/s wr, 6 op/s 2024-09-15T21:55:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:09 smithi203 ceph-mon[35546]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T21:55:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:09 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/848116894' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T21:55:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:09 smithi203 ceph-mon[35546]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:09 smithi174 ceph-mon[35244]: pgmap v292: 129 pgs: 2 creating+activating, 6 creating+peering, 9 unknown, 112 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 B/s wr, 6 op/s 2024-09-15T21:55:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:09 smithi174 ceph-mon[35244]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T21:55:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:09 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/848116894' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T21:55:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:09 smithi174 ceph-mon[35244]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 -- 172.21.15.28:0/920812775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108101420 msgr2=0x7f510810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 --2- 172.21.15.28:0/920812775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108101420 0x7f510810d950 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5100009cd0 tx=0x7f510002a8e0 comp rx=0 tx=0).stop 2024-09-15T21:55:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 -- 172.21.15.28:0/920812775 shutdown_connections 2024-09-15T21:55:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 --2- 172.21.15.28:0/920812775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108101420 0x7f510810d950 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 --2- 172.21.15.28:0/920812775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108100a80 0x7f5108100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:09.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 --2- 172.21.15.28:0/920812775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5108106a80 0x7f5108106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 -- 172.21.15.28:0/920812775 >> 172.21.15.28:0/920812775 conn(0x7f51080fc7a0 msgr2=0x7f51080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 -- 172.21.15.28:0/920812775 shutdown_connections 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 -- 172.21.15.28:0/920812775 wait complete. 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 Processor -- start 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.794+0000 7f510d172640 1 -- start start 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f510d172640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108100a80 0x7f510819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f510d172640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5108101420 0x7f510819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f510d172640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108106a80 0x7f510819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f510d172640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5108110ec0 con 0x7f5108100a80 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f510d172640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5108110d40 con 0x7f5108106a80 2024-09-15T21:55:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f510d172640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5108111040 con 0x7f5108101420 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108100a80 0x7f510819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108100a80 0x7f510819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33808/0 (socket says 172.21.15.28:33808) 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 -- 172.21.15.28:0/2973425784 learned_addr learned my addr 172.21.15.28:0/2973425784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f510759e640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108106a80 0x7f510819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f50fffff640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5108101420 0x7f510819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 -- 172.21.15.28:0/2973425784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5108101420 msgr2=0x7f510819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5108101420 0x7f510819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 -- 172.21.15.28:0/2973425784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108106a80 msgr2=0x7f510819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108106a80 0x7f510819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 -- 172.21.15.28:0/2973425784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510819f6d0 con 0x7f5108100a80 2024-09-15T21:55:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f50fffff640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5108101420 0x7f510819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.795+0000 7f5106d9d640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108100a80 0x7f510819a720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50f800d770 tx=0x7f50f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.796+0000 7f5104d99640 1 -- 172.21.15.28:0/2973425784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8017070 con 0x7f5108100a80 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.796+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510819f9c0 con 0x7f5108100a80 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.796+0000 7f5104d99640 1 -- 172.21.15.28:0/2973425784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f8005500 con 0x7f5108100a80 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.796+0000 7f5104d99640 1 -- 172.21.15.28:0/2973425784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8004e00 con 0x7f5108100a80 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.796+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51081a72a0 con 0x7f5108100a80 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.797+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5108100ee0 con 0x7f5108100a80 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.798+0000 7f5104d99640 1 -- 172.21.15.28:0/2973425784 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f50f8004fa0 con 0x7f5108100a80 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.798+0000 7f5104d99640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f50f40a3240 0x7f50f40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.798+0000 7f50fffff640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f50f40a3240 0x7f50f40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:09.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.799+0000 7f5104d99640 1 -- 172.21.15.28:0/2973425784 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f50f80cf580 con 0x7f5108100a80 2024-09-15T21:55:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.799+0000 7f50fffff640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f50f40a3240 0x7f50f40a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f510810c630 tx=0x7f50f0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:09.803+0000 7f5104d99640 1 -- 172.21.15.28:0/2973425784 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f50f809c060 con 0x7f5108100a80 2024-09-15T21:55:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.014+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51081a7470 con 0x7f5108100a80 2024-09-15T21:55:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.015+0000 7f5104d99640 1 -- 172.21.15.28:0/2973425784 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11851 (secure 0 0 0) 0x7f50f809b960 con 0x7f5108100a80 2024-09-15T21:55:10.017 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:10.017 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-15T21:55:10.018 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"btime":"2024-09-15T21:55:08:461866+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:08.111236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24485,"qdb_cluster":[24485]},"id":1}]} 2024-09-15T21:55:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f50f40a3240 msgr2=0x7f50f40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f50f40a3240 0x7f50f40a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f510810c630 tx=0x7f50f0008040 comp rx=0 tx=0).stop 2024-09-15T21:55:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108100a80 msgr2=0x7f510819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108100a80 0x7f510819a720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50f800d770 tx=0x7f50f800dc30 comp rx=0 tx=0).stop 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 shutdown_connections 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f50f40a3240 0x7f50f40a5700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108106a80 0x7f510819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5108101420 0x7f510819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 --2- 172.21.15.28:0/2973425784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5108100a80 0x7f510819a720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.017+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 >> 172.21.15.28:0/2973425784 conn(0x7f51080fc7a0 msgr2=0x7f510810c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.018+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 shutdown_connections 2024-09-15T21:55:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:10.018+0000 7f510d172640 1 -- 172.21.15.28:0/2973425784 wait complete. 2024-09-15T21:55:10.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:10 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2973425784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:10.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:55:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:10 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2973425784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:10.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:10 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2973425784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.492+0000 7fe0a8732640 1 -- 172.21.15.28:0/704558422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 msgr2=0x7fe0a01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.492+0000 7fe0a8732640 1 --2- 172.21.15.28:0/704558422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01021c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009900 tx=0x7fe09c02a410 comp rx=0 tx=0).stop 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 -- 172.21.15.28:0/704558422 shutdown_connections 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 --2- 172.21.15.28:0/704558422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a0106f10 0x7fe0a010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 --2- 172.21.15.28:0/704558422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a0102790 0x7fe0a01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 --2- 172.21.15.28:0/704558422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01021c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 -- 172.21.15.28:0/704558422 >> 172.21.15.28:0/704558422 conn(0x7fe0a00fd8c0 msgr2=0x7fe0a00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 -- 172.21.15.28:0/704558422 shutdown_connections 2024-09-15T21:55:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 -- 172.21.15.28:0/704558422 wait complete. 2024-09-15T21:55:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 Processor -- start 2024-09-15T21:55:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.493+0000 7fe0a8732640 1 -- start start 2024-09-15T21:55:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a8732640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a8732640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a0102790 0x7fe0a0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a8732640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a0106f10 0x7fe0a019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a8732640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0111f70 con 0x7fe0a0101de0 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a8732640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0111df0 con 0x7fe0a0106f10 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a8732640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01120f0 con 0x7fe0a0102790 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32972/0 (socket says 172.21.15.28:32972) 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 -- 172.21.15.28:0/367857364 learned_addr learned my addr 172.21.15.28:0/367857364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a5ccd640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a0102790 0x7fe0a0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a6ccf640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a0106f10 0x7fe0a019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 -- 172.21.15.28:0/367857364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a0102790 msgr2=0x7fe0a0198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a0102790 0x7fe0a0198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 -- 172.21.15.28:0/367857364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a0106f10 msgr2=0x7fe0a019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a0106f10 0x7fe0a019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a64ce640 1 -- 172.21.15.28:0/367857364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a019d380 con 0x7fe0a0101de0 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.494+0000 7fe0a6ccf640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a0106f10 0x7fe0a019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.495+0000 7fe0a64ce640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01983d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe09c006010 tx=0x7fe09c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.495+0000 7fe0977fe640 1 -- 172.21.15.28:0/367857364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c03f030 con 0x7fe0a0101de0 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.495+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a019d610 con 0x7fe0a0101de0 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.495+0000 7fe0977fe640 1 -- 172.21.15.28:0/367857364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe09c003ce0 con 0x7fe0a0101de0 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.495+0000 7fe0977fe640 1 -- 172.21.15.28:0/367857364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c035070 con 0x7fe0a0101de0 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.495+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a01a4f50 con 0x7fe0a0101de0 2024-09-15T21:55:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.496+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a01021c0 con 0x7fe0a0101de0 2024-09-15T21:55:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.497+0000 7fe0977fe640 1 -- 172.21.15.28:0/367857364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe09c042030 con 0x7fe0a0101de0 2024-09-15T21:55:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.498+0000 7fe0977fe640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe08c0a3240 0x7fe08c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.498+0000 7fe0a5ccd640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe08c0a3240 0x7fe08c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.498+0000 7fe0977fe640 1 -- 172.21.15.28:0/367857364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fe09c0f0510 con 0x7fe0a0101de0 2024-09-15T21:55:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.499+0000 7fe0a5ccd640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe08c0a3240 0x7fe08c0a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00fdce0 tx=0x7fe09000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.501+0000 7fe0977fe640 1 -- 172.21.15.28:0/367857364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe09c0bd050 con 0x7fe0a0101de0 2024-09-15T21:55:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:11 smithi028 ceph-mon[30969]: pgmap v294: 129 pgs: 2 creating+activating, 6 creating+peering, 121 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-15T21:55:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:11 smithi028 ceph-mon[30969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T21:55:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:11 smithi028 ceph-mon[30969]: Cluster is now healthy 2024-09-15T21:55:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.679+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0a010a0f0 con 0x7fe0a0101de0 2024-09-15T21:55:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.680+0000 7fe0977fe640 1 -- 172.21.15.28:0/367857364 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7fe09c0bc950 con 0x7fe0a0101de0 2024-09-15T21:55:11.682 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:11.682 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:08.111056+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:55:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe08c0a3240 msgr2=0x7fe08c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe08c0a3240 0x7fe08c0a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe0a00fdce0 tx=0x7fe09000b390 comp rx=0 tx=0).stop 2024-09-15T21:55:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 msgr2=0x7fe0a01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01983d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe09c006010 tx=0x7fe09c0042f0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 shutdown_connections 2024-09-15T21:55:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe08c0a3240 0x7fe08c0a5700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a0106f10 0x7fe0a019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a0102790 0x7fe0a0198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 --2- 172.21.15.28:0/367857364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a0101de0 0x7fe0a01983d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.682+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 >> 172.21.15.28:0/367857364 conn(0x7fe0a00fd8c0 msgr2=0x7fe0a00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.683+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 shutdown_connections 2024-09-15T21:55:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:11.683+0000 7fe0a8732640 1 -- 172.21.15.28:0/367857364 wait complete. 2024-09-15T21:55:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:11 smithi174 ceph-mon[35244]: pgmap v294: 129 pgs: 2 creating+activating, 6 creating+peering, 121 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-15T21:55:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:11 smithi174 ceph-mon[35244]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T21:55:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:11 smithi174 ceph-mon[35244]: Cluster is now healthy 2024-09-15T21:55:11.869 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:11 smithi203 ceph-mon[35546]: pgmap v294: 129 pgs: 2 creating+activating, 6 creating+peering, 121 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-15T21:55:11.870 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:11 smithi203 ceph-mon[35546]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T21:55:11.870 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:11 smithi203 ceph-mon[35546]: Cluster is now healthy 2024-09-15T21:55:12.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:55:12.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:12 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/367857364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:12.686 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:12.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:12 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/367857364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:12.885 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:12 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/367857364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:13.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 -- 172.21.15.28:0/2469317017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 msgr2=0x7ff6a010bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:13.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/2469317017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a010bbb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009900 tx=0x7ff68c02a480 comp rx=0 tx=0).stop 2024-09-15T21:55:13.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 -- 172.21.15.28:0/2469317017 shutdown_connections 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/2469317017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6a010c180 0x7ff6a010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/2469317017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a010bbb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/2469317017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a0102da0 0x7ff6a0103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 -- 172.21.15.28:0/2469317017 >> 172.21.15.28:0/2469317017 conn(0x7ff6a00fc7a0 msgr2=0x7ff6a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 -- 172.21.15.28:0/2469317017 shutdown_connections 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.142+0000 7ff6a70f5640 1 -- 172.21.15.28:0/2469317017 wait complete. 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 Processor -- start 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 -- start start 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a0102da0 0x7ff6a0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:13.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6a010c180 0x7ff6a019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a0111f60 con 0x7ff6a01036c0 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a0111de0 con 0x7ff6a0102da0 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff6a70f5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a01120e0 con 0x7ff6a010c180 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff69ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff69ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a0198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32990/0 (socket says 172.21.15.28:32990) 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.143+0000 7ff69ffff640 1 -- 172.21.15.28:0/1403356586 learned_addr learned my addr 172.21.15.28:0/1403356586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff6a4e91640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a0102da0 0x7ff6a0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff6a5692640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6a010c180 0x7ff6a019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69ffff640 1 -- 172.21.15.28:0/1403356586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6a010c180 msgr2=0x7ff6a019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69ffff640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6a010c180 0x7ff6a019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69ffff640 1 -- 172.21.15.28:0/1403356586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a0102da0 msgr2=0x7ff6a0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69ffff640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a0102da0 0x7ff6a0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69ffff640 1 -- 172.21.15.28:0/1403356586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a019d3e0 con 0x7ff6a01036c0 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff6a4e91640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a0102da0 0x7ff6a0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69ffff640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a0198970 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff68c006010 tx=0x7ff68c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69dffb640 1 -- 172.21.15.28:0/1403356586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c03f030 con 0x7ff6a01036c0 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a019d670 con 0x7ff6a01036c0 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69dffb640 1 -- 172.21.15.28:0/1403356586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff68c003ce0 con 0x7ff6a01036c0 2024-09-15T21:55:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff69dffb640 1 -- 172.21.15.28:0/1403356586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c035070 con 0x7ff6a01036c0 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.144+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a01a4fb0 con 0x7ff6a01036c0 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.145+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a006b010 con 0x7ff6a01036c0 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.147+0000 7ff69dffb640 1 -- 172.21.15.28:0/1403356586 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff68c042030 con 0x7ff6a01036c0 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.147+0000 7ff69dffb640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff6900a3240 0x7ff6900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.147+0000 7ff6a4e91640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff6900a3240 0x7ff6900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.147+0000 7ff69dffb640 1 -- 172.21.15.28:0/1403356586 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7ff68c0f0400 con 0x7ff6a01036c0 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.148+0000 7ff6a4e91640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff6900a3240 0x7ff6900a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff69400bfd0 tx=0x7ff69400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:13.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.150+0000 7ff69dffb640 1 -- 172.21.15.28:0/1403356586 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff68c0bc950 con 0x7ff6a01036c0 2024-09-15T21:55:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.327+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6a01071c0 con 0x7ff6a01036c0 2024-09-15T21:55:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.329+0000 7ff69dffb640 1 -- 172.21.15.28:0/1403356586 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7ff68c0c0c50 con 0x7ff6a01036c0 2024-09-15T21:55:13.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:13.331 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:08.111056+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:55:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff6900a3240 msgr2=0x7ff6900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff6900a3240 0x7ff6900a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff69400bfd0 tx=0x7ff69400b390 comp rx=0 tx=0).stop 2024-09-15T21:55:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 msgr2=0x7ff6a0198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a0198970 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff68c006010 tx=0x7ff68c0042f0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 shutdown_connections 2024-09-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff6900a3240 0x7ff6900a5700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6a010c180 0x7ff6a019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a01036c0 0x7ff6a0198970 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 --2- 172.21.15.28:0/1403356586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a0102da0 0x7ff6a0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 >> 172.21.15.28:0/1403356586 conn(0x7ff6a00fc7a0 msgr2=0x7ff6a00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 shutdown_connections 2024-09-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:13.331+0000 7ff6a70f5640 1 -- 172.21.15.28:0/1403356586 wait complete. 2024-09-15T21:55:13.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:13 smithi028 ceph-mon[30969]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:13.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:13 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1403356586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:13 smithi174 ceph-mon[35244]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:13 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1403356586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:13.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:13 smithi203 ceph-mon[35546]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:13.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:13 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1403356586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:14.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:55:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 -- 172.21.15.28:0/1525404018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec106f10 msgr2=0x7f7aec10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 --2- 172.21.15.28:0/1525404018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec106f10 0x7f7aec10ea90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009ca0 tx=0x7f7ae402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:55:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 -- 172.21.15.28:0/1525404018 shutdown_connections 2024-09-15T21:55:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 --2- 172.21.15.28:0/1525404018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec106f10 0x7f7aec10ea90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 --2- 172.21.15.28:0/1525404018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aec102790 0x7f7aec1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 --2- 172.21.15.28:0/1525404018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec101de0 0x7f7aec1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 -- 172.21.15.28:0/1525404018 >> 172.21.15.28:0/1525404018 conn(0x7f7aec0fd8c0 msgr2=0x7f7aec0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 -- 172.21.15.28:0/1525404018 shutdown_connections 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.808+0000 7f7af23f5640 1 -- 172.21.15.28:0/1525404018 wait complete. 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 Processor -- start 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 -- start start 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec101de0 0x7f7aec0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aec102790 0x7f7aec074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec106f10 0x7f7aec074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec111f90 con 0x7f7aec101de0 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec111e10 con 0x7f7aec102790 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.809+0000 7f7af23f5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec112110 con 0x7f7aec106f10 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec101de0 0x7f7aec0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:14.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec101de0 0x7f7aec0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33010/0 (socket says 172.21.15.28:33010) 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 -- 172.21.15.28:0/2072013199 learned_addr learned my addr 172.21.15.28:0/2072013199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7af0992640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec106f10 0x7f7aec074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aeb7fe640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aec102790 0x7f7aec074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 -- 172.21.15.28:0/2072013199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec106f10 msgr2=0x7f7aec074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec106f10 0x7f7aec074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 -- 172.21.15.28:0/2072013199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aec102790 msgr2=0x7f7aec074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aec102790 0x7f7aec074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 -- 172.21.15.28:0/2072013199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec1a68d0 con 0x7f7aec101de0 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aeb7fe640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aec102790 0x7f7aec074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7aebfff640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec101de0 0x7f7aec0744f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00c920 tx=0x7f7adc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7ae97fa640 1 -- 172.21.15.28:0/2072013199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc007c70 con 0x7f7aec101de0 2024-09-15T21:55:14.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7ae97fa640 1 -- 172.21.15.28:0/2072013199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7adc007e10 con 0x7f7aec101de0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7ae97fa640 1 -- 172.21.15.28:0/2072013199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc018890 con 0x7f7aec101de0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.810+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aec1a6bc0 con 0x7f7aec101de0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.811+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aec1a7150 con 0x7f7aec101de0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.812+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aec1021c0 con 0x7f7aec101de0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.813+0000 7f7ae97fa640 1 -- 172.21.15.28:0/2072013199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7adc018a70 con 0x7f7aec101de0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.813+0000 7f7ae97fa640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ad40a32c0 0x7f7ad40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.813+0000 7f7aeb7fe640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ad40a32c0 0x7f7ad40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.814+0000 7f7ae97fa640 1 -- 172.21.15.28:0/2072013199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f7adc0cfcb0 con 0x7f7aec101de0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.814+0000 7f7aeb7fe640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ad40a32c0 0x7f7ad40a5780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0006fd0 tx=0x7f7ae0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:14.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:14.817+0000 7f7ae97fa640 1 -- 172.21.15.28:0/2072013199 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7adc014030 con 0x7f7aec101de0 2024-09-15T21:55:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.028+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7aec075a20 con 0x7f7aec101de0 2024-09-15T21:55:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.029+0000 7f7ae97fa640 1 -- 172.21.15.28:0/2072013199 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11851 (secure 0 0 0) 0x7f7adc09c790 con 0x7f7aec101de0 2024-09-15T21:55:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:15.032 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"btime":"2024-09-15T21:55:08:461866+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:08.111236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24485,"qdb_cluster":[24485]},"id":1}]} 2024-09-15T21:55:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-15T21:55:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ad40a32c0 msgr2=0x7f7ad40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ad40a32c0 0x7f7ad40a5780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0006fd0 tx=0x7f7ae0008040 comp rx=0 tx=0).stop 2024-09-15T21:55:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec101de0 msgr2=0x7f7aec0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec101de0 0x7f7aec0744f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00c920 tx=0x7f7adc00cde0 comp rx=0 tx=0).stop 2024-09-15T21:55:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 shutdown_connections 2024-09-15T21:55:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ad40a32c0 0x7f7ad40a5780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec106f10 0x7f7aec074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aec102790 0x7f7aec074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 --2- 172.21.15.28:0/2072013199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aec101de0 0x7f7aec0744f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 >> 172.21.15.28:0/2072013199 conn(0x7f7aec0fd8c0 msgr2=0x7f7aec0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 shutdown_connections 2024-09-15T21:55:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:15.031+0000 7f7af23f5640 1 -- 172.21.15.28:0/2072013199 wait complete. 2024-09-15T21:55:15.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:55:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:15 smithi174 ceph-mon[35244]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:15 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2072013199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:15.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:15 smithi028 ceph-mon[30969]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:15.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:15 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2072013199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:15.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:15 smithi203 ceph-mon[35546]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:15.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:15 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2072013199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 -- 172.21.15.28:0/770530378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074f40 msgr2=0x7f2ee010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 --2- 172.21.15.28:0/770530378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074f40 0x7f2ee010de80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009900 tx=0x7f2ed002a480 comp rx=0 tx=0).stop 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 -- 172.21.15.28:0/770530378 shutdown_connections 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 --2- 172.21.15.28:0/770530378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ee010e3c0 0x7f2ee0110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 --2- 172.21.15.28:0/770530378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074f40 0x7f2ee010de80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 --2- 172.21.15.28:0/770530378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee0074620 0x7f2ee0074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 -- 172.21.15.28:0/770530378 >> 172.21.15.28:0/770530378 conn(0x7f2ee00fd8c0 msgr2=0x7f2ee00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 -- 172.21.15.28:0/770530378 shutdown_connections 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.458+0000 7f2ee7659640 1 -- 172.21.15.28:0/770530378 wait complete. 2024-09-15T21:55:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 Processor -- start 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 -- start start 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074620 0x7f2ee019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee0074f40 0x7f2ee019ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ee010e3c0 0x7f2ee019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee0114140 con 0x7f2ee0074620 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee0113fc0 con 0x7f2ee010e3c0 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee7659640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee01142c0 con 0x7f2ee0074f40 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074620 0x7f2ee019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074620 0x7f2ee019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33034/0 (socket says 172.21.15.28:33034) 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 -- 172.21.15.28:0/1614315281 learned_addr learned my addr 172.21.15.28:0/1614315281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee5bf6640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ee010e3c0 0x7f2ee019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee4bf4640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee0074f40 0x7f2ee019ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 -- 172.21.15.28:0/1614315281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee0074f40 msgr2=0x7f2ee019ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee0074f40 0x7f2ee019ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 -- 172.21.15.28:0/1614315281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ee010e3c0 msgr2=0x7f2ee019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ee010e3c0 0x7f2ee019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee53f5640 1 -- 172.21.15.28:0/1614315281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee019f6a0 con 0x7f2ee0074620 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.459+0000 7f2ee5bf6640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ee010e3c0 0x7f2ee019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.460+0000 7f2ee53f5640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074620 0x7f2ee019a6f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00c890 tx=0x7f2edc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.460+0000 7f2ed67fc640 1 -- 172.21.15.28:0/1614315281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc004220 con 0x7f2ee0074620 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.460+0000 7f2ed67fc640 1 -- 172.21.15.28:0/1614315281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2edc0043c0 con 0x7f2ee0074620 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.460+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee019f990 con 0x7f2ee0074620 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.460+0000 7f2ed67fc640 1 -- 172.21.15.28:0/1614315281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc0049e0 con 0x7f2ee0074620 2024-09-15T21:55:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.460+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee01a7270 con 0x7f2ee0074620 2024-09-15T21:55:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.461+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee0074a00 con 0x7f2ee0074620 2024-09-15T21:55:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.462+0000 7f2ed67fc640 1 -- 172.21.15.28:0/1614315281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2edc0079b0 con 0x7f2ee0074620 2024-09-15T21:55:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.463+0000 7f2ed67fc640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f2ecc0a3240 0x7f2ecc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.463+0000 7f2ee4bf4640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f2ecc0a3240 0x7f2ecc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.463+0000 7f2ed67fc640 1 -- 172.21.15.28:0/1614315281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f2edc0cfcc0 con 0x7f2ee0074620 2024-09-15T21:55:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.463+0000 7f2ee4bf4640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f2ecc0a3240 0x7f2ecc0a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00fff50 tx=0x7f2ed0031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.467+0000 7f2ed67fc640 1 -- 172.21.15.28:0/1614315281 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2edc0d6080 con 0x7f2ee0074620 2024-09-15T21:55:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.643+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ee01095a0 con 0x7f2ee0074620 2024-09-15T21:55:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.645+0000 7f2ed67fc640 1 -- 172.21.15.28:0/1614315281 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f2edc014030 con 0x7f2ee0074620 2024-09-15T21:55:16.647 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:16.647 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:08.111056+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f2ecc0a3240 msgr2=0x7f2ecc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f2ecc0a3240 0x7f2ecc0a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00fff50 tx=0x7f2ed0031490 comp rx=0 tx=0).stop 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074620 msgr2=0x7f2ee019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074620 0x7f2ee019a6f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2edc00c890 tx=0x7f2edc00cd50 comp rx=0 tx=0).stop 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 shutdown_connections 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f2ecc0a3240 0x7f2ecc0a5700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ee010e3c0 0x7f2ee019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee0074f40 0x7f2ee019ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 --2- 172.21.15.28:0/1614315281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ee0074620 0x7f2ee019a6f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 >> 172.21.15.28:0/1614315281 conn(0x7f2ee00fd8c0 msgr2=0x7f2ee00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 shutdown_connections 2024-09-15T21:55:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:16.647+0000 7f2ee7659640 1 -- 172.21.15.28:0/1614315281 wait complete. 2024-09-15T21:55:17.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-15T21:54:59.048835+0000', 'modified': '2024-09-15T21:55:08.111236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24485}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.174:6839/3735159621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3735159621}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3735159621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24485, 'qdb_cluster': [24485]} 2024-09-15T21:55:17.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-15T21:55:17.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T21:55:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:17.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:17 smithi028 ceph-mon[30969]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:17.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:17 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1614315281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:17 smithi174 ceph-mon[35244]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:17 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1614315281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:17.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:17 smithi203 ceph-mon[35546]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:17.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:17 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1614315281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:18.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.066+0000 7fecc7af7640 1 -- 172.21.15.28:0/2647650118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 msgr2=0x7fecc0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.066+0000 7fecc7af7640 1 --2- 172.21.15.28:0/2647650118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0101200 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009900 tx=0x7fecb402a480 comp rx=0 tx=0).stop 2024-09-15T21:55:18.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 -- 172.21.15.28:0/2647650118 shutdown_connections 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 --2- 172.21.15.28:0/2647650118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0102ef0 0x7fecc010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 --2- 172.21.15.28:0/2647650118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecc0102570 0x7fecc01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 --2- 172.21.15.28:0/2647650118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0101200 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 -- 172.21.15.28:0/2647650118 >> 172.21.15.28:0/2647650118 conn(0x7fecc00fc7d0 msgr2=0x7fecc00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 -- 172.21.15.28:0/2647650118 shutdown_connections 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 -- 172.21.15.28:0/2647650118 wait complete. 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 Processor -- start 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.067+0000 7fecc7af7640 1 -- start start 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc7af7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc7af7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0102570 0x7fecc01999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc7af7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecc0102ef0 0x7fecc01a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc7af7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecc0110fa0 con 0x7fecc0100e20 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc7af7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fecc0110e20 con 0x7fecc0102570 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc7af7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fecc0111120 con 0x7fecc0102ef0 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33046/0 (socket says 172.21.15.28:33046) 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 -- 172.21.15.28:0/867248132 learned_addr learned my addr 172.21.15.28:0/867248132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc6094640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecc0102ef0 0x7fecc01a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5092640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0102570 0x7fecc01999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 -- 172.21.15.28:0/867248132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecc0102ef0 msgr2=0x7fecc01a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecc0102ef0 0x7fecc01a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 -- 172.21.15.28:0/867248132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0102570 msgr2=0x7fecc01999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0102570 0x7fecc01999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 -- 172.21.15.28:0/867248132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc01a4a30 con 0x7fecc0100e20 2024-09-15T21:55:18.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.068+0000 7fecc5893640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0199470 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fecb4006010 tx=0x7fecb4003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:18.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.069+0000 7fecb2ffd640 1 -- 172.21.15.28:0/867248132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb403f070 con 0x7fecc0100e20 2024-09-15T21:55:18.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.069+0000 7fecb2ffd640 1 -- 172.21.15.28:0/867248132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb4031e30 con 0x7fecc0100e20 2024-09-15T21:55:18.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.069+0000 7fecb2ffd640 1 -- 172.21.15.28:0/867248132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4035030 con 0x7fecc0100e20 2024-09-15T21:55:18.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.069+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc01a4c00 con 0x7fecc0100e20 2024-09-15T21:55:18.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.069+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc01a5000 con 0x7fecc0100e20 2024-09-15T21:55:18.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.071+0000 7fecb2ffd640 1 -- 172.21.15.28:0/867248132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fecb4042070 con 0x7fecc0100e20 2024-09-15T21:55:18.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.071+0000 7fecc9ed0740 1 Processor -- start 2024-09-15T21:55:18.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc9ed0740 1 -- start start 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x565211175c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111761d0 0x565211184a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565211184fc0 0x5652111873b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652110dc090 con 0x5652111761d0 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565211083a70 con 0x5652111758b0 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5652110d8050 con 0x565211184fc0 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc5893640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111761d0 0x565211184a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc5893640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111761d0 0x565211184a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33048/0 (socket says 172.21.15.28:33048) 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc5893640 1 -- 172.21.15.28:0/4217300409 learned_addr learned my addr 172.21.15.28:0/4217300409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc6094640 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x565211175c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc5092640 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565211184fc0 0x5652111873b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecb2ffd640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fecac0a32c0 0x7fecac0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc6094640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fecac0a32c0 0x7fecac0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc5893640 1 -- 172.21.15.28:0/4217300409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565211184fc0 msgr2=0x5652111873b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc5893640 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565211184fc0 0x5652111873b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.072+0000 7fecc5893640 1 -- 172.21.15.28:0/4217300409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 msgr2=0x565211175c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecc5893640 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x565211175c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecb2ffd640 1 -- 172.21.15.28:0/867248132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fecb40f0370 con 0x7fecc0100e20 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecc5893640 1 -- 172.21.15.28:0/4217300409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565211090130 con 0x5652111761d0 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecc5893640 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111761d0 0x565211184a80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fecb40b90c0 tx=0x7fecb402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e229a33caf1725 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecc6094640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fecac0a32c0 0x7fecac0a5780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fecb800afd0 tx=0x7fecb8004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecabfff640 1 -- 172.21.15.28:0/4217300409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4049070 con 0x5652111761d0 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecabfff640 1 -- 172.21.15.28:0/4217300409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb403be50 con 0x5652111761d0 2024-09-15T21:55:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.073+0000 7fecabfff640 1 -- 172.21.15.28:0/4217300409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb40bcdf0 con 0x5652111761d0 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 -- 172.21.15.28:0/4217300409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111761d0 msgr2=0x565211184a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111761d0 0x565211184a80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fecb40b90c0 tx=0x7fecb402a480 comp rx=0 tx=0).stop 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 -- 172.21.15.28:0/4217300409 shutdown_connections 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x565211184fc0 0x5652111873b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111761d0 0x565211184a80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/4217300409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x565211175c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 -- 172.21.15.28:0/4217300409 >> 172.21.15.28:0/4217300409 conn(0x565211171f60 msgr2=0x565211172340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 -- 172.21.15.28:0/4217300409 shutdown_connections 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 -- 172.21.15.28:0/4217300409 wait complete. 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 -- start start 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.074+0000 7fecc9ed0740 1 Processor -- start 2024-09-15T21:55:18.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x5652111847c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x565211181b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565211187950 con 0x565211184fc0 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565211096f20 con 0x5652111758b0 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x565211187ad0 con 0x5652111761d0 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc9ed0740 10 client.0 _ll_register_callbacks cb 0x565211060030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33058/0 (socket says 172.21.15.28:33058) 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 -- 172.21.15.28:0/2681418509 learned_addr learned my addr 172.21.15.28:0/2681418509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc5893640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x5652111847c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc5092640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x565211181b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 -- 172.21.15.28:0/2681418509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 msgr2=0x565211181b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x565211181b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 -- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 msgr2=0x5652111847c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x5652111847c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 -- 172.21.15.28:0/2681418509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565210e4d2b0 con 0x565211184fc0 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc5893640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x5652111847c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7fecc6094640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fecb800f690 tx=0x7fecb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399b02c27939121c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.075+0000 7feca97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7feca97fa640 1 -- 172.21.15.28:0/2681418509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8019070 con 0x565211184fc0 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb403be50 con 0x565211184fc0 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7feca97fa640 1 -- 172.21.15.28:0/2681418509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fecb801f840 con 0x565211184fc0 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7feca97fa640 1 -- 172.21.15.28:0/2681418509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fecb801fa80 con 0x565211184fc0 2024-09-15T21:55:18.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 10 client.14733 fetch_fsmap learned FSMap version 17 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x565211193210 con 0x565211184fc0 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7feca97fa640 1 -- 172.21.15.28:0/2681418509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb403be50 con 0x565211184fc0 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7feca97fa640 1 -- 172.21.15.28:0/2681418509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7fecb801d420 con 0x565211184fc0 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 10 client.14733 fetch_fsmap finished waiting for FSMap version 17 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 10 client.14733 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 20 client.14733 populate_metadata read hostname 'smithi028' 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5652111958c0 0x565211196d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 4 client.14733 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.076+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652110e9600 con 0x5652111958c0 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.077+0000 7fecc5893640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5652111958c0 0x565211196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.077+0000 7fecc5893640 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5652111958c0 0x565211196d30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81e2dfdda4255f4d server_cookie=875575e05a08091c in_seq=0 out_seq=0 2024-09-15T21:55:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.077+0000 7feca97fa640 10 client.14733 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.078+0000 7feca97fa640 1 -- 172.21.15.28:0/2681418509 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5652110e9600 con 0x5652111958c0 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.078+0000 7feca97fa640 10 client.14733 handle_command_reply: tid=0 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.078+0000 7fecc9ed0740 1 client.14733 shutdown 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.078+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 msgr2=0x565211182050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.078+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fecb800f690 tx=0x7fecb8004a40 comp rx=0 tx=0).stop 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 shutdown_connections 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5652111958c0 0x565211196d30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182050 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x565211181b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2681418509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111758b0 0x5652111847c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7feca97fa640 0 client.14733 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 >> 172.21.15.28:0/2681418509 conn(0x565211171f60 msgr2=0x565211177cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:18.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 shutdown_connections 2024-09-15T21:55:18.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2681418509 wait complete. 2024-09-15T21:55:18.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.079+0000 7fecc9ed0740 20 client.14733 trim_cache size 0 max 16384 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.096+0000 7fecc9ed0740 1 Processor -- start 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.096+0000 7fecc9ed0740 1 -- start start 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111758b0 0x5652111813d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111761d0 0x565211181ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565211195070 con 0x565211184fc0 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565211195740 con 0x5652111761d0 2024-09-15T21:55:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x565211195e10 con 0x5652111758b0 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33064/0 (socket says 172.21.15.28:33064) 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 -- 172.21.15.28:0/2835222462 learned_addr learned my addr 172.21.15.28:0/2835222462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 -- 172.21.15.28:0/2835222462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111758b0 msgr2=0x5652111813d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 --2- 172.21.15.28:0/2835222462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111758b0 0x5652111813d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 -- 172.21.15.28:0/2835222462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111761d0 msgr2=0x565211181ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 --2- 172.21.15.28:0/2835222462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111761d0 0x565211181ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 -- 172.21.15.28:0/2835222462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652111730d0 con 0x565211184fc0 2024-09-15T21:55:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecc6094640 1 --2- 172.21.15.28:0/2835222462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fecb80135f0 tx=0x7fecb8005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a6a20cf5198c9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecab7fe640 1 -- 172.21.15.28:0/2835222462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8019040 con 0x565211184fc0 2024-09-15T21:55:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecab7fe640 1 -- 172.21.15.28:0/2835222462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8024070 con 0x565211184fc0 2024-09-15T21:55:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.097+0000 7fecab7fe640 1 -- 172.21.15.28:0/2835222462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb801fe80 con 0x565211184fc0 2024-09-15T21:55:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2835222462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 msgr2=0x565211182010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2835222462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fecb80135f0 tx=0x7fecb8005f60 comp rx=0 tx=0).stop 2024-09-15T21:55:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2835222462 shutdown_connections 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2835222462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565211184fc0 0x565211182010 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2835222462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652111761d0 0x565211181ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/2835222462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111758b0 0x5652111813d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2835222462 >> 172.21.15.28:0/2835222462 conn(0x565211171f60 msgr2=0x565211187460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2835222462 shutdown_connections 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 -- 172.21.15.28:0/2835222462 wait complete. 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 -- start start 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.098+0000 7fecc9ed0740 1 Processor -- start 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111758b0 0x56521117f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x56521117dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc9ed0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565211184fc0 0x56521117e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565211196bb0 con 0x5652111758b0 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652111964e0 con 0x565211184fc0 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc9ed0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x565211197280 con 0x5652111761d0 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc9ed0740 10 client.0 _ll_register_callbacks cb 0x7fecb8013e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111758b0 0x56521117f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111758b0 0x56521117f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33074/0 (socket says 172.21.15.28:33074) 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 -- 172.21.15.28:0/1202645656 learned_addr learned my addr 172.21.15.28:0/1202645656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc6094640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565211184fc0 0x56521117e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5092640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x56521117dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 -- 172.21.15.28:0/1202645656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 msgr2=0x56521117dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x56521117dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 -- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565211184fc0 msgr2=0x56521117e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:55:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565211184fc0 0x56521117e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 -- 172.21.15.28:0/1202645656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56521119df20 con 0x5652111758b0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5092640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x56521117dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7fecc5893640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111758b0 0x56521117f800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fecb4003f90 tx=0x7fecb40f0180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6ae2ad7ee6a934 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.099+0000 7feca97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7feca97fa640 1 -- 172.21.15.28:0/1202645656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4049040 con 0x5652111758b0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7feca97fa640 1 -- 172.21.15.28:0/1202645656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fecb40ba9f0 con 0x5652111758b0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb8024070 con 0x5652111758b0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7feca97fa640 1 -- 172.21.15.28:0/1202645656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fecb40bc230 con 0x5652111758b0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 10 client.14739 fetch_fsmap learned FSMap version 17 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56521119f1c0 con 0x5652111758b0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7feca97fa640 1 -- 172.21.15.28:0/1202645656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8024070 con 0x5652111758b0 2024-09-15T21:55:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7feca97fa640 1 -- 172.21.15.28:0/1202645656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7fecb40f3aa0 con 0x5652111758b0 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 10 client.14739 fetch_fsmap finished waiting for FSMap version 17 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 10 client.14739 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 20 client.14739 populate_metadata read hostname 'smithi028' 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x565211203ba0 0x5652112028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 4 client.14739 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.100+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "status"}) -- 0x5652110e3b10 con 0x565211203ba0 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc5092640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x565211203ba0 0x5652112028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc5092640 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x565211203ba0 0x5652112028b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd7b2eebeea1b77b server_cookie=9e88932cb6cd002b in_seq=0 out_seq=0 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7feca97fa640 10 client.14739 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7feca97fa640 1 -- 172.21.15.28:0/1202645656 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+320 (crc 0 0 0) 0x5652110e3b10 con 0x565211203ba0 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7feca97fa640 10 client.14739 handle_command_reply: tid=0 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 client.14739 shutdown 2024-09-15T21:55:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111758b0 msgr2=0x56521117f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111758b0 0x56521117f800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fecb4003f90 tx=0x7fecb40f0180 comp rx=0 tx=0).stop 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 shutdown_connections 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x565211203ba0 0x5652112028b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565211184fc0 0x56521117e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652111761d0 0x56521117dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 --2- 172.21.15.28:0/1202645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652111758b0 0x56521117f800 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7feca97fa640 0 client.14739 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.101+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 >> 172.21.15.28:0/1202645656 conn(0x565211171f60 msgr2=0x565211172340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 shutdown_connections 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc9ed0740 1 -- 172.21.15.28:0/1202645656 wait complete. 2024-09-15T21:55:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc9ed0740 20 client.14739 trim_cache size 0 max 16384 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24485, 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 18.677039127, 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-09-15T21:55:18.111 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:55:18.112 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 29.825781229 2024-09-15T21:55:18.112 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:55:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fecac0a32c0 msgr2=0x7fecac0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fecac0a32c0 0x7fecac0a5780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fecb800afd0 tx=0x7fecb8004ab0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 msgr2=0x7fecc0199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0199470 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fecb4006010 tx=0x7fecb4003c80 comp rx=0 tx=0).stop 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 shutdown_connections 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fecac0a32c0 0x7fecac0a5780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecc0102ef0 0x7fecc01a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecc0102570 0x7fecc01999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 --2- 172.21.15.28:0/867248132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0100e20 0x7fecc0199470 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 >> 172.21.15.28:0/867248132 conn(0x7fecc00fc7d0 msgr2=0x7fecc00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 shutdown_connections 2024-09-15T21:55:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:18.102+0000 7fecc7af7640 1 -- 172.21.15.28:0/867248132 wait complete. 2024-09-15T21:55:18.861 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-15T21:55:18.861 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:18.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-15T21:55:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:19.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 -- 172.21.15.28:0/70030716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2098103660 msgr2=0x7f209810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:19.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 --2- 172.21.15.28:0/70030716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2098103660 0x7f209810bb50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f208c009900 tx=0x7f208c02a480 comp rx=0 tx=0).stop 2024-09-15T21:55:19.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 -- 172.21.15.28:0/70030716 shutdown_connections 2024-09-15T21:55:19.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 --2- 172.21.15.28:0/70030716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f209810c170 0x7f209810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 --2- 172.21.15.28:0/70030716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2098103660 0x7f209810bb50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 --2- 172.21.15.28:0/70030716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2098102d40 0x7f2098103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 -- 172.21.15.28:0/70030716 >> 172.21.15.28:0/70030716 conn(0x7f20980fc7a0 msgr2=0x7f20980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 -- 172.21.15.28:0/70030716 shutdown_connections 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.609+0000 7f209dc11640 1 -- 172.21.15.28:0/70030716 wait complete. 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 Processor -- start 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 -- start start 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2098102d40 0x7f20981a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2098103660 0x7f20981a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209810c170 0x7f209819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2098112030 con 0x7f209810c170 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2098111eb0 con 0x7f2098102d40 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.610+0000 7f209dc11640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20981121b0 con 0x7f2098103660 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209810c170 0x7f209819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209810c170 0x7f209819a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33084/0 (socket says 172.21.15.28:33084) 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 -- 172.21.15.28:0/318373822 learned_addr learned my addr 172.21.15.28:0/318373822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f20977fe640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2098102d40 0x7f20981a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2096ffd640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2098103660 0x7f20981a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 -- 172.21.15.28:0/318373822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2098103660 msgr2=0x7f20981a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2098103660 0x7f20981a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 -- 172.21.15.28:0/318373822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2098102d40 msgr2=0x7f20981a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2098102d40 0x7f20981a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 -- 172.21.15.28:0/318373822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209819ad50 con 0x7f209810c170 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2097fff640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209810c170 0x7f209819a620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f209000b970 tx=0x7f209000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2094ff9640 1 -- 172.21.15.28:0/318373822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090019070 con 0x7f209810c170 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2094ff9640 1 -- 172.21.15.28:0/318373822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2090014440 con 0x7f209810c170 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209819b040 con 0x7f209810c170 2024-09-15T21:55:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f2094ff9640 1 -- 172.21.15.28:0/318373822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20900075d0 con 0x7f209810c170 2024-09-15T21:55:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.611+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20981a7330 con 0x7f209810c170 2024-09-15T21:55:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.612+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20981031a0 con 0x7f209810c170 2024-09-15T21:55:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.614+0000 7f2094ff9640 1 -- 172.21.15.28:0/318373822 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f209001c070 con 0x7f209810c170 2024-09-15T21:55:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.614+0000 7f2094ff9640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20800a32c0 0x7f20800a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.614+0000 7f20977fe640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20800a32c0 0x7f20800a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.615+0000 7f2094ff9640 1 -- 172.21.15.28:0/318373822 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f20900cf5f0 con 0x7f209810c170 2024-09-15T21:55:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.615+0000 7f20977fe640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20800a32c0 0x7f20800a5780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2088009990 tx=0x7f2088009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.617+0000 7f2094ff9640 1 -- 172.21.15.28:0/318373822 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f209009c0d0 con 0x7f209810c170 2024-09-15T21:55:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:19 smithi174 ceph-mon[35244]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.831+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f20981a7560 con 0x7f209810c170 2024-09-15T21:55:19.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:19 smithi028 ceph-mon[30969]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.832+0000 7f2094ff9640 1 -- 172.21.15.28:0/318373822 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f209009b9d0 con 0x7f209810c170 2024-09-15T21:55:19.834 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-15T21:55:19.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.834+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20800a32c0 msgr2=0x7f20800a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:19.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.834+0000 7f209dc11640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20800a32c0 0x7f20800a5780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2088009990 tx=0x7f2088009110 comp rx=0 tx=0).stop 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.834+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209810c170 msgr2=0x7f209819a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.834+0000 7f209dc11640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209810c170 0x7f209819a620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f209000b970 tx=0x7f209000bd70 comp rx=0 tx=0).stop 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 shutdown_connections 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f20800a32c0 0x7f20800a5780 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209810c170 0x7f209819a620 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2098103660 0x7f20981a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 --2- 172.21.15.28:0/318373822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2098102d40 0x7f20981a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 >> 172.21.15.28:0/318373822 conn(0x7f20980fc7a0 msgr2=0x7f2098100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 shutdown_connections 2024-09-15T21:55:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:19.835+0000 7f209dc11640 1 -- 172.21.15.28:0/318373822 wait complete. 2024-09-15T21:55:19.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:19 smithi203 ceph-mon[35546]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:20.628 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-15T21:55:20.628 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-15T21:55:20.628 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-15T21:55:20.628 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-15T21:55:20.628 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:20.628 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-15T21:55:20.628 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-15T21:55:20.629 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-15T21:55:20.629 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-15T21:55:20.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:20.629 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-15T21:55:20.646 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42278sec preferred_lft 42278sec 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-15T21:55:20.647 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-15T21:55:20.648 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-15T21:55:20.648 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:55:20.648 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:55:20.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-09-15T21:55:20.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-09-15T21:55:20.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-09-15T21:55:20.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-15T21:55:20.649 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:55:20.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:20 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/318373822' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T21:55:20.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:20 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/318373822' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T21:55:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:55:20.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:20.867 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-15T21:55:20.894 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-15T21:55:20.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:20.896 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-15T21:55:20.965 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-15T21:55:20.966 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-15T21:55:20.966 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-15T21:55:20.966 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-15T21:55:20.966 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-15T21:55:20.966 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:55:20.966 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:55:20.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-15T21:55:20.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-15T21:55:20.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-15T21:55:20.967 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:55:20.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:20 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/318373822' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T21:55:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:55:21.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:55:21.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:21.266 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-15T21:55:21.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:21.284 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-15T21:55:21.340 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:55:21.340 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:55:21.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:55:21.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-15T21:55:21.340 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:55:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:55:21.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:55:21.462 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-15T21:55:21.462 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:55:21.462 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:55:21.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-15T21:55:21.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-15T21:55:21.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-15T21:55:21.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-15T21:55:21.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-15T21:55:21.463 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:55:21.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:55:21.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:55:21.596 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:55:21.596 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:55:21.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-15T21:55:21.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-09-15T21:55:21.597 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:55:21.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:55:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:55:21.676 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-15T21:55:21.676 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T21:55:21.677 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:21.693 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-15T21:55:21.694 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T21:55:21.694 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:21.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-09-15T21:55:21.759 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:21 smithi174 ceph-mon[35244]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:21.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:21 smithi028 ceph-mon[30969]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:21.818 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-09-15T21:55:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-09-15T21:55:21.877 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/fc08b9594c0c89d0e0cc9fc65024ac76748a09930a592ce3d37ad875d5caec1a/merged 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c3bd252309c5990408d8612547e263f0d6ac7cd942cc9aad0b60c39efe999b77/merged 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4756c0be7a64994c8e58ab5ad2aaa39f7de6c373eb306dc4ad03fff6a5eaf290/merged 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/cf1865aa6f6e319a26c6301abb93f0c86ebfd6392d7d99499307f1a231f60fbf/merged 2024-09-15T21:55:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e77e91bf682f2d7e922a591827086c5631bdac4bf2fcb2fb02a62eeab64d94a5/merged 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/0495a4aba722c4c0c09aff37503d2d2c58abce96fad50ae3fedbbdbe72458b7d/merged 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c0b9bcb9f4035a84248f532d0d71ca9c4e53ce960b2ff2bcd1fe8610cc98c066/merged 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/59a05c690ee9a9e646b49656e5dd9d867984396c42405241848fe0aea469b145/merged 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c0fe7e42212075be700aecd7237a90998b00da8489ac0881ae8068df04ab8471/merged 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/57bdb376d03ff777b4de5bb267e52f750dae8c8bfefc1cffc4799d8134b72ef2/merged 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:55:21.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:21.879 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-09-15T21:55:21.932 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-15T21:55:21.933 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-15T21:55:21.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:21 smithi203 ceph-mon[35546]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:21.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-09-15T21:55:22.001 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-09-15T21:55:22.059 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-09-15T21:55:22.059 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-09-15T21:55:22.059 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-09-15T21:55:22.059 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-09-15T21:55:22.059 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-09-15T21:55:22.059 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-09-15T21:55:22.059 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-09-15T21:55:22.060 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/fc08b9594c0c89d0e0cc9fc65024ac76748a09930a592ce3d37ad875d5caec1a/merged 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c3bd252309c5990408d8612547e263f0d6ac7cd942cc9aad0b60c39efe999b77/merged 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4756c0be7a64994c8e58ab5ad2aaa39f7de6c373eb306dc4ad03fff6a5eaf290/merged 2024-09-15T21:55:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/cf1865aa6f6e319a26c6301abb93f0c86ebfd6392d7d99499307f1a231f60fbf/merged 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e77e91bf682f2d7e922a591827086c5631bdac4bf2fcb2fb02a62eeab64d94a5/merged 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/0495a4aba722c4c0c09aff37503d2d2c58abce96fad50ae3fedbbdbe72458b7d/merged 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c0b9bcb9f4035a84248f532d0d71ca9c4e53ce960b2ff2bcd1fe8610cc98c066/merged 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/59a05c690ee9a9e646b49656e5dd9d867984396c42405241848fe0aea469b145/merged 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c0fe7e42212075be700aecd7237a90998b00da8489ac0881ae8068df04ab8471/merged 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/57bdb376d03ff777b4de5bb267e52f750dae8c8bfefc1cffc4799d8134b72ef2/merged 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:55:22.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:22.063 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-09-15T21:55:22.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-09-15T21:55:22.124+0000 7f641b349480 -1 init, newargv = 0x562285483b50 newargc=15 2024-09-15T21:55:22.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[74798]: starting ceph client 2024-09-15T21:55:22.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[74798]: starting fuse 2024-09-15T21:55:23.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-09-15T21:55:23.250 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-09-15T21:55:23.309 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-09-15T21:55:23.309 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-09-15T21:55:23.309 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-09-15T21:55:23.309 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-09-15T21:55:23.309 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-09-15T21:55:23.310 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/fc08b9594c0c89d0e0cc9fc65024ac76748a09930a592ce3d37ad875d5caec1a/merged 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c3bd252309c5990408d8612547e263f0d6ac7cd942cc9aad0b60c39efe999b77/merged 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4756c0be7a64994c8e58ab5ad2aaa39f7de6c373eb306dc4ad03fff6a5eaf290/merged 2024-09-15T21:55:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/cf1865aa6f6e319a26c6301abb93f0c86ebfd6392d7d99499307f1a231f60fbf/merged 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e77e91bf682f2d7e922a591827086c5631bdac4bf2fcb2fb02a62eeab64d94a5/merged 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/0495a4aba722c4c0c09aff37503d2d2c58abce96fad50ae3fedbbdbe72458b7d/merged 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c0b9bcb9f4035a84248f532d0d71ca9c4e53ce960b2ff2bcd1fe8610cc98c066/merged 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/59a05c690ee9a9e646b49656e5dd9d867984396c42405241848fe0aea469b145/merged 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c0fe7e42212075be700aecd7237a90998b00da8489ac0881ae8068df04ab8471/merged 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/57bdb376d03ff777b4de5bb267e52f750dae8c8bfefc1cffc4799d8134b72ef2/merged 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:55:23.312 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:23.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:23.313 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-09-15T21:55:23.372 INFO:teuthology.orchestra.run.smithi028.stdout:120 2024-09-15T21:55:23.372 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-09-15T21:55:23.373 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-15T21:55:23.374 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-09-15T21:55:23.375 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-15T21:55:23.503 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.74798.asok 2024-09-15T21:55:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:55:23.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:23 smithi028 ceph-mon[30969]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:23.515 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.74798.asok 2024-09-15T21:55:23.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:23.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.74798.asok status 2024-09-15T21:55:23.654 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:55:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-09-15T21:55:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "8921c8ef947807b717db041d85f0761b7914d5fc", 2024-09-15T21:55:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T21:55:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-09-15T21:55:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "74798", 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 14751, 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 14751 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T21:55:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 294570641 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 294570641 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.14751 192.168.0.1:0/294570641", 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/294570641", 2024-09-15T21:55:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-09-15T21:55:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 17, 2024-09-15T21:55:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 88, 2024-09-15T21:55:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-09-15T21:55:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-09-15T21:55:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-09-15T21:55:23.657 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:55:23.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:23.663 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-09-15T21:55:23.663 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:23.680 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-09-15T21:55:23.680 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:23.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:23.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:23.747 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-15T21:55:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:23 smithi174 ceph-mon[35244]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T21:55:23.855 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-09-15T21:55:23.856 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-15T21:55:23.856 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:55:23.862 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:55:23.862 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T21:55:23.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T21:55:23.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:23 smithi203 ceph-mon[35546]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 -- 172.21.15.28:0/855313466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908100e20 msgr2=0x7ff908101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 --2- 172.21.15.28:0/855313466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908100e20 0x7ff908101200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009900 tx=0x7ff8f802a480 comp rx=0 tx=0).stop 2024-09-15T21:55:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 -- 172.21.15.28:0/855313466 shutdown_connections 2024-09-15T21:55:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 --2- 172.21.15.28:0/855313466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff908102ef0 0x7ff90810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 --2- 172.21.15.28:0/855313466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff908102570 0x7ff9081029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 --2- 172.21.15.28:0/855313466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908100e20 0x7ff908101200 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 -- 172.21.15.28:0/855313466 >> 172.21.15.28:0/855313466 conn(0x7ff9080fc7d0 msgr2=0x7ff9080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 -- 172.21.15.28:0/855313466 shutdown_connections 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.769+0000 7ff90eb67640 1 -- 172.21.15.28:0/855313466 wait complete. 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 Processor -- start 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 -- start start 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff908100e20 0x7ff908199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff908102570 0x7ff9081998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908102ef0 0x7ff9081a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff908110f80 con 0x7ff908102ef0 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff908110e00 con 0x7ff908102570 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90eb67640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff908111100 con 0x7ff908100e20 2024-09-15T21:55:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908102ef0 0x7ff9081a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908102ef0 0x7ff9081a43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49866/0 (socket says 172.21.15.28:49866) 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 -- 172.21.15.28:0/1595627723 learned_addr learned my addr 172.21.15.28:0/1595627723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff907fff640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff908102570 0x7ff9081998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90c903640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff908100e20 0x7ff908199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 -- 172.21.15.28:0/1595627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff908100e20 msgr2=0x7ff908199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff908100e20 0x7ff908199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 -- 172.21.15.28:0/1595627723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff908102570 msgr2=0x7ff9081998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff908102570 0x7ff9081998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90d104640 1 -- 172.21.15.28:0/1595627723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9081a4930 con 0x7ff908102ef0 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.770+0000 7ff90c903640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff908100e20 0x7ff908199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.771+0000 7ff90d104640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908102ef0 0x7ff9081a43f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff900009fd0 tx=0x7ff90000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.771+0000 7ff905ffb640 1 -- 172.21.15.28:0/1595627723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff900017070 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.771+0000 7ff905ffb640 1 -- 172.21.15.28:0/1595627723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff900003e50 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.771+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9081a4b00 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.771+0000 7ff905ffb640 1 -- 172.21.15.28:0/1595627723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9000106c0 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.771+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9081a5010 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.772+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff908101200 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.773+0000 7ff905ffb640 1 -- 172.21.15.28:0/1595627723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff900003970 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.774+0000 7ff905ffb640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8f00a32c0 0x7ff8f00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.774+0000 7ff90c903640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8f00a32c0 0x7ff8f00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.774+0000 7ff905ffb640 1 -- 172.21.15.28:0/1595627723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7ff9000cf7f0 con 0x7ff908102ef0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.775+0000 7ff90c903640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8f00a32c0 0x7ff8f00a5780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8000c00 tx=0x7ff8f8004100 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.779+0000 7ff905ffb640 1 -- 172.21.15.28:0/1595627723 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff90009c2d0 con 0x7ff908102ef0 2024-09-15T21:55:24.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.967+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff908109090 con 0x7ff908102ef0 2024-09-15T21:55:24.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.967+0000 7ff905ffb640 1 -- 172.21.15.28:0/1595627723 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+589 (secure 0 0 0) 0x7ff90009bbd0 con 0x7ff908102ef0 2024-09-15T21:55:24.969 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/671259623 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:24.969 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/711457183 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:24.969 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1395107553 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:24.969 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2625692755 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:24.970 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2625692755 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:24.970 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/123978199 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:24.970 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2709495038 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:24.970 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2627662465 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:24.970 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3052473616 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:24.970 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3052473616 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:24.970 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-09-15T21:55:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.969+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8f00a32c0 msgr2=0x7ff8f00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.969+0000 7ff90eb67640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8f00a32c0 0x7ff8f00a5780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8000c00 tx=0x7ff8f8004100 comp rx=0 tx=0).stop 2024-09-15T21:55:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.969+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908102ef0 msgr2=0x7ff9081a43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.969+0000 7ff90eb67640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908102ef0 0x7ff9081a43f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff900009fd0 tx=0x7ff90000b260 comp rx=0 tx=0).stop 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 shutdown_connections 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8f00a32c0 0x7ff8f00a5780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff908102ef0 0x7ff9081a43f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff908102570 0x7ff9081998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 --2- 172.21.15.28:0/1595627723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff908100e20 0x7ff908199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 >> 172.21.15.28:0/1595627723 conn(0x7ff9080fc7d0 msgr2=0x7ff9080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 shutdown_connections 2024-09-15T21:55:24.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:24.970+0000 7ff90eb67640 1 -- 172.21.15.28:0/1595627723 wait complete. 2024-09-15T21:55:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:25 smithi028 ceph-mon[30969]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:25 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1595627723' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T21:55:25.707 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:55:25.707 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T21:55:25.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T21:55:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:25 smithi174 ceph-mon[35244]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:25 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1595627723' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T21:55:25.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:25 smithi203 ceph-mon[35546]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:55:25.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:25 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1595627723' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T21:55:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/3056768180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8108290 msgr2=0x7f5ec810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/3056768180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8108290 0x7f5ec810ec30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0009cd0 tx=0x7f5ec002a8e0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/3056768180 shutdown_connections 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/3056768180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8108290 0x7f5ec810ec30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/3056768180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ec8103740 0x7f5ec8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/3056768180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ec8102d90 0x7f5ec8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/3056768180 >> 172.21.15.28:0/3056768180 conn(0x7f5ec80fc7d0 msgr2=0x7f5ec80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/3056768180 shutdown_connections 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/3056768180 wait complete. 2024-09-15T21:55:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.528+0000 7f5ecd7f8640 1 Processor -- start 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ecd7f8640 1 -- start start 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ecd7f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8102d90 0x7f5ec8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ecd7f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ec8103740 0x7f5ec8198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ecd7f8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ec8108290 0x7f5ec819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ecd7f8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec8112090 con 0x7f5ec8102d90 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ecd7f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec8111f10 con 0x7f5ec8103740 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ecd7f8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec8112210 con 0x7f5ec8108290 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8102d90 0x7f5ec8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8102d90 0x7f5ec8198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49888/0 (socket says 172.21.15.28:49888) 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 -- 172.21.15.28:0/4227046909 learned_addr learned my addr 172.21.15.28:0/4227046909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec77fe640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ec8108290 0x7f5ec819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec67fc640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ec8103740 0x7f5ec8198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 -- 172.21.15.28:0/4227046909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ec8108290 msgr2=0x7f5ec819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ec8108290 0x7f5ec819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 -- 172.21.15.28:0/4227046909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ec8103740 msgr2=0x7f5ec8198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ec8103740 0x7f5ec8198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 -- 172.21.15.28:0/4227046909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec819d520 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec67fc640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ec8103740 0x7f5ec8198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5ec6ffd640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8102d90 0x7f5ec8198570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800d770 tx=0x7f5eb800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.529+0000 7f5eaffff640 1 -- 172.21.15.28:0/4227046909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8017070 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.530+0000 7f5eaffff640 1 -- 172.21.15.28:0/4227046909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb8005500 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.530+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec819d810 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.530+0000 7f5eaffff640 1 -- 172.21.15.28:0/4227046909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8004e00 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.530+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec81a50f0 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.531+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ec8103170 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.532+0000 7f5eaffff640 1 -- 172.21.15.28:0/4227046909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5eb8004fa0 con 0x7f5ec8102d90 2024-09-15T21:55:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.532+0000 7f5eaffff640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5eb00a32c0 0x7f5eb00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.533+0000 7f5ec67fc640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5eb00a32c0 0x7f5eb00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.533+0000 7f5eaffff640 1 -- 172.21.15.28:0/4227046909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f5eb80cf520 con 0x7f5ec8102d90 2024-09-15T21:55:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.533+0000 7f5ec67fc640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5eb00a32c0 0x7f5eb00a5780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80fee20 tx=0x7f5ebc005cb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.537+0000 7f5eaffff640 1 -- 172.21.15.28:0/4227046909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5eb809c000 con 0x7f5ec8102d90 2024-09-15T21:55:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.722+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5ec810a2a0 con 0x7f5ec8102d90 2024-09-15T21:55:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.723+0000 7f5eaffff640 1 -- 172.21.15.28:0/4227046909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+589 (secure 0 0 0) 0x7f5eb80860d0 con 0x7f5ec8102d90 2024-09-15T21:55:26.724 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/671259623 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/711457183 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1395107553 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2625692755 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2625692755 2024-09-16T21:47:57.453457+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/123978199 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2709495038 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2627662465 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3052473616 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3052473616 2024-09-16T21:47:32.835958+0000 2024-09-15T21:55:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-09-15T21:55:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5eb00a32c0 msgr2=0x7f5eb00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5eb00a32c0 0x7f5eb00a5780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80fee20 tx=0x7f5ebc005cb0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8102d90 msgr2=0x7f5ec8198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8102d90 0x7f5ec8198570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800d770 tx=0x7f5eb800dc30 comp rx=0 tx=0).stop 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 shutdown_connections 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f5eb00a32c0 0x7f5eb00a5780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ec8108290 0x7f5ec819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ec8103740 0x7f5ec8198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 --2- 172.21.15.28:0/4227046909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec8102d90 0x7f5ec8198570 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 >> 172.21.15.28:0/4227046909 conn(0x7f5ec80fc7d0 msgr2=0x7f5ec8105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 shutdown_connections 2024-09-15T21:55:26.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:26.725+0000 7f5ecd7f8640 1 -- 172.21.15.28:0/4227046909 wait complete. 2024-09-15T21:55:27.463 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-09-15T21:55:27.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:27.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-15T21:55:27.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[74798]: fuse finished with error 0 and tester_r 0 2024-09-15T21:55:27.492 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:55:27.511 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:55:27.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:55:27.511 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-15T21:55:27.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:55:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:27 smithi174 ceph-mon[35244]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-15T21:55:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:27 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4227046909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T21:55:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:27.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:27 smithi028 ceph-mon[30969]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-15T21:55:27.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:27 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4227046909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T21:55:27.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:27 smithi203 ceph-mon[35546]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-15T21:55:27.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:27 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4227046909' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 -- 172.21.15.28:0/2674731896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc108290 msgr2=0x7ff8dc10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/2674731896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc108290 0x7ff8dc10ec30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009cd0 tx=0x7ff8d402a8e0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 -- 172.21.15.28:0/2674731896 shutdown_connections 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/2674731896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc108290 0x7ff8dc10ec30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/2674731896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8dc103740 0x7ff8dc107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/2674731896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8dc102d90 0x7ff8dc103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 -- 172.21.15.28:0/2674731896 >> 172.21.15.28:0/2674731896 conn(0x7ff8dc0fc7d0 msgr2=0x7ff8dc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.358+0000 7ff8e33d3640 1 -- 172.21.15.28:0/2674731896 shutdown_connections 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 -- 172.21.15.28:0/2674731896 wait complete. 2024-09-15T21:55:28.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 Processor -- start 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 -- start start 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8dc102d90 0x7ff8dc1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc103740 0x7ff8dc100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8dc108290 0x7ff8dc101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc1120d0 con 0x7ff8dc103740 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc111f50 con 0x7ff8dc108290 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.359+0000 7ff8e33d3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc112250 con 0x7ff8dc102d90 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc103740 0x7ff8dc100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc103740 0x7ff8dc100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49892/0 (socket says 172.21.15.28:49892) 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 -- 172.21.15.28:0/3711565306 learned_addr learned my addr 172.21.15.28:0/3711565306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e1970640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8dc108290 0x7ff8dc101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:28.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e116f640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8dc102d90 0x7ff8dc1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 -- 172.21.15.28:0/3711565306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8dc102d90 msgr2=0x7ff8dc1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8dc102d90 0x7ff8dc1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 -- 172.21.15.28:0/3711565306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8dc108290 msgr2=0x7ff8dc101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8dc108290 0x7ff8dc101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 -- 172.21.15.28:0/3711565306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc1a6a70 con 0x7ff8dc103740 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e116f640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8dc102d90 0x7ff8dc1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e1970640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8dc108290 0x7ff8dc101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e096e640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc103740 0x7ff8dc100f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00b450 tx=0x7ff8cc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8d27fc640 1 -- 172.21.15.28:0/3711565306 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc015020 con 0x7ff8dc103740 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8d27fc640 1 -- 172.21.15.28:0/3711565306 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8cc011d70 con 0x7ff8dc103740 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc1a6d60 con 0x7ff8dc103740 2024-09-15T21:55:28.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8d27fc640 1 -- 172.21.15.28:0/3711565306 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc00f340 con 0x7ff8dc103740 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.360+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc1a7240 con 0x7ff8dc103740 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.361+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc103170 con 0x7ff8dc103740 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.363+0000 7ff8d27fc640 1 -- 172.21.15.28:0/3711565306 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff8cc018020 con 0x7ff8dc103740 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.363+0000 7ff8d27fc640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8c80a32c0 0x7ff8c80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.363+0000 7ff8d27fc640 1 -- 172.21.15.28:0/3711565306 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7ff8cc0cfb40 con 0x7ff8dc103740 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.363+0000 7ff8e116f640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8c80a32c0 0x7ff8c80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.364+0000 7ff8e116f640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8c80a32c0 0x7ff8c80a5780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009b40 tx=0x7ff8d8005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:28.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.366+0000 7ff8d27fc640 1 -- 172.21.15.28:0/3711565306 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff8cc01f020 con 0x7ff8dc103740 2024-09-15T21:55:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.543+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8dc101f20 con 0x7ff8dc103740 2024-09-15T21:55:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.545+0000 7ff8d27fc640 1 -- 172.21.15.28:0/3711565306 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7ff8cc00d040 con 0x7ff8dc103740 2024-09-15T21:55:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:28.547 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:08.111056+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8c80a32c0 msgr2=0x7ff8c80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8c80a32c0 0x7ff8c80a5780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009b40 tx=0x7ff8d8005e50 comp rx=0 tx=0).stop 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc103740 msgr2=0x7ff8dc100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc103740 0x7ff8dc100f30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00b450 tx=0x7ff8cc00b910 comp rx=0 tx=0).stop 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 shutdown_connections 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff8c80a32c0 0x7ff8c80a5780 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8dc108290 0x7ff8dc101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc103740 0x7ff8dc100f30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 --2- 172.21.15.28:0/3711565306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8dc102d90 0x7ff8dc1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 >> 172.21.15.28:0/3711565306 conn(0x7ff8dc0fc7d0 msgr2=0x7ff8dc105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 shutdown_connections 2024-09-15T21:55:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:28.547+0000 7ff8e33d3640 1 -- 172.21.15.28:0/3711565306 wait complete. 2024-09-15T21:55:29.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:55:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:29.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:29 smithi028 ceph-mon[30969]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-15T21:55:29.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:29 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3711565306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:29.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:29 smithi174 ceph-mon[35244]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-15T21:55:29.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:29 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3711565306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:29.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:29 smithi203 ceph-mon[35546]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-15T21:55:29.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:29 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3711565306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 -- 172.21.15.28:0/1581695457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 msgr2=0x7fbf2410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 --2- 172.21.15.28:0/1581695457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2410ea90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009ca0 tx=0x7fbf1c02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:55:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 -- 172.21.15.28:0/1581695457 shutdown_connections 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 --2- 172.21.15.28:0/1581695457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2410ea90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 --2- 172.21.15.28:0/1581695457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf24102790 0x7fbf241067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 --2- 172.21.15.28:0/1581695457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf24101de0 0x7fbf241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 -- 172.21.15.28:0/1581695457 >> 172.21.15.28:0/1581695457 conn(0x7fbf240fd8c0 msgr2=0x7fbf240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 -- 172.21.15.28:0/1581695457 shutdown_connections 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 -- 172.21.15.28:0/1581695457 wait complete. 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.984+0000 7fbf29c59640 1 Processor -- start 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf29c59640 1 -- start start 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf29c59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf24101de0 0x7fbf24198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf29c59640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf24102790 0x7fbf24198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf29c59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf29c59640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf24111f70 con 0x7fbf24106f10 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf29c59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbf24111df0 con 0x7fbf24101de0 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf29c59640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbf241120f0 con 0x7fbf24102790 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2419cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49906/0 (socket says 172.21.15.28:49906) 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 -- 172.21.15.28:0/3424221026 learned_addr learned my addr 172.21.15.28:0/3424221026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf22ffd640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf24102790 0x7fbf24198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf237fe640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf24101de0 0x7fbf24198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 -- 172.21.15.28:0/3424221026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf24102790 msgr2=0x7fbf24198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf24102790 0x7fbf24198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 -- 172.21.15.28:0/3424221026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf24101de0 msgr2=0x7fbf24198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf24101de0 0x7fbf24198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.985+0000 7fbf23fff640 1 -- 172.21.15.28:0/3424221026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2419d2e0 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.986+0000 7fbf23fff640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2419cc00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c031a20 tx=0x7fbf1c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.986+0000 7fbf20ff9640 1 -- 172.21.15.28:0/3424221026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c003c30 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.986+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2419d570 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.986+0000 7fbf20ff9640 1 -- 172.21.15.28:0/3424221026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf1c003dd0 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.986+0000 7fbf20ff9640 1 -- 172.21.15.28:0/3424221026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c02fcf0 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.986+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf241a4eb0 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.987+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf241021c0 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.988+0000 7fbf20ff9640 1 -- 172.21.15.28:0/3424221026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbf1c004050 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.989+0000 7fbf20ff9640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbf0c0a3240 0x7fbf0c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.989+0000 7fbf20ff9640 1 -- 172.21.15.28:0/3424221026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fbf1c0f0f00 con 0x7fbf24106f10 2024-09-15T21:55:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.989+0000 7fbf237fe640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbf0c0a3240 0x7fbf0c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:29.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.989+0000 7fbf237fe640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbf0c0a3240 0x7fbf0c0a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf14005e10 tx=0x7fbf14005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:29.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:29.993+0000 7fbf20ff9640 1 -- 172.21.15.28:0/3424221026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbf1c0bda10 con 0x7fbf24106f10 2024-09-15T21:55:30.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.168+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf2410a150 con 0x7fbf24106f10 2024-09-15T21:55:30.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.170+0000 7fbf20ff9640 1 -- 172.21.15.28:0/3424221026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7fbf1c02fe90 con 0x7fbf24106f10 2024-09-15T21:55:30.172 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:30.172 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:08.111056+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:55:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbf0c0a3240 msgr2=0x7fbf0c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbf0c0a3240 0x7fbf0c0a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf14005e10 tx=0x7fbf14005da0 comp rx=0 tx=0).stop 2024-09-15T21:55:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 msgr2=0x7fbf2419cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2419cc00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c031a20 tx=0x7fbf1c031a50 comp rx=0 tx=0).stop 2024-09-15T21:55:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 shutdown_connections 2024-09-15T21:55:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbf0c0a3240 0x7fbf0c0a5700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:30.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf24106f10 0x7fbf2419cc00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf24102790 0x7fbf24198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 --2- 172.21.15.28:0/3424221026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf24101de0 0x7fbf24198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 >> 172.21.15.28:0/3424221026 conn(0x7fbf240fd8c0 msgr2=0x7fbf240febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 shutdown_connections 2024-09-15T21:55:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:30.172+0000 7fbf29c59640 1 -- 172.21.15.28:0/3424221026 wait complete. 2024-09-15T21:55:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:30 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3424221026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:30.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:30 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3424221026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:30.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-15T21:55:30.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:30 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3424221026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:31.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:31 smithi028 ceph-mon[30969]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 -- 172.21.15.28:0/2454549034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 msgr2=0x7f3a14104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2454549034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a14104e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009900 tx=0x7f39fc02a410 comp rx=0 tx=0).stop 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 -- 172.21.15.28:0/2454549034 shutdown_connections 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2454549034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a14105560 0x7f3a1410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2454549034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a14104e30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2454549034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a14100430 0x7f3a14100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 -- 172.21.15.28:0/2454549034 >> 172.21.15.28:0/2454549034 conn(0x7f3a14077230 msgr2=0x7f3a140fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 -- 172.21.15.28:0/2454549034 shutdown_connections 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 -- 172.21.15.28:0/2454549034 wait complete. 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.693+0000 7f3a1937a640 1 Processor -- start 2024-09-15T21:55:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a1937a640 1 -- start start 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a1937a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a14100430 0x7f3a1410e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a1937a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a1410ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a1937a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a14105560 0x7f3a1410f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a1937a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a14111f50 con 0x7f3a14100de0 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a1937a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a14111dd0 con 0x7f3a14100430 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a1937a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141120d0 con 0x7f3a14105560 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a1410ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a1410ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41018/0 (socket says 172.21.15.28:41018) 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 -- 172.21.15.28:0/2507975130 learned_addr learned my addr 172.21.15.28:0/2507975130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a12ffd640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a14100430 0x7f3a1410e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a137fe640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a14105560 0x7f3a1410f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 -- 172.21.15.28:0/2507975130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a14105560 msgr2=0x7f3a1410f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a14105560 0x7f3a1410f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 -- 172.21.15.28:0/2507975130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a14100430 msgr2=0x7f3a1410e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a14100430 0x7f3a1410e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a127fc640 1 -- 172.21.15.28:0/2507975130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a141a6890 con 0x7f3a14100de0 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.694+0000 7f3a12ffd640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a14100430 0x7f3a1410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.695+0000 7f3a127fc640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a1410ed40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f39fc004770 tx=0x7f39fc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.695+0000 7f3a0bfff640 1 -- 172.21.15.28:0/2507975130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc03f070 con 0x7f3a14100de0 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.695+0000 7f3a0bfff640 1 -- 172.21.15.28:0/2507975130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39fc031ca0 con 0x7f3a14100de0 2024-09-15T21:55:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.695+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a141a6b20 con 0x7f3a14100de0 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.695+0000 7f3a0bfff640 1 -- 172.21.15.28:0/2507975130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc035030 con 0x7f3a14100de0 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.695+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a141a7000 con 0x7f3a14100de0 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.696+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a14100810 con 0x7f3a14100de0 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.697+0000 7f3a0bfff640 1 -- 172.21.15.28:0/2507975130 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f39fc042070 con 0x7f3a14100de0 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.698+0000 7f3a0bfff640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3a000a3240 0x7f3a000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.698+0000 7f3a12ffd640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3a000a3240 0x7f3a000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.698+0000 7f3a0bfff640 1 -- 172.21.15.28:0/2507975130 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f39fc0f0440 con 0x7f3a14100de0 2024-09-15T21:55:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.698+0000 7f3a12ffd640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3a000a3240 0x7f3a000a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a04007e70 tx=0x7f3a04007850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.703+0000 7f3a0bfff640 1 -- 172.21.15.28:0/2507975130 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f39fc0bcf20 con 0x7f3a14100de0 2024-09-15T21:55:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:31 smithi174 ceph-mon[35244]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:31.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:31.916+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3a141081b0 con 0x7f3a14100de0 2024-09-15T21:55:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:31 smithi203 ceph-mon[35546]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.512+0000 7f3a0bfff640 1 -- 172.21.15.28:0/2507975130 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f39fc0bc820 con 0x7f3a14100de0 2024-09-15T21:55:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3a000a3240 msgr2=0x7f3a000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3a000a3240 0x7f3a000a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a04007e70 tx=0x7f3a04007850 comp rx=0 tx=0).stop 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 msgr2=0x7f3a1410ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a1410ed40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f39fc004770 tx=0x7f39fc003c10 comp rx=0 tx=0).stop 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 shutdown_connections 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3a000a3240 0x7f3a000a5700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a14105560 0x7f3a1410f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a14100de0 0x7f3a1410ed40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 --2- 172.21.15.28:0/2507975130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a14100430 0x7f3a1410e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 >> 172.21.15.28:0/2507975130 conn(0x7f3a14077230 msgr2=0x7f3a14103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 shutdown_connections 2024-09-15T21:55:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:32.516+0000 7f3a1937a640 1 -- 172.21.15.28:0/2507975130 wait complete. 2024-09-15T21:55:32.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:32 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2507975130' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T21:55:32.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:32 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2507975130' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T21:55:33.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:32 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2507975130' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T21:55:33.280 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-15T21:55:33.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-09-15T21:55:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:33.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:33 smithi028 ceph-mon[30969]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:33.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:33 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2507975130' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T21:55:33.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:33 smithi028 ceph-mon[30969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:33.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:33 smithi203 ceph-mon[35546]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:33.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:33 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2507975130' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T21:55:33.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:33 smithi203 ceph-mon[35546]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:34.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:33 smithi174 ceph-mon[35244]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:34.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:33 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2507975130' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T21:55:34.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:33 smithi174 ceph-mon[35244]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 -- 172.21.15.28:0/2754376642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 msgr2=0x7fbd80106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/2754376642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd80106e60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009930 tx=0x7fbd7c02a420 comp rx=0 tx=0).stop 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 -- 172.21.15.28:0/2754376642 shutdown_connections 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/2754376642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd80101420 0x7fbd8010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/2754376642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd80100a80 0x7fbd80100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/2754376642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd80106e60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 -- 172.21.15.28:0/2754376642 >> 172.21.15.28:0/2754376642 conn(0x7fbd800fc7a0 msgr2=0x7fbd800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 -- 172.21.15.28:0/2754376642 shutdown_connections 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.073+0000 7fbd87c2b640 1 -- 172.21.15.28:0/2754376642 wait complete. 2024-09-15T21:55:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 Processor -- start 2024-09-15T21:55:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 -- start start 2024-09-15T21:55:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd80100a80 0x7fbd800fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd80101420 0x7fbd800fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd800fd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd80110ea0 con 0x7fbd80106a80 2024-09-15T21:55:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbd80110d20 con 0x7fbd80101420 2024-09-15T21:55:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd87c2b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd80111020 con 0x7fbd80100a80 2024-09-15T21:55:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd861c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd800fd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd861c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd800fd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41026/0 (socket says 172.21.15.28:41026) 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd861c8640 1 -- 172.21.15.28:0/1915343245 learned_addr learned my addr 172.21.15.28:0/1915343245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd851c6640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd80101420 0x7fbd800fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.074+0000 7fbd859c7640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd80100a80 0x7fbd800fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd861c8640 1 -- 172.21.15.28:0/1915343245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd80100a80 msgr2=0x7fbd800fcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd861c8640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd80100a80 0x7fbd800fcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd861c8640 1 -- 172.21.15.28:0/1915343245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd80101420 msgr2=0x7fbd800fd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd861c8640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd80101420 0x7fbd800fd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd861c8640 1 -- 172.21.15.28:0/1915343245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd801a6960 con 0x7fbd80106a80 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd861c8640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd800fd7f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800b970 tx=0x7fbd7800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd76ffd640 1 -- 172.21.15.28:0/1915343245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd78019070 con 0x7fbd80106a80 2024-09-15T21:55:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd76ffd640 1 -- 172.21.15.28:0/1915343245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd78014440 con 0x7fbd80106a80 2024-09-15T21:55:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd801a6c50 con 0x7fbd80106a80 2024-09-15T21:55:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd76ffd640 1 -- 172.21.15.28:0/1915343245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd780075d0 con 0x7fbd80106a80 2024-09-15T21:55:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.075+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd801a7140 con 0x7fbd80106a80 2024-09-15T21:55:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.076+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd80100ee0 con 0x7fbd80106a80 2024-09-15T21:55:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.079+0000 7fbd76ffd640 1 -- 172.21.15.28:0/1915343245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbd7801c070 con 0x7fbd80106a80 2024-09-15T21:55:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.079+0000 7fbd76ffd640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbd6c0a3240 0x7fbd6c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.079+0000 7fbd859c7640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbd6c0a3240 0x7fbd6c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.079+0000 7fbd76ffd640 1 -- 172.21.15.28:0/1915343245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fbd780cf570 con 0x7fbd80106a80 2024-09-15T21:55:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.080+0000 7fbd859c7640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbd6c0a3240 0x7fbd6c0a5700 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009900 tx=0x7fbd7c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.082+0000 7fbd76ffd640 1 -- 172.21.15.28:0/1915343245 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbd7809ba00 con 0x7fbd80106a80 2024-09-15T21:55:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.295+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fbd8010a000 con 0x7fbd80106a80 2024-09-15T21:55:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.521+0000 7fbd76ffd640 1 -- 172.21.15.28:0/1915343245 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fbd7809fd00 con 0x7fbd80106a80 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbd6c0a3240 msgr2=0x7fbd6c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbd6c0a3240 0x7fbd6c0a5700 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009900 tx=0x7fbd7c0058d0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 msgr2=0x7fbd800fd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd800fd7f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800b970 tx=0x7fbd7800bd70 comp rx=0 tx=0).stop 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 shutdown_connections 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbd6c0a3240 0x7fbd6c0a5700 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd80106a80 0x7fbd800fd7f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd80101420 0x7fbd800fd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 --2- 172.21.15.28:0/1915343245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd80100a80 0x7fbd800fcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 >> 172.21.15.28:0/1915343245 conn(0x7fbd800fc7a0 msgr2=0x7fbd8006aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 shutdown_connections 2024-09-15T21:55:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:34.523+0000 7fbd87c2b640 1 -- 172.21.15.28:0/1915343245 wait complete. 2024-09-15T21:55:34.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:34 smithi028 ceph-mon[30969]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:34.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:34 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1915343245' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-15T21:55:34.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:34 smithi203 ceph-mon[35546]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:34.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:34 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1915343245' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-15T21:55:35.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:34 smithi174 ceph-mon[35244]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T21:55:35.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:34 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1915343245' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-15T21:55:35.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-15T21:55:35.596 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:35 smithi028 ceph-mon[30969]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T21:55:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:35 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1915343245' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-15T21:55:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:35 smithi028 ceph-mon[30969]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T21:55:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:35 smithi028 ceph-mon[30969]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:35.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:35 smithi028 ceph-mon[30969]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-15T21:55:35.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:35 smithi028 ceph-mon[30969]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-15T21:55:35.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:35 smithi203 ceph-mon[35546]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T21:55:35.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:35 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1915343245' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-15T21:55:35.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:35 smithi203 ceph-mon[35546]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T21:55:35.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:35 smithi203 ceph-mon[35546]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:35.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:35 smithi203 ceph-mon[35546]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-15T21:55:35.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:35 smithi203 ceph-mon[35546]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-15T21:55:36.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:35 smithi174 ceph-mon[35244]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T21:55:36.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:35 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1915343245' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-15T21:55:36.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:35 smithi174 ceph-mon[35244]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T21:55:36.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:35 smithi174 ceph-mon[35244]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T21:55:36.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:35 smithi174 ceph-mon[35244]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-15T21:55:36.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:35 smithi174 ceph-mon[35244]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-15T21:55:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 -- 172.21.15.28:0/3704544059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 msgr2=0x7f886010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 --2- 172.21.15.28:0/3704544059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f886010ea90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8858009860 tx=0x7f885802a340 comp rx=0 tx=0).stop 2024-09-15T21:55:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 -- 172.21.15.28:0/3704544059 shutdown_connections 2024-09-15T21:55:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 --2- 172.21.15.28:0/3704544059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f886010ea90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 --2- 172.21.15.28:0/3704544059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8860102790 0x7f88601067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 --2- 172.21.15.28:0/3704544059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8860101de0 0x7f88601021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 -- 172.21.15.28:0/3704544059 >> 172.21.15.28:0/3704544059 conn(0x7f88600fd8c0 msgr2=0x7f88600ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 -- 172.21.15.28:0/3704544059 shutdown_connections 2024-09-15T21:55:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.084+0000 7f8865573640 1 -- 172.21.15.28:0/3704544059 wait complete. 2024-09-15T21:55:36.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 Processor -- start 2024-09-15T21:55:36.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 -- start start 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8860101de0 0x7f88600744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8860102790 0x7f8860074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f8860074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8860111f70 con 0x7f8860106f10 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8860111df0 con 0x7f8860101de0 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f8865573640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f88601120f0 con 0x7f8860102790 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f885f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f8860074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f885f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f8860074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41044/0 (socket says 172.21.15.28:41044) 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f885f7fe640 1 -- 172.21.15.28:0/2714354676 learned_addr learned my addr 172.21.15.28:0/2714354676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.085+0000 7f885e7fc640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8860102790 0x7f8860074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f885effd640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8860101de0 0x7f88600744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f885f7fe640 1 -- 172.21.15.28:0/2714354676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8860102790 msgr2=0x7f8860074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f885f7fe640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8860102790 0x7f8860074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f885f7fe640 1 -- 172.21.15.28:0/2714354676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8860101de0 msgr2=0x7f88600744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f885f7fe640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8860101de0 0x7f88600744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f885f7fe640 1 -- 172.21.15.28:0/2714354676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88601a68b0 con 0x7f8860106f10 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f885f7fe640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f8860074f70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8858031000 tx=0x7f8858033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f8857fff640 1 -- 172.21.15.28:0/2714354676 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885803f020 con 0x7f8860106f10 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88601a6b40 con 0x7f8860106f10 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f8857fff640 1 -- 172.21.15.28:0/2714354676 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8858004240 con 0x7f8860106f10 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f8857fff640 1 -- 172.21.15.28:0/2714354676 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858037040 con 0x7f8860106f10 2024-09-15T21:55:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.086+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88601a7020 con 0x7f8860106f10 2024-09-15T21:55:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.088+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88601021c0 con 0x7f8860106f10 2024-09-15T21:55:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.089+0000 7f8857fff640 1 -- 172.21.15.28:0/2714354676 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f8858042020 con 0x7f8860106f10 2024-09-15T21:55:36.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.089+0000 7f8857fff640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f884c0a3240 0x7f884c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.090+0000 7f885effd640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f884c0a3240 0x7f884c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.090+0000 7f8857fff640 1 -- 172.21.15.28:0/2714354676 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f88580efbd0 con 0x7f8860106f10 2024-09-15T21:55:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.090+0000 7f885effd640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f884c0a3240 0x7f884c0a5700 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8850009530 tx=0x7f8850009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.094+0000 7f8857fff640 1 -- 172.21.15.28:0/2714354676 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8858049020 con 0x7f8860106f10 2024-09-15T21:55:36.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.308+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f886010a0f0 con 0x7f8860106f10 2024-09-15T21:55:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.540+0000 7f8857fff640 1 -- 172.21.15.28:0/2714354676 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f88580bc6b0 con 0x7f8860106f10 2024-09-15T21:55:36.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f884c0a3240 msgr2=0x7f884c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:36.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f884c0a3240 0x7f884c0a5700 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8850009530 tx=0x7f8850009260 comp rx=0 tx=0).stop 2024-09-15T21:55:36.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 msgr2=0x7f8860074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:36.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f8860074f70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8858031000 tx=0x7f8858033d90 comp rx=0 tx=0).stop 2024-09-15T21:55:36.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 shutdown_connections 2024-09-15T21:55:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f884c0a3240 0x7f884c0a5700 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8860106f10 0x7f8860074f70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8860102790 0x7f8860074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 --2- 172.21.15.28:0/2714354676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8860101de0 0x7f88600744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.542+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 >> 172.21.15.28:0/2714354676 conn(0x7f88600fd8c0 msgr2=0x7f88600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.543+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 shutdown_connections 2024-09-15T21:55:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:36.543+0000 7f8865573640 1 -- 172.21.15.28:0/2714354676 wait complete. 2024-09-15T21:55:36.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:36 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] up:active 2024-09-15T21:55:36.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:36 smithi028 ceph-mon[30969]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T21:55:36.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:36 smithi028 ceph-mon[30969]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-15T21:55:36.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:36 smithi028 ceph-mon[30969]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-15T21:55:36.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:36 smithi028 ceph-mon[30969]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-15T21:55:36.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:36 smithi028 ceph-mon[30969]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-09-15T21:55:36.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:36 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2714354676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T21:55:36.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:36 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] up:active 2024-09-15T21:55:36.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:36 smithi203 ceph-mon[35546]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T21:55:36.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:36 smithi203 ceph-mon[35546]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-15T21:55:36.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:36 smithi203 ceph-mon[35546]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-15T21:55:36.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:36 smithi203 ceph-mon[35546]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-15T21:55:36.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:36 smithi203 ceph-mon[35546]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-09-15T21:55:36.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:36 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2714354676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T21:55:37.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:36 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] up:active 2024-09-15T21:55:37.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:36 smithi174 ceph-mon[35244]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T21:55:37.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:36 smithi174 ceph-mon[35244]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-15T21:55:37.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:36 smithi174 ceph-mon[35244]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-15T21:55:37.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:36 smithi174 ceph-mon[35244]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-15T21:55:37.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:36 smithi174 ceph-mon[35244]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-09-15T21:55:37.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:36 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2714354676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T21:55:37.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-15T21:55:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:37 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2714354676' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T21:55:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:37 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] up:active 2024-09-15T21:55:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:37 smithi028 ceph-mon[30969]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-15T21:55:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:37 smithi028 ceph-mon[30969]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-15T21:55:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:37 smithi028 ceph-mon[30969]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-09-15T21:55:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:37 smithi028 ceph-mon[30969]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-09-15T21:55:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:37 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2714354676' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T21:55:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:37 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] up:active 2024-09-15T21:55:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:37 smithi203 ceph-mon[35546]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-15T21:55:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:37 smithi203 ceph-mon[35546]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-15T21:55:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:37 smithi203 ceph-mon[35546]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-09-15T21:55:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:37 smithi203 ceph-mon[35546]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-09-15T21:55:38.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:37 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2714354676' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T21:55:38.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:37 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] up:active 2024-09-15T21:55:38.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:37 smithi174 ceph-mon[35244]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-15T21:55:38.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:37 smithi174 ceph-mon[35244]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-15T21:55:38.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:37 smithi174 ceph-mon[35244]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-09-15T21:55:38.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:37 smithi174 ceph-mon[35244]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-09-15T21:55:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 -- 172.21.15.28:0/3718814009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100a80 msgr2=0x7f0074100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 --2- 172.21.15.28:0/3718814009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100a80 0x7f0074100ee0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f005c009930 tx=0x7f005c02a420 comp rx=0 tx=0).stop 2024-09-15T21:55:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 -- 172.21.15.28:0/3718814009 shutdown_connections 2024-09-15T21:55:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 --2- 172.21.15.28:0/3718814009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0074101420 0x7f007410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 --2- 172.21.15.28:0/3718814009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100a80 0x7f0074100ee0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 --2- 172.21.15.28:0/3718814009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0074106a80 0x7f0074106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 -- 172.21.15.28:0/3718814009 >> 172.21.15.28:0/3718814009 conn(0x7f00740fc7a0 msgr2=0x7f00740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 -- 172.21.15.28:0/3718814009 shutdown_connections 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.193+0000 7f0079322640 1 -- 172.21.15.28:0/3718814009 wait complete. 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 Processor -- start 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 -- start start 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0074100a80 0x7f0074198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074101420 0x7f00741989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0074106a80 0x7f007419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0074111fa0 con 0x7f0074101420 2024-09-15T21:55:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0074111e20 con 0x7f0074100a80 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f0079322640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0074112120 con 0x7f0074106a80 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.194+0000 7f006bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074101420 0x7f00741989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074101420 0x7f00741989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41054/0 (socket says 172.21.15.28:41054) 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 -- 172.21.15.28:0/878332886 learned_addr learned my addr 172.21.15.28:0/878332886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f0072ffd640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0074100a80 0x7f0074198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f00737fe640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0074106a80 0x7f007419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 -- 172.21.15.28:0/878332886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0074106a80 msgr2=0x7f007419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0074106a80 0x7f007419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 -- 172.21.15.28:0/878332886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0074100a80 msgr2=0x7f0074198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0074100a80 0x7f0074198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 -- 172.21.15.28:0/878332886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007419d430 con 0x7f0074101420 2024-09-15T21:55:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f00737fe640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0074106a80 0x7f007419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f0072ffd640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0074100a80 0x7f0074198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f006bfff640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074101420 0x7f00741989c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f005c009230 tx=0x7f005c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f0070ff9640 1 -- 172.21.15.28:0/878332886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c03f030 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f0070ff9640 1 -- 172.21.15.28:0/878332886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f005c003ad0 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007419d6c0 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f0070ff9640 1 -- 172.21.15.28:0/878332886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c035070 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.195+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00741a5000 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.197+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00741021c0 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.198+0000 7f0070ff9640 1 -- 172.21.15.28:0/878332886 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f005c042030 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.198+0000 7f0070ff9640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f00600a3240 0x7f00600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.199+0000 7f0072ffd640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f00600a3240 0x7f00600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.199+0000 7f0070ff9640 1 -- 172.21.15.28:0/878332886 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f005c0f0510 con 0x7f0074101420 2024-09-15T21:55:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.199+0000 7f0072ffd640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f00600a3240 0x7f00600a5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0064005fd0 tx=0x7f006400b5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.201+0000 7f0070ff9640 1 -- 172.21.15.28:0/878332886 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f005c0bcff0 con 0x7f0074101420 2024-09-15T21:55:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.415+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f0074109010 con 0x7f0074101420 2024-09-15T21:55:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.554+0000 7f0070ff9640 1 -- 172.21.15.28:0/878332886 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7f005c0bc8f0 con 0x7f0074101420 2024-09-15T21:55:38.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.556+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f00600a3240 msgr2=0x7f00600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.556+0000 7f0079322640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f00600a3240 0x7f00600a5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0064005fd0 tx=0x7f006400b5d0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.556+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074101420 msgr2=0x7f00741989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.556+0000 7f0079322640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074101420 0x7f00741989c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f005c009230 tx=0x7f005c004270 comp rx=0 tx=0).stop 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 shutdown_connections 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f00600a3240 0x7f00600a5700 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0074106a80 0x7f007419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074101420 0x7f00741989c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 --2- 172.21.15.28:0/878332886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0074100a80 0x7f0074198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 >> 172.21.15.28:0/878332886 conn(0x7f00740fc7a0 msgr2=0x7f00740fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 shutdown_connections 2024-09-15T21:55:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:38.557+0000 7f0079322640 1 -- 172.21.15.28:0/878332886 wait complete. 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: pgmap v308: 129 pgs: 129 active+clean; 582 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] up:active 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: Cluster is now healthy 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-09-15T21:55:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:38 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/878332886' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T21:55:38.760 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: 2024-09-15T21:55:38.497+0000 7f08b7d22640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-15T21:55:38.760 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: 2024-09-15T21:55:38.497+0000 7f08b7d22640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-15T21:55:38.760 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: 2024-09-15T21:55:38.497+0000 7f08b7d22640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-09-15T21:55:38.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: pgmap v308: 129 pgs: 129 active+clean; 582 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-09-15T21:55:38.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] up:active 2024-09-15T21:55:38.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-15T21:55:38.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T21:55:38.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: Cluster is now healthy 2024-09-15T21:55:38.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-09-15T21:55:38.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-15T21:55:38.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-09-15T21:55:38.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:38 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/878332886' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T21:55:38.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: pgmap v308: 129 pgs: 129 active+clean; 582 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 1 op/s 2024-09-15T21:55:38.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] up:active 2024-09-15T21:55:38.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-15T21:55:38.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T21:55:38.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: Cluster is now healthy 2024-09-15T21:55:38.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-09-15T21:55:38.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-15T21:55:38.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-09-15T21:55:38.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:38 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/878332886' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T21:55:39.245 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-09-15T21:55:39.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-15T21:55:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:39.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:39 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/878332886' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T21:55:39.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:39 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] up:active 2024-09-15T21:55:39.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:39 smithi028 ceph-mon[30969]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-15T21:55:39.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:39 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:55:39.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:39 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/878332886' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T21:55:39.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:39 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] up:active 2024-09-15T21:55:39.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:39 smithi203 ceph-mon[35546]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-15T21:55:39.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:39 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:55:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 -- 172.21.15.28:0/2053125570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100d00 msgr2=0x7fe434106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2053125570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100d00 0x7fe434106840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009930 tx=0x7fe41c02a420 comp rx=0 tx=0).stop 2024-09-15T21:55:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 -- 172.21.15.28:0/2053125570 shutdown_connections 2024-09-15T21:55:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2053125570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434106f50 0x7fe43410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2053125570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100d00 0x7fe434106840 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2053125570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe434100350 0x7fe434100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 -- 172.21.15.28:0/2053125570 >> 172.21.15.28:0/2053125570 conn(0x7fe434077210 msgr2=0x7fe4340fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 -- 172.21.15.28:0/2053125570 shutdown_connections 2024-09-15T21:55:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.992+0000 7fe43a90b640 1 -- 172.21.15.28:0/2053125570 wait complete. 2024-09-15T21:55:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.993+0000 7fe43a90b640 1 Processor -- start 2024-09-15T21:55:39.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.993+0000 7fe43a90b640 1 -- start start 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.993+0000 7fe43a90b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100350 0x7fe4341012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe43a90b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100d00 0x7fe434101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe43a90b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe434106f50 0x7fe434101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe43a90b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe434111f60 con 0x7fe434100350 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe43a90b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe434111de0 con 0x7fe434106f50 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe43a90b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe4341120e0 con 0x7fe434100d00 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100350 0x7fe4341012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100350 0x7fe4341012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41082/0 (socket says 172.21.15.28:41082) 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 -- 172.21.15.28:0/2705431241 learned_addr learned my addr 172.21.15.28:0/2705431241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe42bfff640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100d00 0x7fe434101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe438ea8640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe434106f50 0x7fe434101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 -- 172.21.15.28:0/2705431241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100d00 msgr2=0x7fe434101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100d00 0x7fe434101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 -- 172.21.15.28:0/2705431241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe434106f50 msgr2=0x7fe434101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe434106f50 0x7fe434101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 -- 172.21.15.28:0/2705431241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4341a68a0 con 0x7fe434100350 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe42bfff640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100d00 0x7fe434101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe433fff640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100350 0x7fe4341012d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe42400c920 tx=0x7fe42400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe431ffb640 1 -- 172.21.15.28:0/2705431241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424007c70 con 0x7fe434100350 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe431ffb640 1 -- 172.21.15.28:0/2705431241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe424007e10 con 0x7fe434100350 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.994+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4341a6b90 con 0x7fe434100350 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.995+0000 7fe431ffb640 1 -- 172.21.15.28:0/2705431241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424018890 con 0x7fe434100350 2024-09-15T21:55:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.995+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4341a70a0 con 0x7fe434100350 2024-09-15T21:55:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.996+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe434103410 con 0x7fe434100350 2024-09-15T21:55:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.997+0000 7fe431ffb640 1 -- 172.21.15.28:0/2705431241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe424018a50 con 0x7fe434100350 2024-09-15T21:55:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.997+0000 7fe431ffb640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe4200a3240 0x7fe4200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.998+0000 7fe42bfff640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe4200a3240 0x7fe4200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.998+0000 7fe431ffb640 1 -- 172.21.15.28:0/2705431241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fe4240cfd70 con 0x7fe434100350 2024-09-15T21:55:40.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:39.998+0000 7fe42bfff640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe4200a3240 0x7fe4200a5700 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe41c006010 tx=0x7fe41c0314b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:40.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.002+0000 7fe431ffb640 1 -- 172.21.15.28:0/2705431241 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe424014030 con 0x7fe434100350 2024-09-15T21:55:40.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:39 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/878332886' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T21:55:40.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:39 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] up:active 2024-09-15T21:55:40.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:39 smithi174 ceph-mon[35244]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-15T21:55:40.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:39 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:55:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.200+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe434062630 con 0x7fe4200a3240 2024-09-15T21:55:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.226+0000 7fe431ffb640 1 -- 172.21.15.28:0/2705431241 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe434062630 con 0x7fe4200a3240 2024-09-15T21:55:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe4200a3240 msgr2=0x7fe4200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe4200a3240 0x7fe4200a5700 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe41c006010 tx=0x7fe41c0314b0 comp rx=0 tx=0).stop 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100350 msgr2=0x7fe4341012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100350 0x7fe4341012d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe42400c920 tx=0x7fe42400cde0 comp rx=0 tx=0).stop 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 shutdown_connections 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe4200a3240 0x7fe4200a5700 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe434106f50 0x7fe434101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe434100d00 0x7fe434101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 --2- 172.21.15.28:0/2705431241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe434100350 0x7fe4341012d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 >> 172.21.15.28:0/2705431241 conn(0x7fe434077210 msgr2=0x7fe434105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 shutdown_connections 2024-09-15T21:55:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:40.229+0000 7fe43a90b640 1 -- 172.21.15.28:0/2705431241 wait complete. 2024-09-15T21:55:40.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:55:40 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:55:40.204+0000 7fa6bc68a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:55:40.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:55:40 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:55:40.204+0000 7fa6bc68a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:55:40.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:55:40 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:55:40.204+0000 7fa6bc68a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:55:40.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:55:40 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:55:40.204+0000 7fa6bc68a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:55:40.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:55:40 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:55:40.204+0000 7fa6bc68a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:55:40.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:40 smithi028 ceph-mon[30969]: pgmap v309: 129 pgs: 129 active+clean; 585 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-09-15T21:55:40.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:40 smithi028 ceph-mon[30969]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:40.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-15T21:55:40.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:40 smithi203 ceph-mon[35546]: pgmap v309: 129 pgs: 129 active+clean; 585 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-09-15T21:55:40.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:40 smithi203 ceph-mon[35546]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:41.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:40 smithi174 ceph-mon[35244]: pgmap v309: 129 pgs: 129 active+clean; 585 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-09-15T21:55:41.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:40 smithi174 ceph-mon[35244]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 -- 172.21.15.28:0/2493400258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff56810c170 msgr2=0x7ff56810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/2493400258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff56810c170 0x7ff56810e610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff560009ca0 tx=0x7ff56002a950 comp rx=0 tx=0).stop 2024-09-15T21:55:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 -- 172.21.15.28:0/2493400258 shutdown_connections 2024-09-15T21:55:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/2493400258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff56810c170 0x7ff56810e610 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/2493400258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff568103660 0x7ff56810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/2493400258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff568102d40 0x7ff568103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 -- 172.21.15.28:0/2493400258 >> 172.21.15.28:0/2493400258 conn(0x7ff5680fc7a0 msgr2=0x7ff5680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 -- 172.21.15.28:0/2493400258 shutdown_connections 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.830+0000 7ff56e9f1640 1 -- 172.21.15.28:0/2493400258 wait complete. 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 Processor -- start 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 -- start start 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568102d40 0x7ff5681009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff568103660 0x7ff568100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff56810c170 0x7ff568101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff568112030 con 0x7ff568102d40 2024-09-15T21:55:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff568111eb0 con 0x7ff56810c170 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.831+0000 7ff56e9f1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5681121b0 con 0x7ff568103660 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568102d40 0x7ff5681009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568102d40 0x7ff5681009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35532/0 (socket says 172.21.15.28:35532) 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 -- 172.21.15.28:0/1709522258 learned_addr learned my addr 172.21.15.28:0/1709522258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff5677fe640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff568103660 0x7ff568100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff56cf8e640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff56810c170 0x7ff568101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 -- 172.21.15.28:0/1709522258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff568103660 msgr2=0x7ff568100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff568103660 0x7ff568100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 -- 172.21.15.28:0/1709522258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff56810c170 msgr2=0x7ff568101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff56810c170 0x7ff568101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 -- 172.21.15.28:0/1709522258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5681aaff0 con 0x7ff568102d40 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff567fff640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568102d40 0x7ff5681009a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff55800b4a0 tx=0x7ff55800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff5657fa640 1 -- 172.21.15.28:0/1709522258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558014070 con 0x7ff568102d40 2024-09-15T21:55:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff5657fa640 1 -- 172.21.15.28:0/1709522258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff558010af0 con 0x7ff568102d40 2024-09-15T21:55:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5681ab220 con 0x7ff568102d40 2024-09-15T21:55:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff5657fa640 1 -- 172.21.15.28:0/1709522258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558017d10 con 0x7ff568102d40 2024-09-15T21:55:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.832+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5681ab7b0 con 0x7ff568102d40 2024-09-15T21:55:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.833+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5681031a0 con 0x7ff568102d40 2024-09-15T21:55:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.835+0000 7ff5657fa640 1 -- 172.21.15.28:0/1709522258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff558017070 con 0x7ff568102d40 2024-09-15T21:55:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.835+0000 7ff5657fa640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5500a32c0 0x7ff5500a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.836+0000 7ff5657fa640 1 -- 172.21.15.28:0/1709522258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7ff5580ceef0 con 0x7ff568102d40 2024-09-15T21:55:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.836+0000 7ff5677fe640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5500a32c0 0x7ff5500a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.836+0000 7ff5677fe640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5500a32c0 0x7ff5500a5780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff5680fee00 tx=0x7ff55c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:41.840+0000 7ff5657fa640 1 -- 172.21.15.28:0/1709522258 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff55809b9d0 con 0x7ff568102d40 2024-09-15T21:55:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.041+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff568062630 con 0x7ff5500a32c0 2024-09-15T21:55:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.065+0000 7ff5657fa640 1 -- 172.21.15.28:0/1709522258 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff568062630 con 0x7ff5500a32c0 2024-09-15T21:55:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5500a32c0 msgr2=0x7ff5500a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5500a32c0 0x7ff5500a5780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff5680fee00 tx=0x7ff55c008040 comp rx=0 tx=0).stop 2024-09-15T21:55:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568102d40 msgr2=0x7ff5681009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568102d40 0x7ff5681009a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff55800b4a0 tx=0x7ff55800b960 comp rx=0 tx=0).stop 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 shutdown_connections 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ff5500a32c0 0x7ff5500a5780 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff56810c170 0x7ff568101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff568103660 0x7ff568100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 --2- 172.21.15.28:0/1709522258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff568102d40 0x7ff5681009a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 >> 172.21.15.28:0/1709522258 conn(0x7ff5680fc7a0 msgr2=0x7ff5680fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 shutdown_connections 2024-09-15T21:55:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:42.067+0000 7ff56e9f1640 1 -- 172.21.15.28:0/1709522258 wait complete. 2024-09-15T21:55:42.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:42 smithi028 ceph-mon[30969]: pgmap v310: 129 pgs: 129 active+clean; 587 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-09-15T21:55:42.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:42 smithi028 ceph-mon[30969]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:42.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-15T21:55:42.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:42 smithi203 ceph-mon[35546]: pgmap v310: 129 pgs: 129 active+clean; 587 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-09-15T21:55:42.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:42 smithi203 ceph-mon[35546]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:42 smithi174 ceph-mon[35244]: pgmap v310: 129 pgs: 129 active+clean; 587 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-09-15T21:55:43.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:42 smithi174 ceph-mon[35244]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:43.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 -- 172.21.15.28:0/651770872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 msgr2=0x7f271810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:43.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 --2- 172.21.15.28:0/651770872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810d950 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2710009cd0 tx=0x7f271002a8e0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 -- 172.21.15.28:0/651770872 shutdown_connections 2024-09-15T21:55:43.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 --2- 172.21.15.28:0/651770872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810d950 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 --2- 172.21.15.28:0/651770872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718100a80 0x7f2718100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 --2- 172.21.15.28:0/651770872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2718106a80 0x7f2718106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 -- 172.21.15.28:0/651770872 >> 172.21.15.28:0/651770872 conn(0x7f27180fc7a0 msgr2=0x7f27180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 -- 172.21.15.28:0/651770872 shutdown_connections 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.506+0000 7f271fc18640 1 -- 172.21.15.28:0/651770872 wait complete. 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 Processor -- start 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 -- start start 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718100a80 0x7f271810e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2718106a80 0x7f271810ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2718111fa0 con 0x7f2718101420 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2718111e20 con 0x7f2718100a80 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271fc18640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2718112120 con 0x7f2718106a80 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271d1b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271d1b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35554/0 (socket says 172.21.15.28:35554) 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.507+0000 7f271d1b3640 1 -- 172.21.15.28:0/3097571729 learned_addr learned my addr 172.21.15.28:0/3097571729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d9b4640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718100a80 0x7f271810e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d1b3640 1 -- 172.21.15.28:0/3097571729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2718106a80 msgr2=0x7f271810ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271e1b5640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2718106a80 0x7f271810ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d1b3640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2718106a80 0x7f271810ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d1b3640 1 -- 172.21.15.28:0/3097571729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718100a80 msgr2=0x7f271810e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d1b3640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718100a80 0x7f271810e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d1b3640 1 -- 172.21.15.28:0/3097571729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27181a68f0 con 0x7f2718101420 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d9b4640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718100a80 0x7f271810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271d1b3640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810e920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f270800d950 tx=0x7f270800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f270effd640 1 -- 172.21.15.28:0/3097571729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708017030 con 0x7f2718101420 2024-09-15T21:55:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f270effd640 1 -- 172.21.15.28:0/3097571729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2708004020 con 0x7f2718101420 2024-09-15T21:55:43.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f270effd640 1 -- 172.21.15.28:0/3097571729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080047a0 con 0x7f2718101420 2024-09-15T21:55:43.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27181a6be0 con 0x7f2718101420 2024-09-15T21:55:43.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.508+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27181a7170 con 0x7f2718101420 2024-09-15T21:55:43.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.509+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2718100ee0 con 0x7f2718101420 2024-09-15T21:55:43.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.510+0000 7f270effd640 1 -- 172.21.15.28:0/3097571729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f27080041d0 con 0x7f2718101420 2024-09-15T21:55:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.511+0000 7f270effd640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f27040a3350 0x7f27040a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.511+0000 7f270effd640 1 -- 172.21.15.28:0/3097571729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f27080cf700 con 0x7f2718101420 2024-09-15T21:55:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.511+0000 7f271d9b4640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f27040a3350 0x7f27040a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.511+0000 7f271d9b4640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f27040a3350 0x7f27040a5810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2714005fd0 tx=0x7f27140094d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.515+0000 7f270effd640 1 -- 172.21.15.28:0/3097571729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f270809c080 con 0x7f2718101420 2024-09-15T21:55:43.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:43 smithi028 ceph-mon[30969]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T21:55:43.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.719+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f2718062630 con 0x7f27040a3350 2024-09-15T21:55:43.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.724+0000 7f270effd640 1 -- 172.21.15.28:0/3097571729 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2718062630 con 0x7f27040a3350 2024-09-15T21:55:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/6ba490c6-d8bd-4d99-80a9-04d57d9da6fe 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f27040a3350 msgr2=0x7f27040a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f27040a3350 0x7f27040a5810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2714005fd0 tx=0x7f27140094d0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 msgr2=0x7f271810e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810e920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f270800d950 tx=0x7f270800de10 comp rx=0 tx=0).stop 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 shutdown_connections 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f27040a3350 0x7f27040a5810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2718106a80 0x7f271810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2718101420 0x7f271810e920 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 --2- 172.21.15.28:0/3097571729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2718100a80 0x7f271810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:43.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 >> 172.21.15.28:0/3097571729 conn(0x7f27180fc7a0 msgr2=0x7f27180fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:43.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 shutdown_connections 2024-09-15T21:55:43.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:43.726+0000 7f271fc18640 1 -- 172.21.15.28:0/3097571729 wait complete. 2024-09-15T21:55:43.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:43 smithi203 ceph-mon[35546]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T21:55:44.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:43 smithi174 ceph-mon[35244]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T21:55:44.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-15T21:55:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:44 smithi028 ceph-mon[30969]: pgmap v311: 129 pgs: 129 active+clean; 587 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-09-15T21:55:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:44 smithi028 ceph-mon[30969]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:44.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:44 smithi203 ceph-mon[35546]: pgmap v311: 129 pgs: 129 active+clean; 587 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-09-15T21:55:44.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:44 smithi203 ceph-mon[35546]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:45.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:44 smithi174 ceph-mon[35244]: pgmap v311: 129 pgs: 129 active+clean; 587 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-09-15T21:55:45.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:44 smithi174 ceph-mon[35244]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 -- 172.21.15.28:0/1978589889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0101420 msgr2=0x7f8ee010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/1978589889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0101420 0x7f8ee010d950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8009cd0 tx=0x7f8ed802a8e0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 -- 172.21.15.28:0/1978589889 shutdown_connections 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/1978589889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0101420 0x7f8ee010d950 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/1978589889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ee0100a80 0x7f8ee0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/1978589889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ee0106a80 0x7f8ee0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 -- 172.21.15.28:0/1978589889 >> 172.21.15.28:0/1978589889 conn(0x7f8ee00fc7a0 msgr2=0x7f8ee00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 -- 172.21.15.28:0/1978589889 shutdown_connections 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.205+0000 7f8ee5b69640 1 -- 172.21.15.28:0/1978589889 wait complete. 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 Processor -- start 2024-09-15T21:55:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 -- start start 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ee0100a80 0x7f8ee01972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ee0101420 0x7f8ee0197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0106a80 0x7f8ee019bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee0110e80 con 0x7f8ee0106a80 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee0110d00 con 0x7f8ee0100a80 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8ee5b69640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee0111000 con 0x7f8ee0101420 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8edffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0106a80 0x7f8ee019bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8edffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0106a80 0x7f8ee019bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35578/0 (socket says 172.21.15.28:35578) 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8edffff640 1 -- 172.21.15.28:0/2176746996 learned_addr learned my addr 172.21.15.28:0/2176746996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.206+0000 7f8edf7fe640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ee0100a80 0x7f8ee01972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edeffd640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ee0101420 0x7f8ee0197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edffff640 1 -- 172.21.15.28:0/2176746996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ee0101420 msgr2=0x7f8ee0197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edffff640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ee0101420 0x7f8ee0197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edffff640 1 -- 172.21.15.28:0/2176746996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ee0100a80 msgr2=0x7f8ee01972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edffff640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ee0100a80 0x7f8ee01972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edffff640 1 -- 172.21.15.28:0/2176746996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee019c280 con 0x7f8ee0106a80 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edf7fe640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ee0100a80 0x7f8ee01972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edffff640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0106a80 0x7f8ee019bba0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8005cc0 tx=0x7f8ed8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edcff9640 1 -- 172.21.15.28:0/2176746996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8004040 con 0x7f8ee0106a80 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee019c510 con 0x7f8ee0106a80 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edcff9640 1 -- 172.21.15.28:0/2176746996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ed80046c0 con 0x7f8ee0106a80 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8edcff9640 1 -- 172.21.15.28:0/2176746996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8046480 con 0x7f8ee0106a80 2024-09-15T21:55:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.207+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee01a3e50 con 0x7f8ee0106a80 2024-09-15T21:55:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.208+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee0100ee0 con 0x7f8ee0106a80 2024-09-15T21:55:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.209+0000 7f8edcff9640 1 -- 172.21.15.28:0/2176746996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8ed80305a0 con 0x7f8ee0106a80 2024-09-15T21:55:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.210+0000 7f8edcff9640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8ec80a32d0 0x7f8ec80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.210+0000 7f8edcff9640 1 -- 172.21.15.28:0/2176746996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f8ed802e030 con 0x7f8ee0106a80 2024-09-15T21:55:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.210+0000 7f8edf7fe640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8ec80a32d0 0x7f8ec80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.211+0000 7f8edf7fe640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8ec80a32d0 0x7f8ec80a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0005fd0 tx=0x7f8ed0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.215+0000 7f8edcff9640 1 -- 172.21.15.28:0/2176746996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8ed804b060 con 0x7f8ee0106a80 2024-09-15T21:55:45.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.416+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8ee0062630 con 0x7f8ec80a32d0 2024-09-15T21:55:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.442+0000 7f8edcff9640 1 -- 172.21.15.28:0/2176746996 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8ee0062630 con 0x7f8ec80a32d0 2024-09-15T21:55:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.444+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8ec80a32d0 msgr2=0x7f8ec80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.444+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8ec80a32d0 0x7f8ec80a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0005fd0 tx=0x7f8ed0005850 comp rx=0 tx=0).stop 2024-09-15T21:55:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.444+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0106a80 msgr2=0x7f8ee019bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.444+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0106a80 0x7f8ee019bba0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8005cc0 tx=0x7f8ed8003970 comp rx=0 tx=0).stop 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 shutdown_connections 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8ec80a32d0 0x7f8ec80a5790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0106a80 0x7f8ee019bba0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ee0101420 0x7f8ee0197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 --2- 172.21.15.28:0/2176746996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ee0100a80 0x7f8ee01972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 >> 172.21.15.28:0/2176746996 conn(0x7f8ee00fc7a0 msgr2=0x7f8ee00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 shutdown_connections 2024-09-15T21:55:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:45.445+0000 7f8ee5b69640 1 -- 172.21.15.28:0/2176746996 wait complete. 2024-09-15T21:55:45.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:45 smithi028 ceph-mon[30969]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:45.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:45 smithi203 ceph-mon[35546]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:46.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:45 smithi174 ceph-mon[35244]: from='client.14829 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:46.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-15T21:55:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:46.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:46 smithi028 ceph-mon[30969]: pgmap v312: 129 pgs: 129 active+clean; 612 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 22 op/s 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.858+0000 7f76ed883640 1 -- 172.21.15.28:0/3187054027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 msgr2=0x7f76e81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.858+0000 7f76ed883640 1 --2- 172.21.15.28:0/3187054027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81021c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009900 tx=0x7f76e002a410 comp rx=0 tx=0).stop 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.858+0000 7f76ed883640 1 -- 172.21.15.28:0/3187054027 shutdown_connections 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.858+0000 7f76ed883640 1 --2- 172.21.15.28:0/3187054027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76e8106f10 0x7f76e810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.858+0000 7f76ed883640 1 --2- 172.21.15.28:0/3187054027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76e8102790 0x7f76e81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.858+0000 7f76ed883640 1 --2- 172.21.15.28:0/3187054027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81021c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.858+0000 7f76ed883640 1 -- 172.21.15.28:0/3187054027 >> 172.21.15.28:0/3187054027 conn(0x7f76e80fd8c0 msgr2=0x7f76e80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:46.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.859+0000 7f76ed883640 1 -- 172.21.15.28:0/3187054027 shutdown_connections 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.859+0000 7f76ed883640 1 -- 172.21.15.28:0/3187054027 wait complete. 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.859+0000 7f76ed883640 1 Processor -- start 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.859+0000 7f76ed883640 1 -- start start 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.859+0000 7f76ed883640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.859+0000 7f76ed883640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76e8102790 0x7f76e8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ed883640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76e8106f10 0x7f76e819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ed883640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76e8111f70 con 0x7f76e8101de0 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ed883640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f76e8111df0 con 0x7f76e8102790 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ed883640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f76e81120f0 con 0x7f76e8106f10 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35596/0 (socket says 172.21.15.28:35596) 2024-09-15T21:55:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 -- 172.21.15.28:0/4036132284 learned_addr learned my addr 172.21.15.28:0/4036132284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e67fc640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76e8102790 0x7f76e8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e77fe640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76e8106f10 0x7f76e819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 -- 172.21.15.28:0/4036132284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76e8106f10 msgr2=0x7f76e819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76e8106f10 0x7f76e819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 -- 172.21.15.28:0/4036132284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76e8102790 msgr2=0x7f76e8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76e8102790 0x7f76e8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 -- 172.21.15.28:0/4036132284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e819d480 con 0x7f76e8101de0 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e77fe640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76e8106f10 0x7f76e819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76e6ffd640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81984d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f76e0006010 tx=0x7f76e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ec881640 1 -- 172.21.15.28:0/4036132284 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e003f030 con 0x7f76e8101de0 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ec881640 1 -- 172.21.15.28:0/4036132284 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e0003ce0 con 0x7f76e8101de0 2024-09-15T21:55:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e819d710 con 0x7f76e8101de0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ec881640 1 -- 172.21.15.28:0/4036132284 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0035070 con 0x7f76e8101de0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.860+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e81a5050 con 0x7f76e8101de0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.862+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76e81032c0 con 0x7f76e8101de0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.863+0000 7f76ec881640 1 -- 172.21.15.28:0/4036132284 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f76e0042030 con 0x7f76e8101de0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.864+0000 7f76ec881640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f76d40a32d0 0x7f76d40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.864+0000 7f76e67fc640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f76d40a32d0 0x7f76d40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.864+0000 7f76ec881640 1 -- 172.21.15.28:0/4036132284 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f76e00f0400 con 0x7f76e8101de0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.864+0000 7f76e67fc640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f76d40a32d0 0x7f76d40a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f76e80fdd00 tx=0x7f76d000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:46.868+0000 7f76ec881640 1 -- 172.21.15.28:0/4036132284 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f76e00bcde0 con 0x7f76e8101de0 2024-09-15T21:55:46.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:46 smithi203 ceph-mon[35546]: pgmap v312: 129 pgs: 129 active+clean; 612 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 22 op/s 2024-09-15T21:55:47.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:46 smithi174 ceph-mon[35244]: pgmap v312: 129 pgs: 129 active+clean; 612 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.0 KiB/s wr, 22 op/s 2024-09-15T21:55:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.074+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f76e8062630 con 0x7f76d40a32d0 2024-09-15T21:55:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.079+0000 7f76ec881640 1 -- 172.21.15.28:0/4036132284 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f76e8062630 con 0x7f76d40a32d0 2024-09-15T21:55:47.080 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/485a13b0-7b8e-4b77-ac19-edf83448c265 2024-09-15T21:55:47.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.080+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f76d40a32d0 msgr2=0x7f76d40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:47.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f76d40a32d0 0x7f76d40a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f76e80fdd00 tx=0x7f76d000b390 comp rx=0 tx=0).stop 2024-09-15T21:55:47.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 msgr2=0x7f76e81984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81984d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f76e0006010 tx=0x7f76e00042f0 comp rx=0 tx=0).stop 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 shutdown_connections 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f76d40a32d0 0x7f76d40a5790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76e8106f10 0x7f76e819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76e8102790 0x7f76e8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 --2- 172.21.15.28:0/4036132284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76e8101de0 0x7f76e81984d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 >> 172.21.15.28:0/4036132284 conn(0x7f76e80fd8c0 msgr2=0x7f76e80fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 shutdown_connections 2024-09-15T21:55:47.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:47.081+0000 7f76ed883640 1 -- 172.21.15.28:0/4036132284 wait complete. 2024-09-15T21:55:47.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:55:47.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:47 smithi028 ceph-mon[30969]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:47 smithi203 ceph-mon[35546]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:48.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:47 smithi174 ceph-mon[35244]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:55:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/591995959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 msgr2=0x7f4f8810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/591995959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8810bb50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009900 tx=0x7f4f7c02a480 comp rx=0 tx=0).stop 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/591995959 shutdown_connections 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/591995959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f8810c170 0x7f4f8810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/591995959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8810bb50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/591995959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f88102d40 0x7f4f88103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/591995959 >> 172.21.15.28:0/591995959 conn(0x7f4f880fc7a0 msgr2=0x7f4f880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/591995959 shutdown_connections 2024-09-15T21:55:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.546+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/591995959 wait complete. 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 Processor -- start 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 -- start start 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f88102d40 0x7f4f8819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8810c170 0x7f4f8819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f88111f70 con 0x7f4f88103660 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4f88111df0 con 0x7f4f8810c170 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f8e0a6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f881120f0 con 0x7f4f88102d40 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f86ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f86ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8819ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35614/0 (socket says 172.21.15.28:35614) 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.547+0000 7f4f86ffd640 1 -- 172.21.15.28:0/3827458945 learned_addr learned my addr 172.21.15.28:0/3827458945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f877fe640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f88102d40 0x7f4f8819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f87fff640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8810c170 0x7f4f8819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f86ffd640 1 -- 172.21.15.28:0/3827458945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f88102d40 msgr2=0x7f4f8819a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f86ffd640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f88102d40 0x7f4f8819a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f86ffd640 1 -- 172.21.15.28:0/3827458945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8810c170 msgr2=0x7f4f8819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f86ffd640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8810c170 0x7f4f8819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f86ffd640 1 -- 172.21.15.28:0/3827458945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8819f600 con 0x7f4f88103660 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f86ffd640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8819ab90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c006010 tx=0x7f4f7c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f84ff9640 1 -- 172.21.15.28:0/3827458945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c03f030 con 0x7f4f88103660 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f84ff9640 1 -- 172.21.15.28:0/3827458945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f7c003ce0 con 0x7f4f88103660 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f84ff9640 1 -- 172.21.15.28:0/3827458945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c035070 con 0x7f4f88103660 2024-09-15T21:55:48.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8819f890 con 0x7f4f88103660 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.548+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f881a71d0 con 0x7f4f88103660 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.549+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8806b070 con 0x7f4f88103660 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.550+0000 7f4f84ff9640 1 -- 172.21.15.28:0/3827458945 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4f7c042030 con 0x7f4f88103660 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.551+0000 7f4f84ff9640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f700a32d0 0x7f4f700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.551+0000 7f4f877fe640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f700a32d0 0x7f4f700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.551+0000 7f4f84ff9640 1 -- 172.21.15.28:0/3827458945 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f4f7c0f0280 con 0x7f4f88103660 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.552+0000 7f4f877fe640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f700a32d0 0x7f4f700a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800bbe0 tx=0x7f4f7800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.555+0000 7f4f84ff9640 1 -- 172.21.15.28:0/3827458945 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4f7c0bcd00 con 0x7f4f88103660 2024-09-15T21:55:48.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.771+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f881a73a0 con 0x7f4f88103660 2024-09-15T21:55:48.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.772+0000 7f4f84ff9640 1 -- 172.21.15.28:0/3827458945 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f4f7c0bc600 con 0x7f4f88103660 2024-09-15T21:55:48.774 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"btime":"2024-09-15T21:55:38:559165+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:38.559150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:55:48.775 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-09-15T21:55:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.774+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f700a32d0 msgr2=0x7f4f700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.774+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f700a32d0 0x7f4f700a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800bbe0 tx=0x7f4f7800b390 comp rx=0 tx=0).stop 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.774+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 msgr2=0x7f4f8819ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.774+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8819ab90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c006010 tx=0x7f4f7c0042f0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 shutdown_connections 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4f700a32d0 0x7f4f700a5790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f8810c170 0x7f4f8819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f88103660 0x7f4f8819ab90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 --2- 172.21.15.28:0/3827458945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f88102d40 0x7f4f8819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 >> 172.21.15.28:0/3827458945 conn(0x7f4f880fc7a0 msgr2=0x7f4f880fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 shutdown_connections 2024-09-15T21:55:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:48.775+0000 7f4f8e0a6640 1 -- 172.21.15.28:0/3827458945 wait complete. 2024-09-15T21:55:48.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:48 smithi028 ceph-mon[30969]: pgmap v313: 129 pgs: 129 active+clean; 612 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.4 KiB/s wr, 22 op/s 2024-09-15T21:55:48.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:48 smithi203 ceph-mon[35546]: pgmap v313: 129 pgs: 129 active+clean; 612 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.4 KiB/s wr, 22 op/s 2024-09-15T21:55:49.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:48 smithi174 ceph-mon[35244]: pgmap v313: 129 pgs: 129 active+clean; 612 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.4 KiB/s wr, 22 op/s 2024-09-15T21:55:49.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:55:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:49.846 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:49 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3827458945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:49.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:49 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3827458945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:50.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:49 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3827458945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:55:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 -- 172.21.15.28:0/136663476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 msgr2=0x7f8fe01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 --2- 172.21.15.28:0/136663476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe01067e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009900 tx=0x7f8fc802a410 comp rx=0 tx=0).stop 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 -- 172.21.15.28:0/136663476 shutdown_connections 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 --2- 172.21.15.28:0/136663476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0106f10 0x7f8fe010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 --2- 172.21.15.28:0/136663476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe01067e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 --2- 172.21.15.28:0/136663476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0101de0 0x7f8fe01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 -- 172.21.15.28:0/136663476 >> 172.21.15.28:0/136663476 conn(0x7f8fe00fd8c0 msgr2=0x7f8fe00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 -- 172.21.15.28:0/136663476 shutdown_connections 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.190+0000 7f8fe609a640 1 -- 172.21.15.28:0/136663476 wait complete. 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 Processor -- start 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 -- start start 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0101de0 0x7f8fe00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0106f10 0x7f8fe0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0111f30 con 0x7f8fe0102790 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0111db0 con 0x7f8fe0101de0 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.191+0000 7f8fe609a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01120b0 con 0x7f8fe0106f10 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe0074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35632/0 (socket says 172.21.15.28:35632) 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 -- 172.21.15.28:0/79986431 learned_addr learned my addr 172.21.15.28:0/79986431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdf7fe640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0101de0 0x7f8fe00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdffff640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0106f10 0x7f8fe0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 -- 172.21.15.28:0/79986431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0106f10 msgr2=0x7f8fe0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0106f10 0x7f8fe0074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 -- 172.21.15.28:0/79986431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0101de0 msgr2=0x7f8fe00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0101de0 0x7f8fe00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 -- 172.21.15.28:0/79986431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe01a6980 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdeffd640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe0074a30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8006010 tx=0x7f8fc80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdcff9640 1 -- 172.21.15.28:0/79986431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc803f030 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe01a6c10 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdcff9640 1 -- 172.21.15.28:0/79986431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc8003ce0 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fdcff9640 1 -- 172.21.15.28:0/79986431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8035070 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.192+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe01a70f0 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.193+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe01021c0 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.194+0000 7f8fdcff9640 1 -- 172.21.15.28:0/79986431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8fc8042030 con 0x7f8fe0102790 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.195+0000 7f8fdcff9640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8fcc0a32d0 0x7f8fcc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.195+0000 7f8fdf7fe640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8fcc0a32d0 0x7f8fcc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.195+0000 7f8fdcff9640 1 -- 172.21.15.28:0/79986431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f8fc80f0250 con 0x7f8fe0102790 2024-09-15T21:55:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.196+0000 7f8fdf7fe640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8fcc0a32d0 0x7f8fcc0a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000bbe0 tx=0x7f8fd000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.199+0000 7f8fdcff9640 1 -- 172.21.15.28:0/79986431 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8fc80bcc30 con 0x7f8fe0102790 2024-09-15T21:55:50.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.377+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8fe010a0f0 con 0x7f8fe0102790 2024-09-15T21:55:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.378+0000 7f8fdcff9640 1 -- 172.21.15.28:0/79986431 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f8fc80bc530 con 0x7f8fe0102790 2024-09-15T21:55:50.380 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:50.380 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:08.111056+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:55:50.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.380+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8fcc0a32d0 msgr2=0x7f8fcc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:50.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.380+0000 7f8fe609a640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8fcc0a32d0 0x7f8fcc0a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000bbe0 tx=0x7f8fd000b390 comp rx=0 tx=0).stop 2024-09-15T21:55:50.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.380+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 msgr2=0x7f8fe0074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:50.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.380+0000 7f8fe609a640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe0074a30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8006010 tx=0x7f8fc80042f0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 shutdown_connections 2024-09-15T21:55:50.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8fcc0a32d0 0x7f8fcc0a5790 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fe0106f10 0x7f8fe0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe0102790 0x7f8fe0074a30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 --2- 172.21.15.28:0/79986431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0101de0 0x7f8fe00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 >> 172.21.15.28:0/79986431 conn(0x7f8fe00fd8c0 msgr2=0x7f8fe00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 shutdown_connections 2024-09-15T21:55:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:50.381+0000 7f8fe609a640 1 -- 172.21.15.28:0/79986431 wait complete. 2024-09-15T21:55:50.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:50 smithi028 ceph-mon[30969]: pgmap v314: 129 pgs: 129 active+clean; 635 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-09-15T21:55:50.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:50 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/79986431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:50 smithi203 ceph-mon[35546]: pgmap v314: 129 pgs: 129 active+clean; 635 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-09-15T21:55:50.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:50 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/79986431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:51.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:50 smithi174 ceph-mon[35244]: pgmap v314: 129 pgs: 129 active+clean; 635 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-09-15T21:55:51.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:50 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/79986431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:51.069 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T21:55:51.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T21:55:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 -- 172.21.15.28:0/1213809916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 msgr2=0x7f0444106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 --2- 172.21.15.28:0/1213809916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f0444106e60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0440009930 tx=0x7f044002a420 comp rx=0 tx=0).stop 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 -- 172.21.15.28:0/1213809916 shutdown_connections 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 --2- 172.21.15.28:0/1213809916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444101420 0x7f044410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 --2- 172.21.15.28:0/1213809916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0444100a80 0x7f0444100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 --2- 172.21.15.28:0/1213809916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f0444106e60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 -- 172.21.15.28:0/1213809916 >> 172.21.15.28:0/1213809916 conn(0x7f04440fc7a0 msgr2=0x7f04440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 -- 172.21.15.28:0/1213809916 shutdown_connections 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.866+0000 7f044aee9640 1 -- 172.21.15.28:0/1213809916 wait complete. 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 Processor -- start 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 -- start start 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0444100a80 0x7f044419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444101420 0x7f044419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f044419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0444110e80 con 0x7f0444106a80 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0444110d00 con 0x7f0444100a80 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f044aee9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0444111000 con 0x7f0444101420 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f0449486640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f044419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f0449486640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f044419ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51450/0 (socket says 172.21.15.28:51450) 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f0449486640 1 -- 172.21.15.28:0/705719791 learned_addr learned my addr 172.21.15.28:0/705719791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f0448c85640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0444100a80 0x7f044419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.867+0000 7f043bfff640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444101420 0x7f044419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0449486640 1 -- 172.21.15.28:0/705719791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444101420 msgr2=0x7f044419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0449486640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444101420 0x7f044419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0449486640 1 -- 172.21.15.28:0/705719791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0444100a80 msgr2=0x7f044419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:51.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0449486640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0444100a80 0x7f044419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0449486640 1 -- 172.21.15.28:0/705719791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044419f610 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f043bfff640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444101420 0x7f044419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0449486640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f044419ef30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f043c00cad0 tx=0x7f043c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0439ffb640 1 -- 172.21.15.28:0/705719791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c00ae20 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0439ffb640 1 -- 172.21.15.28:0/705719791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f043c005020 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044419f900 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f0439ffb640 1 -- 172.21.15.28:0/705719791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c00f6f0 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.868+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04441a71e0 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.869+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0444100ee0 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.870+0000 7f0439ffb640 1 -- 172.21.15.28:0/705719791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f043c0051d0 con 0x7f0444106a80 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.871+0000 7f0439ffb640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f042c0a3350 0x7f042c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.871+0000 7f0439ffb640 1 -- 172.21.15.28:0/705719791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f043c0cff80 con 0x7f0444106a80 2024-09-15T21:55:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.871+0000 7f0448c85640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f042c0a3350 0x7f042c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.872+0000 7f0448c85640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f042c0a3350 0x7f042c0a5810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0440009900 tx=0x7f04400058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:51.875+0000 7f0439ffb640 1 -- 172.21.15.28:0/705719791 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f043c014030 con 0x7f0444106a80 2024-09-15T21:55:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.047+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f044406aed0 con 0x7f042c0a3350 2024-09-15T21:55:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.056+0000 7f0439ffb640 1 -- 172.21.15.28:0/705719791 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340393 (secure 0 0 0) 0x7f044406aed0 con 0x7f042c0a3350 2024-09-15T21:55:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:55:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f042c0a3350 msgr2=0x7f042c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f042c0a3350 0x7f042c0a5810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0440009900 tx=0x7f04400058d0 comp rx=0 tx=0).stop 2024-09-15T21:55:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 msgr2=0x7f044419ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f044419ef30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f043c00cad0 tx=0x7f043c00cf90 comp rx=0 tx=0).stop 2024-09-15T21:55:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 shutdown_connections 2024-09-15T21:55:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f042c0a3350 0x7f042c0a5810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0444106a80 0x7f044419ef30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444101420 0x7f044419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 --2- 172.21.15.28:0/705719791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0444100a80 0x7f044419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 >> 172.21.15.28:0/705719791 conn(0x7f04440fc7a0 msgr2=0x7f044406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 shutdown_connections 2024-09-15T21:55:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:52.060+0000 7f044aee9640 1 -- 172.21.15.28:0/705719791 wait complete. 2024-09-15T21:55:52.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:52 smithi028 ceph-mon[30969]: pgmap v315: 129 pgs: 129 active+clean; 635 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 18 op/s 2024-09-15T21:55:52.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:52 smithi028 ceph-mon[30969]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:55:52.738 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-15T21:55:51.550489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650223,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":285,"num_read_kb":259,"num_write":427,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363012,"kb_used_data":13764,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578812,"statfs":{"total":1151940427776,"available":1151568703488,"internally_reserved":0,"allocated":14094336,"data_stored":7756503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51051,"num_objects":27,"num_object_clones":0,"num_object_copies":83,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":135,"num_read_kb":124,"num_write":88,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003665"},"pg_stats":[{"pgid":"2.1f","version":"88'13","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.583446+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:55:48.583446+0000","last_peered":"2024-09-15T21:55:48.583446+0000","last_clean":"2024-09-15T21:55:48.583446+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:55:48.583446+0000","last_undegraded":"2024-09-15T21:55:48.583446+0000","last_fullsized":"2024-09-15T21:55:48.583446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627418+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:55:08.627418+0000","last_peered":"2024-09-15T21:55:08.627418+0000","last_clean":"2024-09-15T21:55:08.627418+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:55:08.627418+0000","last_undegraded":"2024-09-15T21:55:08.627418+0000","last_fullsized":"2024-09-15T21:55:08.627418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803661+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:55:08.803661+0000","last_peered":"2024-09-15T21:55:08.803661+0000","last_clean":"2024-09-15T21:55:08.803661+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:55:08.803661+0000","last_undegraded":"2024-09-15T21:55:08.803661+0000","last_fullsized":"2024-09-15T21:55:08.803661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627711+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:55:08.627711+0000","last_peered":"2024-09-15T21:55:08.627711+0000","last_clean":"2024-09-15T21:55:08.627711+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:55:08.627711+0000","last_undegraded":"2024-09-15T21:55:08.627711+0000","last_fullsized":"2024-09-15T21:55:08.627711+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803860+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:55:08.803860+0000","last_peered":"2024-09-15T21:55:08.803860+0000","last_clean":"2024-09-15T21:55:08.803860+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:55:08.803860+0000","last_undegraded":"2024-09-15T21:55:08.803860+0000","last_fullsized":"2024-09-15T21:55:08.803860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803735+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:55:08.803735+0000","last_peered":"2024-09-15T21:55:08.803735+0000","last_clean":"2024-09-15T21:55:08.803735+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:55:08.803735+0000","last_undegraded":"2024-09-15T21:55:08.803735+0000","last_fullsized":"2024-09-15T21:55:08.803735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:38.566678+0000","last_change":"2024-09-15T21:54:56.346287+0000","last_active":"2024-09-15T21:55:38.566678+0000","last_peered":"2024-09-15T21:55:38.566678+0000","last_clean":"2024-09-15T21:55:38.566678+0000","last_became_active":"2024-09-15T21:54:56.345880+0000","last_became_peered":"2024-09-15T21:54:56.345880+0000","last_unstale":"2024-09-15T21:55:38.566678+0000","last_undegraded":"2024-09-15T21:55:38.566678+0000","last_fullsized":"2024-09-15T21:55:38.566678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:30.899587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803836+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:55:08.803836+0000","last_peered":"2024-09-15T21:55:08.803836+0000","last_clean":"2024-09-15T21:55:08.803836+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:55:08.803836+0000","last_undegraded":"2024-09-15T21:55:08.803836+0000","last_fullsized":"2024-09-15T21:55:08.803836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804964+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:55:08.804964+0000","last_peered":"2024-09-15T21:55:08.804964+0000","last_clean":"2024-09-15T21:55:08.804964+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:55:08.804964+0000","last_undegraded":"2024-09-15T21:55:08.804964+0000","last_fullsized":"2024-09-15T21:55:08.804964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561353+0000","last_change":"2024-09-15T21:54:56.343604+0000","last_active":"2024-09-15T21:55:37.561353+0000","last_peered":"2024-09-15T21:55:37.561353+0000","last_clean":"2024-09-15T21:55:37.561353+0000","last_became_active":"2024-09-15T21:54:56.342956+0000","last_became_peered":"2024-09-15T21:54:56.342956+0000","last_unstale":"2024-09-15T21:55:37.561353+0000","last_undegraded":"2024-09-15T21:55:37.561353+0000","last_fullsized":"2024-09-15T21:55:37.561353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:43.299751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805132+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:55:08.805132+0000","last_peered":"2024-09-15T21:55:08.805132+0000","last_clean":"2024-09-15T21:55:08.805132+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:55:08.805132+0000","last_undegraded":"2024-09-15T21:55:08.805132+0000","last_fullsized":"2024-09-15T21:55:08.805132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804038+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:55:08.804038+0000","last_peered":"2024-09-15T21:55:08.804038+0000","last_clean":"2024-09-15T21:55:08.804038+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:55:08.804038+0000","last_undegraded":"2024-09-15T21:55:08.804038+0000","last_fullsized":"2024-09-15T21:55:08.804038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804372+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:55:08.804372+0000","last_peered":"2024-09-15T21:55:08.804372+0000","last_clean":"2024-09-15T21:55:08.804372+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:55:08.804372+0000","last_undegraded":"2024-09-15T21:55:08.804372+0000","last_fullsized":"2024-09-15T21:55:08.804372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:34.537983+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:55:34.537983+0000","last_peered":"2024-09-15T21:55:34.537983+0000","last_clean":"2024-09-15T21:55:34.537983+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:55:34.537983+0000","last_undegraded":"2024-09-15T21:55:34.537983+0000","last_fullsized":"2024-09-15T21:55:34.537983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805248+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:55:08.805248+0000","last_peered":"2024-09-15T21:55:08.805248+0000","last_clean":"2024-09-15T21:55:08.805248+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:55:08.805248+0000","last_undegraded":"2024-09-15T21:55:08.805248+0000","last_fullsized":"2024-09-15T21:55:08.805248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627523+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:55:08.627523+0000","last_peered":"2024-09-15T21:55:08.627523+0000","last_clean":"2024-09-15T21:55:08.627523+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:55:08.627523+0000","last_undegraded":"2024-09-15T21:55:08.627523+0000","last_fullsized":"2024-09-15T21:55:08.627523+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:36.556866+0000","last_change":"2024-09-15T21:54:56.343335+0000","last_active":"2024-09-15T21:55:36.556866+0000","last_peered":"2024-09-15T21:55:36.556866+0000","last_clean":"2024-09-15T21:55:36.556866+0000","last_became_active":"2024-09-15T21:54:56.343011+0000","last_became_peered":"2024-09-15T21:54:56.343011+0000","last_unstale":"2024-09-15T21:55:36.556866+0000","last_undegraded":"2024-09-15T21:55:36.556866+0000","last_fullsized":"2024-09-15T21:55:36.556866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:57.013184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804454+0000","last_change":"2024-09-15T21:55:05.471517+0000","last_active":"2024-09-15T21:55:08.804454+0000","last_peered":"2024-09-15T21:55:08.804454+0000","last_clean":"2024-09-15T21:55:08.804454+0000","last_became_active":"2024-09-15T21:55:05.471076+0000","last_became_peered":"2024-09-15T21:55:05.471076+0000","last_unstale":"2024-09-15T21:55:08.804454+0000","last_undegraded":"2024-09-15T21:55:08.804454+0000","last_fullsized":"2024-09-15T21:55:08.804454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:50.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465792+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:55:08.465792+0000","last_peered":"2024-09-15T21:55:08.465792+0000","last_clean":"2024-09-15T21:55:08.465792+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:55:08.465792+0000","last_undegraded":"2024-09-15T21:55:08.465792+0000","last_fullsized":"2024-09-15T21:55:08.465792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.560825+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:55:37.560825+0000","last_peered":"2024-09-15T21:55:37.560825+0000","last_clean":"2024-09-15T21:55:37.560825+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:55:37.560825+0000","last_undegraded":"2024-09-15T21:55:37.560825+0000","last_fullsized":"2024-09-15T21:55:37.560825+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464880+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:55:08.464880+0000","last_peered":"2024-09-15T21:55:08.464880+0000","last_clean":"2024-09-15T21:55:08.464880+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:55:08.464880+0000","last_undegraded":"2024-09-15T21:55:08.464880+0000","last_fullsized":"2024-09-15T21:55:08.464880+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803403+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:55:08.803403+0000","last_peered":"2024-09-15T21:55:08.803403+0000","last_clean":"2024-09-15T21:55:08.803403+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:55:08.803403+0000","last_undegraded":"2024-09-15T21:55:08.803403+0000","last_fullsized":"2024-09-15T21:55:08.803403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:34.537395+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:55:34.537395+0000","last_peered":"2024-09-15T21:55:34.537395+0000","last_clean":"2024-09-15T21:55:34.537395+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:55:34.537395+0000","last_undegraded":"2024-09-15T21:55:34.537395+0000","last_fullsized":"2024-09-15T21:55:34.537395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805291+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:55:08.805291+0000","last_peered":"2024-09-15T21:55:08.805291+0000","last_clean":"2024-09-15T21:55:08.805291+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:55:08.805291+0000","last_undegraded":"2024-09-15T21:55:08.805291+0000","last_fullsized":"2024-09-15T21:55:08.805291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803758+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:55:08.803758+0000","last_peered":"2024-09-15T21:55:08.803758+0000","last_clean":"2024-09-15T21:55:08.803758+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:55:08.803758+0000","last_undegraded":"2024-09-15T21:55:08.803758+0000","last_fullsized":"2024-09-15T21:55:08.803758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.560871+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:55:37.560871+0000","last_peered":"2024-09-15T21:55:37.560871+0000","last_clean":"2024-09-15T21:55:37.560871+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:55:37.560871+0000","last_undegraded":"2024-09-15T21:55:37.560871+0000","last_fullsized":"2024-09-15T21:55:37.560871+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803987+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:55:08.803987+0000","last_peered":"2024-09-15T21:55:08.803987+0000","last_clean":"2024-09-15T21:55:08.803987+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:55:08.803987+0000","last_undegraded":"2024-09-15T21:55:08.803987+0000","last_fullsized":"2024-09-15T21:55:08.803987+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627805+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:55:08.627805+0000","last_peered":"2024-09-15T21:55:08.627805+0000","last_clean":"2024-09-15T21:55:08.627805+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:55:08.627805+0000","last_undegraded":"2024-09-15T21:55:08.627805+0000","last_fullsized":"2024-09-15T21:55:08.627805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'15","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.438379+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:55:48.438379+0000","last_peered":"2024-09-15T21:55:48.438379+0000","last_clean":"2024-09-15T21:55:48.438379+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:55:48.438379+0000","last_undegraded":"2024-09-15T21:55:48.438379+0000","last_fullsized":"2024-09-15T21:55:48.438379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627135+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:55:08.627135+0000","last_peered":"2024-09-15T21:55:08.627135+0000","last_clean":"2024-09-15T21:55:08.627135+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:55:08.627135+0000","last_undegraded":"2024-09-15T21:55:08.627135+0000","last_fullsized":"2024-09-15T21:55:08.627135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595166+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:55:09.595166+0000","last_peered":"2024-09-15T21:55:09.595166+0000","last_clean":"2024-09-15T21:55:09.595166+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:55:09.595166+0000","last_undegraded":"2024-09-15T21:55:09.595166+0000","last_fullsized":"2024-09-15T21:55:09.595166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.562416+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:55:37.562416+0000","last_peered":"2024-09-15T21:55:37.562416+0000","last_clean":"2024-09-15T21:55:37.562416+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:55:37.562416+0000","last_undegraded":"2024-09-15T21:55:37.562416+0000","last_fullsized":"2024-09-15T21:55:37.562416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465946+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:55:08.465946+0000","last_peered":"2024-09-15T21:55:08.465946+0000","last_clean":"2024-09-15T21:55:08.465946+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:55:08.465946+0000","last_undegraded":"2024-09-15T21:55:08.465946+0000","last_fullsized":"2024-09-15T21:55:08.465946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465123+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:55:08.465123+0000","last_peered":"2024-09-15T21:55:08.465123+0000","last_clean":"2024-09-15T21:55:08.465123+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:55:08.465123+0000","last_undegraded":"2024-09-15T21:55:08.465123+0000","last_fullsized":"2024-09-15T21:55:08.465123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464769+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:55:08.464769+0000","last_peered":"2024-09-15T21:55:08.464769+0000","last_clean":"2024-09-15T21:55:08.464769+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:55:08.464769+0000","last_undegraded":"2024-09-15T21:55:08.464769+0000","last_fullsized":"2024-09-15T21:55:08.464769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803549+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:55:08.803549+0000","last_peered":"2024-09-15T21:55:08.803549+0000","last_clean":"2024-09-15T21:55:08.803549+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:55:08.803549+0000","last_undegraded":"2024-09-15T21:55:08.803549+0000","last_fullsized":"2024-09-15T21:55:08.803549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465116+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:55:08.465116+0000","last_peered":"2024-09-15T21:55:08.465116+0000","last_clean":"2024-09-15T21:55:08.465116+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:55:08.465116+0000","last_undegraded":"2024-09-15T21:55:08.465116+0000","last_fullsized":"2024-09-15T21:55:08.465116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:46.581144+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:55:46.581144+0000","last_peered":"2024-09-15T21:55:46.581144+0000","last_clean":"2024-09-15T21:55:46.581144+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:55:46.581144+0000","last_undegraded":"2024-09-15T21:55:46.581144+0000","last_fullsized":"2024-09-15T21:55:46.581144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595051+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:55:09.595051+0000","last_peered":"2024-09-15T21:55:09.595051+0000","last_clean":"2024-09-15T21:55:09.595051+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:55:09.595051+0000","last_undegraded":"2024-09-15T21:55:09.595051+0000","last_fullsized":"2024-09-15T21:55:09.595051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803791+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:55:08.803791+0000","last_peered":"2024-09-15T21:55:08.803791+0000","last_clean":"2024-09-15T21:55:08.803791+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:55:08.803791+0000","last_undegraded":"2024-09-15T21:55:08.803791+0000","last_fullsized":"2024-09-15T21:55:08.803791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:45.582714+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:55:45.582714+0000","last_peered":"2024-09-15T21:55:45.582714+0000","last_clean":"2024-09-15T21:55:45.582714+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:55:45.582714+0000","last_undegraded":"2024-09-15T21:55:45.582714+0000","last_fullsized":"2024-09-15T21:55:45.582714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595204+0000","last_change":"2024-09-15T21:55:05.496449+0000","last_active":"2024-09-15T21:55:09.595204+0000","last_peered":"2024-09-15T21:55:09.595204+0000","last_clean":"2024-09-15T21:55:09.595204+0000","last_became_active":"2024-09-15T21:55:05.496018+0000","last_became_peered":"2024-09-15T21:55:05.496018+0000","last_unstale":"2024-09-15T21:55:09.595204+0000","last_undegraded":"2024-09-15T21:55:09.595204+0000","last_fullsized":"2024-09-15T21:55:09.595204+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:18:44.250045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594596+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:55:09.594596+0000","last_peered":"2024-09-15T21:55:09.594596+0000","last_clean":"2024-09-15T21:55:09.594596+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:55:09.594596+0000","last_undegraded":"2024-09-15T21:55:09.594596+0000","last_fullsized":"2024-09-15T21:55:09.594596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:35.542432+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:55:35.542432+0000","last_peered":"2024-09-15T21:55:35.542432+0000","last_clean":"2024-09-15T21:55:35.542432+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:55:35.542432+0000","last_undegraded":"2024-09-15T21:55:35.542432+0000","last_fullsized":"2024-09-15T21:55:35.542432+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805007+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:55:08.805007+0000","last_peered":"2024-09-15T21:55:08.805007+0000","last_clean":"2024-09-15T21:55:08.805007+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:55:08.805007+0000","last_undegraded":"2024-09-15T21:55:08.805007+0000","last_fullsized":"2024-09-15T21:55:08.805007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627400+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:55:08.627400+0000","last_peered":"2024-09-15T21:55:08.627400+0000","last_clean":"2024-09-15T21:55:08.627400+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:55:08.627400+0000","last_undegraded":"2024-09-15T21:55:08.627400+0000","last_fullsized":"2024-09-15T21:55:08.627400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:38.567038+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:55:38.567038+0000","last_peered":"2024-09-15T21:55:38.567038+0000","last_clean":"2024-09-15T21:55:38.567038+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:55:38.567038+0000","last_undegraded":"2024-09-15T21:55:38.567038+0000","last_fullsized":"2024-09-15T21:55:38.567038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804812+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:55:08.804812+0000","last_peered":"2024-09-15T21:55:08.804812+0000","last_clean":"2024-09-15T21:55:08.804812+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:55:08.804812+0000","last_undegraded":"2024-09-15T21:55:08.804812+0000","last_fullsized":"2024-09-15T21:55:08.804812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804670+0000","last_change":"2024-09-15T21:55:05.468725+0000","last_active":"2024-09-15T21:55:08.804670+0000","last_peered":"2024-09-15T21:55:08.804670+0000","last_clean":"2024-09-15T21:55:08.804670+0000","last_became_active":"2024-09-15T21:55:05.468356+0000","last_became_peered":"2024-09-15T21:55:05.468356+0000","last_unstale":"2024-09-15T21:55:08.804670+0000","last_undegraded":"2024-09-15T21:55:08.804670+0000","last_fullsized":"2024-09-15T21:55:08.804670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:04:49.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804855+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:55:08.804855+0000","last_peered":"2024-09-15T21:55:08.804855+0000","last_clean":"2024-09-15T21:55:08.804855+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:55:08.804855+0000","last_undegraded":"2024-09-15T21:55:08.804855+0000","last_fullsized":"2024-09-15T21:55:08.804855+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'6","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:46.582866+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:55:46.582866+0000","last_peered":"2024-09-15T21:55:46.582866+0000","last_clean":"2024-09-15T21:55:46.582866+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:55:46.582866+0000","last_undegraded":"2024-09-15T21:55:46.582866+0000","last_fullsized":"2024-09-15T21:55:46.582866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803813+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:55:08.803813+0000","last_peered":"2024-09-15T21:55:08.803813+0000","last_clean":"2024-09-15T21:55:08.803813+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:55:08.803813+0000","last_undegraded":"2024-09-15T21:55:08.803813+0000","last_fullsized":"2024-09-15T21:55:08.803813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804052+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:55:08.804052+0000","last_peered":"2024-09-15T21:55:08.804052+0000","last_clean":"2024-09-15T21:55:08.804052+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:55:08.804052+0000","last_undegraded":"2024-09-15T21:55:08.804052+0000","last_fullsized":"2024-09-15T21:55:08.804052+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561599+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:55:37.561599+0000","last_peered":"2024-09-15T21:55:37.561599+0000","last_clean":"2024-09-15T21:55:37.561599+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:55:37.561599+0000","last_undegraded":"2024-09-15T21:55:37.561599+0000","last_fullsized":"2024-09-15T21:55:37.561599+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803554+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:55:08.803554+0000","last_peered":"2024-09-15T21:55:08.803554+0000","last_clean":"2024-09-15T21:55:08.803554+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:55:08.803554+0000","last_undegraded":"2024-09-15T21:55:08.803554+0000","last_fullsized":"2024-09-15T21:55:08.803554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804410+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:55:08.804410+0000","last_peered":"2024-09-15T21:55:08.804410+0000","last_clean":"2024-09-15T21:55:08.804410+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:55:08.804410+0000","last_undegraded":"2024-09-15T21:55:08.804410+0000","last_fullsized":"2024-09-15T21:55:08.804410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561644+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:55:37.561644+0000","last_peered":"2024-09-15T21:55:37.561644+0000","last_clean":"2024-09-15T21:55:37.561644+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:55:37.561644+0000","last_undegraded":"2024-09-15T21:55:37.561644+0000","last_fullsized":"2024-09-15T21:55:37.561644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803480+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:55:08.803480+0000","last_peered":"2024-09-15T21:55:08.803480+0000","last_clean":"2024-09-15T21:55:08.803480+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:55:08.803480+0000","last_undegraded":"2024-09-15T21:55:08.803480+0000","last_fullsized":"2024-09-15T21:55:08.803480+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465949+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:55:08.465949+0000","last_peered":"2024-09-15T21:55:08.465949+0000","last_clean":"2024-09-15T21:55:08.465949+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:55:08.465949+0000","last_undegraded":"2024-09-15T21:55:08.465949+0000","last_fullsized":"2024-09-15T21:55:08.465949+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.539254+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:55:37.539254+0000","last_peered":"2024-09-15T21:55:37.539254+0000","last_clean":"2024-09-15T21:55:37.539254+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:55:37.539254+0000","last_undegraded":"2024-09-15T21:55:37.539254+0000","last_fullsized":"2024-09-15T21:55:37.539254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803913+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:55:08.803913+0000","last_peered":"2024-09-15T21:55:08.803913+0000","last_clean":"2024-09-15T21:55:08.803913+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:55:08.803913+0000","last_undegraded":"2024-09-15T21:55:08.803913+0000","last_fullsized":"2024-09-15T21:55:08.803913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804182+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:55:08.804182+0000","last_peered":"2024-09-15T21:55:08.804182+0000","last_clean":"2024-09-15T21:55:08.804182+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:55:08.804182+0000","last_undegraded":"2024-09-15T21:55:08.804182+0000","last_fullsized":"2024-09-15T21:55:08.804182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561639+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:55:37.561639+0000","last_peered":"2024-09-15T21:55:37.561639+0000","last_clean":"2024-09-15T21:55:37.561639+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:55:37.561639+0000","last_undegraded":"2024-09-15T21:55:37.561639+0000","last_fullsized":"2024-09-15T21:55:37.561639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595242+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:55:09.595242+0000","last_peered":"2024-09-15T21:55:09.595242+0000","last_clean":"2024-09-15T21:55:09.595242+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:55:09.595242+0000","last_undegraded":"2024-09-15T21:55:09.595242+0000","last_fullsized":"2024-09-15T21:55:09.595242+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804684+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:55:08.804684+0000","last_peered":"2024-09-15T21:55:08.804684+0000","last_clean":"2024-09-15T21:55:08.804684+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:55:08.804684+0000","last_undegraded":"2024-09-15T21:55:08.804684+0000","last_fullsized":"2024-09-15T21:55:08.804684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.587319+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:55:48.587319+0000","last_peered":"2024-09-15T21:55:48.587319+0000","last_clean":"2024-09-15T21:55:48.587319+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:55:48.587319+0000","last_undegraded":"2024-09-15T21:55:48.587319+0000","last_fullsized":"2024-09-15T21:55:48.587319+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464945+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:55:08.464945+0000","last_peered":"2024-09-15T21:55:08.464945+0000","last_clean":"2024-09-15T21:55:08.464945+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:55:08.464945+0000","last_undegraded":"2024-09-15T21:55:08.464945+0000","last_fullsized":"2024-09-15T21:55:08.464945+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804532+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:55:08.804532+0000","last_peered":"2024-09-15T21:55:08.804532+0000","last_clean":"2024-09-15T21:55:08.804532+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:55:08.804532+0000","last_undegraded":"2024-09-15T21:55:08.804532+0000","last_fullsized":"2024-09-15T21:55:08.804532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561862+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:55:37.561862+0000","last_peered":"2024-09-15T21:55:37.561862+0000","last_clean":"2024-09-15T21:55:37.561862+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:55:37.561862+0000","last_undegraded":"2024-09-15T21:55:37.561862+0000","last_fullsized":"2024-09-15T21:55:37.561862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465823+0000","last_change":"2024-09-15T21:55:05.469602+0000","last_active":"2024-09-15T21:55:08.465823+0000","last_peered":"2024-09-15T21:55:08.465823+0000","last_clean":"2024-09-15T21:55:08.465823+0000","last_became_active":"2024-09-15T21:55:05.468858+0000","last_became_peered":"2024-09-15T21:55:05.468858+0000","last_unstale":"2024-09-15T21:55:08.465823+0000","last_undegraded":"2024-09-15T21:55:08.465823+0000","last_fullsized":"2024-09-15T21:55:08.465823+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:58:09.823131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594797+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:55:09.594797+0000","last_peered":"2024-09-15T21:55:09.594797+0000","last_clean":"2024-09-15T21:55:09.594797+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:55:09.594797+0000","last_undegraded":"2024-09-15T21:55:09.594797+0000","last_fullsized":"2024-09-15T21:55:09.594797+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:38.566755+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:55:38.566755+0000","last_peered":"2024-09-15T21:55:38.566755+0000","last_clean":"2024-09-15T21:55:38.566755+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:55:38.566755+0000","last_undegraded":"2024-09-15T21:55:38.566755+0000","last_fullsized":"2024-09-15T21:55:38.566755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805112+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:55:08.805112+0000","last_peered":"2024-09-15T21:55:08.805112+0000","last_clean":"2024-09-15T21:55:08.805112+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:55:08.805112+0000","last_undegraded":"2024-09-15T21:55:08.805112+0000","last_fullsized":"2024-09-15T21:55:08.805112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804208+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:55:08.804208+0000","last_peered":"2024-09-15T21:55:08.804208+0000","last_clean":"2024-09-15T21:55:08.804208+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:55:08.804208+0000","last_undegraded":"2024-09-15T21:55:08.804208+0000","last_fullsized":"2024-09-15T21:55:08.804208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.589953+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:55:48.589953+0000","last_peered":"2024-09-15T21:55:48.589953+0000","last_clean":"2024-09-15T21:55:48.589953+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:55:48.589953+0000","last_undegraded":"2024-09-15T21:55:48.589953+0000","last_fullsized":"2024-09-15T21:55:48.589953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52413,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":65,"num_write":33,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464513+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:55:08.464513+0000","last_peered":"2024-09-15T21:55:08.464513+0000","last_clean":"2024-09-15T21:55:08.464513+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:55:08.464513+0000","last_undegraded":"2024-09-15T21:55:08.464513+0000","last_fullsized":"2024-09-15T21:55:08.464513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805179+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:55:08.805179+0000","last_peered":"2024-09-15T21:55:08.805179+0000","last_clean":"2024-09-15T21:55:08.805179+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:55:08.805179+0000","last_undegraded":"2024-09-15T21:55:08.805179+0000","last_fullsized":"2024-09-15T21:55:08.805179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:36.555627+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:55:36.555627+0000","last_peered":"2024-09-15T21:55:36.555627+0000","last_clean":"2024-09-15T21:55:36.555627+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:55:36.555627+0000","last_undegraded":"2024-09-15T21:55:36.555627+0000","last_fullsized":"2024-09-15T21:55:36.555627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804764+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:55:08.804764+0000","last_peered":"2024-09-15T21:55:08.804764+0000","last_clean":"2024-09-15T21:55:08.804764+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:55:08.804764+0000","last_undegraded":"2024-09-15T21:55:08.804764+0000","last_fullsized":"2024-09-15T21:55:08.804764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803890+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:55:08.803890+0000","last_peered":"2024-09-15T21:55:08.803890+0000","last_clean":"2024-09-15T21:55:08.803890+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:55:08.803890+0000","last_undegraded":"2024-09-15T21:55:08.803890+0000","last_fullsized":"2024-09-15T21:55:08.803890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561367+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:55:37.561367+0000","last_peered":"2024-09-15T21:55:37.561367+0000","last_clean":"2024-09-15T21:55:37.561367+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:55:37.561367+0000","last_undegraded":"2024-09-15T21:55:37.561367+0000","last_fullsized":"2024-09-15T21:55:37.561367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464287+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:55:08.464287+0000","last_peered":"2024-09-15T21:55:08.464287+0000","last_clean":"2024-09-15T21:55:08.464287+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:55:08.464287+0000","last_undegraded":"2024-09-15T21:55:08.464287+0000","last_fullsized":"2024-09-15T21:55:08.464287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803824+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:55:08.803824+0000","last_peered":"2024-09-15T21:55:08.803824+0000","last_clean":"2024-09-15T21:55:08.803824+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:55:08.803824+0000","last_undegraded":"2024-09-15T21:55:08.803824+0000","last_fullsized":"2024-09-15T21:55:08.803824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.584791+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:55:48.584791+0000","last_peered":"2024-09-15T21:55:48.584791+0000","last_clean":"2024-09-15T21:55:48.584791+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:55:48.584791+0000","last_undegraded":"2024-09-15T21:55:48.584791+0000","last_fullsized":"2024-09-15T21:55:48.584791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627307+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:55:08.627307+0000","last_peered":"2024-09-15T21:55:08.627307+0000","last_clean":"2024-09-15T21:55:08.627307+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:55:08.627307+0000","last_undegraded":"2024-09-15T21:55:08.627307+0000","last_fullsized":"2024-09-15T21:55:08.627307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803692+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:55:08.803692+0000","last_peered":"2024-09-15T21:55:08.803692+0000","last_clean":"2024-09-15T21:55:08.803692+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:55:08.803692+0000","last_undegraded":"2024-09-15T21:55:08.803692+0000","last_fullsized":"2024-09-15T21:55:08.803692+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464256+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:55:08.464256+0000","last_peered":"2024-09-15T21:55:08.464256+0000","last_clean":"2024-09-15T21:55:08.464256+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:55:08.464256+0000","last_undegraded":"2024-09-15T21:55:08.464256+0000","last_fullsized":"2024-09-15T21:55:08.464256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561779+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:55:37.561779+0000","last_peered":"2024-09-15T21:55:37.561779+0000","last_clean":"2024-09-15T21:55:37.561779+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:55:37.561779+0000","last_undegraded":"2024-09-15T21:55:37.561779+0000","last_fullsized":"2024-09-15T21:55:37.561779+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466096+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:55:08.466096+0000","last_peered":"2024-09-15T21:55:08.466096+0000","last_clean":"2024-09-15T21:55:08.466096+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:55:08.466096+0000","last_undegraded":"2024-09-15T21:55:08.466096+0000","last_fullsized":"2024-09-15T21:55:08.466096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627287+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:55:08.627287+0000","last_peered":"2024-09-15T21:55:08.627287+0000","last_clean":"2024-09-15T21:55:08.627287+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:55:08.627287+0000","last_undegraded":"2024-09-15T21:55:08.627287+0000","last_fullsized":"2024-09-15T21:55:08.627287+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:35.543528+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:55:35.543528+0000","last_peered":"2024-09-15T21:55:35.543528+0000","last_clean":"2024-09-15T21:55:35.543528+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:55:35.543528+0000","last_undegraded":"2024-09-15T21:55:35.543528+0000","last_fullsized":"2024-09-15T21:55:35.543528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:36.556668+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:55:36.556668+0000","last_peered":"2024-09-15T21:55:36.556668+0000","last_clean":"2024-09-15T21:55:36.556668+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:55:36.556668+0000","last_undegraded":"2024-09-15T21:55:36.556668+0000","last_fullsized":"2024-09-15T21:55:36.556668+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804632+0000","last_change":"2024-09-15T21:54:58.444763+0000","last_active":"2024-09-15T21:55:08.804632+0000","last_peered":"2024-09-15T21:55:08.804632+0000","last_clean":"2024-09-15T21:55:08.804632+0000","last_became_active":"2024-09-15T21:54:58.443771+0000","last_became_peered":"2024-09-15T21:54:58.443771+0000","last_unstale":"2024-09-15T21:55:08.804632+0000","last_undegraded":"2024-09-15T21:55:08.804632+0000","last_fullsized":"2024-09-15T21:55:08.804632+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:49:25.519374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804183+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:55:08.804183+0000","last_peered":"2024-09-15T21:55:08.804183+0000","last_clean":"2024-09-15T21:55:08.804183+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:55:08.804183+0000","last_undegraded":"2024-09-15T21:55:08.804183+0000","last_fullsized":"2024-09-15T21:55:08.804183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:49.586838+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:55:49.586838+0000","last_peered":"2024-09-15T21:55:49.586838+0000","last_clean":"2024-09-15T21:55:49.586838+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:55:49.586838+0000","last_undegraded":"2024-09-15T21:55:49.586838+0000","last_fullsized":"2024-09-15T21:55:49.586838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803977+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:55:08.803977+0000","last_peered":"2024-09-15T21:55:08.803977+0000","last_clean":"2024-09-15T21:55:08.803977+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:55:08.803977+0000","last_undegraded":"2024-09-15T21:55:08.803977+0000","last_fullsized":"2024-09-15T21:55:08.803977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804801+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:55:08.804801+0000","last_peered":"2024-09-15T21:55:08.804801+0000","last_clean":"2024-09-15T21:55:08.804801+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:55:08.804801+0000","last_undegraded":"2024-09-15T21:55:08.804801+0000","last_fullsized":"2024-09-15T21:55:08.804801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465700+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:55:08.465700+0000","last_peered":"2024-09-15T21:55:08.465700+0000","last_clean":"2024-09-15T21:55:08.465700+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:55:08.465700+0000","last_undegraded":"2024-09-15T21:55:08.465700+0000","last_fullsized":"2024-09-15T21:55:08.465700+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804058+0000","last_change":"2024-09-15T21:54:58.445335+0000","last_active":"2024-09-15T21:55:08.804058+0000","last_peered":"2024-09-15T21:55:08.804058+0000","last_clean":"2024-09-15T21:55:08.804058+0000","last_became_active":"2024-09-15T21:54:58.444947+0000","last_became_peered":"2024-09-15T21:54:58.444947+0000","last_unstale":"2024-09-15T21:55:08.804058+0000","last_undegraded":"2024-09-15T21:55:08.804058+0000","last_fullsized":"2024-09-15T21:55:08.804058+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:12:55.939425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804355+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:55:08.804355+0000","last_peered":"2024-09-15T21:55:08.804355+0000","last_clean":"2024-09-15T21:55:08.804355+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:55:08.804355+0000","last_undegraded":"2024-09-15T21:55:08.804355+0000","last_fullsized":"2024-09-15T21:55:08.804355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803417+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:55:08.803417+0000","last_peered":"2024-09-15T21:55:08.803417+0000","last_clean":"2024-09-15T21:55:08.803417+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:55:08.803417+0000","last_undegraded":"2024-09-15T21:55:08.803417+0000","last_fullsized":"2024-09-15T21:55:08.803417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595101+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:55:09.595101+0000","last_peered":"2024-09-15T21:55:09.595101+0000","last_clean":"2024-09-15T21:55:09.595101+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:55:09.595101+0000","last_undegraded":"2024-09-15T21:55:09.595101+0000","last_fullsized":"2024-09-15T21:55:09.595101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466034+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:55:08.466034+0000","last_peered":"2024-09-15T21:55:08.466034+0000","last_clean":"2024-09-15T21:55:08.466034+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:55:08.466034+0000","last_undegraded":"2024-09-15T21:55:08.466034+0000","last_fullsized":"2024-09-15T21:55:08.466034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804603+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:55:08.804603+0000","last_peered":"2024-09-15T21:55:08.804603+0000","last_clean":"2024-09-15T21:55:08.804603+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:55:08.804603+0000","last_undegraded":"2024-09-15T21:55:08.804603+0000","last_fullsized":"2024-09-15T21:55:08.804603+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627638+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:55:08.627638+0000","last_peered":"2024-09-15T21:55:08.627638+0000","last_clean":"2024-09-15T21:55:08.627638+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:55:08.627638+0000","last_undegraded":"2024-09-15T21:55:08.627638+0000","last_fullsized":"2024-09-15T21:55:08.627638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804801+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:55:08.804801+0000","last_peered":"2024-09-15T21:55:08.804801+0000","last_clean":"2024-09-15T21:55:08.804801+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:55:08.804801+0000","last_undegraded":"2024-09-15T21:55:08.804801+0000","last_fullsized":"2024-09-15T21:55:08.804801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627282+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:55:08.627282+0000","last_peered":"2024-09-15T21:55:08.627282+0000","last_clean":"2024-09-15T21:55:08.627282+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:55:08.627282+0000","last_undegraded":"2024-09-15T21:55:08.627282+0000","last_fullsized":"2024-09-15T21:55:08.627282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804683+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:55:08.804683+0000","last_peered":"2024-09-15T21:55:08.804683+0000","last_clean":"2024-09-15T21:55:08.804683+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:55:08.804683+0000","last_undegraded":"2024-09-15T21:55:08.804683+0000","last_fullsized":"2024-09-15T21:55:08.804683+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594882+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:55:09.594882+0000","last_peered":"2024-09-15T21:55:09.594882+0000","last_clean":"2024-09-15T21:55:09.594882+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:55:09.594882+0000","last_undegraded":"2024-09-15T21:55:09.594882+0000","last_fullsized":"2024-09-15T21:55:09.594882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465681+0000","last_change":"2024-09-15T21:54:58.454311+0000","last_active":"2024-09-15T21:55:08.465681+0000","last_peered":"2024-09-15T21:55:08.465681+0000","last_clean":"2024-09-15T21:55:08.465681+0000","last_became_active":"2024-09-15T21:54:58.453714+0000","last_became_peered":"2024-09-15T21:54:58.453714+0000","last_unstale":"2024-09-15T21:55:08.465681+0000","last_undegraded":"2024-09-15T21:55:08.465681+0000","last_fullsized":"2024-09-15T21:55:08.465681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:42.307149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804964+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:55:08.804964+0000","last_peered":"2024-09-15T21:55:08.804964+0000","last_clean":"2024-09-15T21:55:08.804964+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:55:08.804964+0000","last_undegraded":"2024-09-15T21:55:08.804964+0000","last_fullsized":"2024-09-15T21:55:08.804964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464918+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:55:08.464918+0000","last_peered":"2024-09-15T21:55:08.464918+0000","last_clean":"2024-09-15T21:55:08.464918+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:55:08.464918+0000","last_undegraded":"2024-09-15T21:55:08.464918+0000","last_fullsized":"2024-09-15T21:55:08.464918+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466245+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:55:08.466245+0000","last_peered":"2024-09-15T21:55:08.466245+0000","last_clean":"2024-09-15T21:55:08.466245+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:55:08.466245+0000","last_undegraded":"2024-09-15T21:55:08.466245+0000","last_fullsized":"2024-09-15T21:55:08.466245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803407+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:55:08.803407+0000","last_peered":"2024-09-15T21:55:08.803407+0000","last_clean":"2024-09-15T21:55:08.803407+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:55:08.803407+0000","last_undegraded":"2024-09-15T21:55:08.803407+0000","last_fullsized":"2024-09-15T21:55:08.803407+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803319+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:55:08.803319+0000","last_peered":"2024-09-15T21:55:08.803319+0000","last_clean":"2024-09-15T21:55:08.803319+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:55:08.803319+0000","last_undegraded":"2024-09-15T21:55:08.803319+0000","last_fullsized":"2024-09-15T21:55:08.803319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803300+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:55:08.803300+0000","last_peered":"2024-09-15T21:55:08.803300+0000","last_clean":"2024-09-15T21:55:08.803300+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:55:08.803300+0000","last_undegraded":"2024-09-15T21:55:08.803300+0000","last_fullsized":"2024-09-15T21:55:08.803300+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594874+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:55:09.594874+0000","last_peered":"2024-09-15T21:55:09.594874+0000","last_clean":"2024-09-15T21:55:09.594874+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:55:09.594874+0000","last_undegraded":"2024-09-15T21:55:09.594874+0000","last_fullsized":"2024-09-15T21:55:09.594874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804677+0000","last_change":"2024-09-15T21:54:58.443632+0000","last_active":"2024-09-15T21:55:08.804677+0000","last_peered":"2024-09-15T21:55:08.804677+0000","last_clean":"2024-09-15T21:55:08.804677+0000","last_became_active":"2024-09-15T21:54:58.443273+0000","last_became_peered":"2024-09-15T21:54:58.443273+0000","last_unstale":"2024-09-15T21:55:08.804677+0000","last_undegraded":"2024-09-15T21:55:08.804677+0000","last_fullsized":"2024-09-15T21:55:08.804677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:40.732569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804729+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:55:08.804729+0000","last_peered":"2024-09-15T21:55:08.804729+0000","last_clean":"2024-09-15T21:55:08.804729+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:55:08.804729+0000","last_undegraded":"2024-09-15T21:55:08.804729+0000","last_fullsized":"2024-09-15T21:55:08.804729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465520+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:55:08.465520+0000","last_peered":"2024-09-15T21:55:08.465520+0000","last_clean":"2024-09-15T21:55:08.465520+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:55:08.465520+0000","last_undegraded":"2024-09-15T21:55:08.465520+0000","last_fullsized":"2024-09-15T21:55:08.465520+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804312+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:55:08.804312+0000","last_peered":"2024-09-15T21:55:08.804312+0000","last_clean":"2024-09-15T21:55:08.804312+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:55:08.804312+0000","last_undegraded":"2024-09-15T21:55:08.804312+0000","last_fullsized":"2024-09-15T21:55:08.804312+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627164+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:55:08.627164+0000","last_peered":"2024-09-15T21:55:08.627164+0000","last_clean":"2024-09-15T21:55:08.627164+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:55:08.627164+0000","last_undegraded":"2024-09-15T21:55:08.627164+0000","last_fullsized":"2024-09-15T21:55:08.627164+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804242+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:55:08.804242+0000","last_peered":"2024-09-15T21:55:08.804242+0000","last_clean":"2024-09-15T21:55:08.804242+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:55:08.804242+0000","last_undegraded":"2024-09-15T21:55:08.804242+0000","last_fullsized":"2024-09-15T21:55:08.804242+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465389+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:55:08.465389+0000","last_peered":"2024-09-15T21:55:08.465389+0000","last_clean":"2024-09-15T21:55:08.465389+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:55:08.465389+0000","last_undegraded":"2024-09-15T21:55:08.465389+0000","last_fullsized":"2024-09-15T21:55:08.465389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803822+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:55:08.803822+0000","last_peered":"2024-09-15T21:55:08.803822+0000","last_clean":"2024-09-15T21:55:08.803822+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:55:08.803822+0000","last_undegraded":"2024-09-15T21:55:08.803822+0000","last_fullsized":"2024-09-15T21:55:08.803822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804571+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:55:08.804571+0000","last_peered":"2024-09-15T21:55:08.804571+0000","last_clean":"2024-09-15T21:55:08.804571+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:55:08.804571+0000","last_undegraded":"2024-09-15T21:55:08.804571+0000","last_fullsized":"2024-09-15T21:55:08.804571+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803544+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:55:08.803544+0000","last_peered":"2024-09-15T21:55:08.803544+0000","last_clean":"2024-09-15T21:55:08.803544+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:55:08.803544+0000","last_undegraded":"2024-09-15T21:55:08.803544+0000","last_fullsized":"2024-09-15T21:55:08.803544+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803696+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:55:08.803696+0000","last_peered":"2024-09-15T21:55:08.803696+0000","last_clean":"2024-09-15T21:55:08.803696+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:55:08.803696+0000","last_undegraded":"2024-09-15T21:55:08.803696+0000","last_fullsized":"2024-09-15T21:55:08.803696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466206+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:55:08.466206+0000","last_peered":"2024-09-15T21:55:08.466206+0000","last_clean":"2024-09-15T21:55:08.466206+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:55:08.466206+0000","last_undegraded":"2024-09-15T21:55:08.466206+0000","last_fullsized":"2024-09-15T21:55:08.466206+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59621,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":149,"num_read_kb":140,"num_write":244,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":178863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514519,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":487584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776155,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1679360,"data_stored":1126420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939682,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":491217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168614,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":487499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397548,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1613824,"data_stored":1083998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561071,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":482625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1622016,"data_stored":1079580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":486699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":482333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":528006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":536068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":484474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":54165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":55171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":54149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:55:52.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T21:55:52.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:52 smithi203 ceph-mon[35546]: pgmap v315: 129 pgs: 129 active+clean; 635 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 18 op/s 2024-09-15T21:55:52.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:52 smithi203 ceph-mon[35546]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:55:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:52 smithi174 ceph-mon[35244]: pgmap v315: 129 pgs: 129 active+clean; 635 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.1 KiB/s wr, 18 op/s 2024-09-15T21:55:53.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:52 smithi174 ceph-mon[35244]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:55:53.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 -- 172.21.15.28:0/1065206076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec102790 msgr2=0x7fbfec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:53.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 --2- 172.21.15.28:0/1065206076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec102790 0x7fbfec1067e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009860 tx=0x7fbfdc02a340 comp rx=0 tx=0).stop 2024-09-15T21:55:53.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 -- 172.21.15.28:0/1065206076 shutdown_connections 2024-09-15T21:55:53.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 --2- 172.21.15.28:0/1065206076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec106f10 0x7fbfec10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 --2- 172.21.15.28:0/1065206076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec102790 0x7fbfec1067e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 --2- 172.21.15.28:0/1065206076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfec101de0 0x7fbfec1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 -- 172.21.15.28:0/1065206076 >> 172.21.15.28:0/1065206076 conn(0x7fbfec0fd8c0 msgr2=0x7fbfec0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 -- 172.21.15.28:0/1065206076 shutdown_connections 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.528+0000 7fbff388b640 1 -- 172.21.15.28:0/1065206076 wait complete. 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 Processor -- start 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 -- start start 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfec101de0 0x7fbfec198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec102790 0x7fbfec198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec106f10 0x7fbfec19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec111f30 con 0x7fbfec106f10 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec111db0 con 0x7fbfec102790 2024-09-15T21:55:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff388b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec1120b0 con 0x7fbfec101de0 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec106f10 0x7fbfec19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec106f10 0x7fbfec19cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51468/0 (socket says 172.21.15.28:51468) 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1627640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfec101de0 0x7fbfec198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 -- 172.21.15.28:0/3182700564 learned_addr learned my addr 172.21.15.28:0/3182700564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff0e26640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec102790 0x7fbfec198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 -- 172.21.15.28:0/3182700564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfec101de0 msgr2=0x7fbfec198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfec101de0 0x7fbfec198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 -- 172.21.15.28:0/3182700564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec102790 msgr2=0x7fbfec198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec102790 0x7fbfec198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.529+0000 7fbff1e28640 1 -- 172.21.15.28:0/3182700564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfec19d4c0 con 0x7fbfec106f10 2024-09-15T21:55:53.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.530+0000 7fbff1e28640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec106f10 0x7fbfec19cde0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400e940 tx=0x7fbfe400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.530+0000 7fbfe27fc640 1 -- 172.21.15.28:0/3182700564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400cd80 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.530+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfec19d7b0 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.530+0000 7fbfe27fc640 1 -- 172.21.15.28:0/3182700564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe40035f0 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.530+0000 7fbfe27fc640 1 -- 172.21.15.28:0/3182700564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4010690 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.530+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfec1a5090 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.531+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfec1021c0 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.532+0000 7fbfe27fc640 1 -- 172.21.15.28:0/3182700564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbfe40037a0 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.533+0000 7fbfe27fc640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbfd80a32d0 0x7fbfd80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.533+0000 7fbfe27fc640 1 -- 172.21.15.28:0/3182700564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fbfe40d0050 con 0x7fbfec106f10 2024-09-15T21:55:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.533+0000 7fbff1627640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbfd80a32d0 0x7fbfd80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.533+0000 7fbff1627640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbfd80a32d0 0x7fbfd80a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8005fd0 tx=0x7fbfe8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.536+0000 7fbfe27fc640 1 -- 172.21.15.28:0/3182700564 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbfe409ca60 con 0x7fbfec106f10 2024-09-15T21:55:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.735+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbfec062630 con 0x7fbfd80a32d0 2024-09-15T21:55:53.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:53 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:55:53.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.750+0000 7fbfe27fc640 1 -- 172.21.15.28:0/3182700564 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340391 (secure 0 0 0) 0x7fbfec062630 con 0x7fbfd80a32d0 2024-09-15T21:55:53.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:53.759 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.753+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbfd80a32d0 msgr2=0x7fbfd80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.753+0000 7fbff388b640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbfd80a32d0 0x7fbfd80a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8005fd0 tx=0x7fbfe8009500 comp rx=0 tx=0).stop 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec106f10 msgr2=0x7fbfec19cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec106f10 0x7fbfec19cde0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400e940 tx=0x7fbfe400ee00 comp rx=0 tx=0).stop 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 shutdown_connections 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbfd80a32d0 0x7fbfd80a5790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfec106f10 0x7fbfec19cde0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfec102790 0x7fbfec198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 --2- 172.21.15.28:0/3182700564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfec101de0 0x7fbfec198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 >> 172.21.15.28:0/3182700564 conn(0x7fbfec0fd8c0 msgr2=0x7fbfec0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 shutdown_connections 2024-09-15T21:55:53.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:53.754+0000 7fbff388b640 1 -- 172.21.15.28:0/3182700564 wait complete. 2024-09-15T21:55:53.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:53 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:55:54.001 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:53 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:54 smithi028 ceph-mon[30969]: pgmap v316: 129 pgs: 129 active+clean; 635 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-15T21:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:54 smithi028 ceph-mon[30969]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:55:54.827 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-15T21:55:53.550931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650223,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":300,"num_read_kb":264,"num_write":427,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":349,"ondisk_log_size":349,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363020,"kb_used_data":13772,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578804,"statfs":{"total":1151940427776,"available":1151568695296,"internally_reserved":0,"allocated":14102528,"data_stored":7765571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":49218,"num_objects":6,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":126,"num_read_kb":110,"num_write":43,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003399"},"pg_stats":[{"pgid":"2.1f","version":"88'13","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.583446+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:55:48.583446+0000","last_peered":"2024-09-15T21:55:48.583446+0000","last_clean":"2024-09-15T21:55:48.583446+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:55:48.583446+0000","last_undegraded":"2024-09-15T21:55:48.583446+0000","last_fullsized":"2024-09-15T21:55:48.583446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627418+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:55:08.627418+0000","last_peered":"2024-09-15T21:55:08.627418+0000","last_clean":"2024-09-15T21:55:08.627418+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:55:08.627418+0000","last_undegraded":"2024-09-15T21:55:08.627418+0000","last_fullsized":"2024-09-15T21:55:08.627418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803661+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:55:08.803661+0000","last_peered":"2024-09-15T21:55:08.803661+0000","last_clean":"2024-09-15T21:55:08.803661+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:55:08.803661+0000","last_undegraded":"2024-09-15T21:55:08.803661+0000","last_fullsized":"2024-09-15T21:55:08.803661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627711+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:55:08.627711+0000","last_peered":"2024-09-15T21:55:08.627711+0000","last_clean":"2024-09-15T21:55:08.627711+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:55:08.627711+0000","last_undegraded":"2024-09-15T21:55:08.627711+0000","last_fullsized":"2024-09-15T21:55:08.627711+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803860+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:55:08.803860+0000","last_peered":"2024-09-15T21:55:08.803860+0000","last_clean":"2024-09-15T21:55:08.803860+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:55:08.803860+0000","last_undegraded":"2024-09-15T21:55:08.803860+0000","last_fullsized":"2024-09-15T21:55:08.803860+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803735+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:55:08.803735+0000","last_peered":"2024-09-15T21:55:08.803735+0000","last_clean":"2024-09-15T21:55:08.803735+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:55:08.803735+0000","last_undegraded":"2024-09-15T21:55:08.803735+0000","last_fullsized":"2024-09-15T21:55:08.803735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:38.566678+0000","last_change":"2024-09-15T21:54:56.346287+0000","last_active":"2024-09-15T21:55:38.566678+0000","last_peered":"2024-09-15T21:55:38.566678+0000","last_clean":"2024-09-15T21:55:38.566678+0000","last_became_active":"2024-09-15T21:54:56.345880+0000","last_became_peered":"2024-09-15T21:54:56.345880+0000","last_unstale":"2024-09-15T21:55:38.566678+0000","last_undegraded":"2024-09-15T21:55:38.566678+0000","last_fullsized":"2024-09-15T21:55:38.566678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:30.899587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803836+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:55:08.803836+0000","last_peered":"2024-09-15T21:55:08.803836+0000","last_clean":"2024-09-15T21:55:08.803836+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:55:08.803836+0000","last_undegraded":"2024-09-15T21:55:08.803836+0000","last_fullsized":"2024-09-15T21:55:08.803836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804964+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:55:08.804964+0000","last_peered":"2024-09-15T21:55:08.804964+0000","last_clean":"2024-09-15T21:55:08.804964+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:55:08.804964+0000","last_undegraded":"2024-09-15T21:55:08.804964+0000","last_fullsized":"2024-09-15T21:55:08.804964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561353+0000","last_change":"2024-09-15T21:54:56.343604+0000","last_active":"2024-09-15T21:55:37.561353+0000","last_peered":"2024-09-15T21:55:37.561353+0000","last_clean":"2024-09-15T21:55:37.561353+0000","last_became_active":"2024-09-15T21:54:56.342956+0000","last_became_peered":"2024-09-15T21:54:56.342956+0000","last_unstale":"2024-09-15T21:55:37.561353+0000","last_undegraded":"2024-09-15T21:55:37.561353+0000","last_fullsized":"2024-09-15T21:55:37.561353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:43.299751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805132+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:55:08.805132+0000","last_peered":"2024-09-15T21:55:08.805132+0000","last_clean":"2024-09-15T21:55:08.805132+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:55:08.805132+0000","last_undegraded":"2024-09-15T21:55:08.805132+0000","last_fullsized":"2024-09-15T21:55:08.805132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804038+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:55:08.804038+0000","last_peered":"2024-09-15T21:55:08.804038+0000","last_clean":"2024-09-15T21:55:08.804038+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:55:08.804038+0000","last_undegraded":"2024-09-15T21:55:08.804038+0000","last_fullsized":"2024-09-15T21:55:08.804038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804372+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:55:08.804372+0000","last_peered":"2024-09-15T21:55:08.804372+0000","last_clean":"2024-09-15T21:55:08.804372+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:55:08.804372+0000","last_undegraded":"2024-09-15T21:55:08.804372+0000","last_fullsized":"2024-09-15T21:55:08.804372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:34.537983+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:55:34.537983+0000","last_peered":"2024-09-15T21:55:34.537983+0000","last_clean":"2024-09-15T21:55:34.537983+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:55:34.537983+0000","last_undegraded":"2024-09-15T21:55:34.537983+0000","last_fullsized":"2024-09-15T21:55:34.537983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805248+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:55:08.805248+0000","last_peered":"2024-09-15T21:55:08.805248+0000","last_clean":"2024-09-15T21:55:08.805248+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:55:08.805248+0000","last_undegraded":"2024-09-15T21:55:08.805248+0000","last_fullsized":"2024-09-15T21:55:08.805248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627523+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:55:08.627523+0000","last_peered":"2024-09-15T21:55:08.627523+0000","last_clean":"2024-09-15T21:55:08.627523+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:55:08.627523+0000","last_undegraded":"2024-09-15T21:55:08.627523+0000","last_fullsized":"2024-09-15T21:55:08.627523+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:36.556866+0000","last_change":"2024-09-15T21:54:56.343335+0000","last_active":"2024-09-15T21:55:36.556866+0000","last_peered":"2024-09-15T21:55:36.556866+0000","last_clean":"2024-09-15T21:55:36.556866+0000","last_became_active":"2024-09-15T21:54:56.343011+0000","last_became_peered":"2024-09-15T21:54:56.343011+0000","last_unstale":"2024-09-15T21:55:36.556866+0000","last_undegraded":"2024-09-15T21:55:36.556866+0000","last_fullsized":"2024-09-15T21:55:36.556866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:57.013184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804454+0000","last_change":"2024-09-15T21:55:05.471517+0000","last_active":"2024-09-15T21:55:08.804454+0000","last_peered":"2024-09-15T21:55:08.804454+0000","last_clean":"2024-09-15T21:55:08.804454+0000","last_became_active":"2024-09-15T21:55:05.471076+0000","last_became_peered":"2024-09-15T21:55:05.471076+0000","last_unstale":"2024-09-15T21:55:08.804454+0000","last_undegraded":"2024-09-15T21:55:08.804454+0000","last_fullsized":"2024-09-15T21:55:08.804454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:50.232177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465792+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:55:08.465792+0000","last_peered":"2024-09-15T21:55:08.465792+0000","last_clean":"2024-09-15T21:55:08.465792+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:55:08.465792+0000","last_undegraded":"2024-09-15T21:55:08.465792+0000","last_fullsized":"2024-09-15T21:55:08.465792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.560825+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:55:37.560825+0000","last_peered":"2024-09-15T21:55:37.560825+0000","last_clean":"2024-09-15T21:55:37.560825+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:55:37.560825+0000","last_undegraded":"2024-09-15T21:55:37.560825+0000","last_fullsized":"2024-09-15T21:55:37.560825+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464880+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:55:08.464880+0000","last_peered":"2024-09-15T21:55:08.464880+0000","last_clean":"2024-09-15T21:55:08.464880+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:55:08.464880+0000","last_undegraded":"2024-09-15T21:55:08.464880+0000","last_fullsized":"2024-09-15T21:55:08.464880+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803403+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:55:08.803403+0000","last_peered":"2024-09-15T21:55:08.803403+0000","last_clean":"2024-09-15T21:55:08.803403+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:55:08.803403+0000","last_undegraded":"2024-09-15T21:55:08.803403+0000","last_fullsized":"2024-09-15T21:55:08.803403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:34.537395+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:55:34.537395+0000","last_peered":"2024-09-15T21:55:34.537395+0000","last_clean":"2024-09-15T21:55:34.537395+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:55:34.537395+0000","last_undegraded":"2024-09-15T21:55:34.537395+0000","last_fullsized":"2024-09-15T21:55:34.537395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805291+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:55:08.805291+0000","last_peered":"2024-09-15T21:55:08.805291+0000","last_clean":"2024-09-15T21:55:08.805291+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:55:08.805291+0000","last_undegraded":"2024-09-15T21:55:08.805291+0000","last_fullsized":"2024-09-15T21:55:08.805291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803758+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:55:08.803758+0000","last_peered":"2024-09-15T21:55:08.803758+0000","last_clean":"2024-09-15T21:55:08.803758+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:55:08.803758+0000","last_undegraded":"2024-09-15T21:55:08.803758+0000","last_fullsized":"2024-09-15T21:55:08.803758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.560871+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:55:37.560871+0000","last_peered":"2024-09-15T21:55:37.560871+0000","last_clean":"2024-09-15T21:55:37.560871+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:55:37.560871+0000","last_undegraded":"2024-09-15T21:55:37.560871+0000","last_fullsized":"2024-09-15T21:55:37.560871+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803987+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:55:08.803987+0000","last_peered":"2024-09-15T21:55:08.803987+0000","last_clean":"2024-09-15T21:55:08.803987+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:55:08.803987+0000","last_undegraded":"2024-09-15T21:55:08.803987+0000","last_fullsized":"2024-09-15T21:55:08.803987+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627805+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:55:08.627805+0000","last_peered":"2024-09-15T21:55:08.627805+0000","last_clean":"2024-09-15T21:55:08.627805+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:55:08.627805+0000","last_undegraded":"2024-09-15T21:55:08.627805+0000","last_fullsized":"2024-09-15T21:55:08.627805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'15","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.438379+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:55:48.438379+0000","last_peered":"2024-09-15T21:55:48.438379+0000","last_clean":"2024-09-15T21:55:48.438379+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:55:48.438379+0000","last_undegraded":"2024-09-15T21:55:48.438379+0000","last_fullsized":"2024-09-15T21:55:48.438379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627135+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:55:08.627135+0000","last_peered":"2024-09-15T21:55:08.627135+0000","last_clean":"2024-09-15T21:55:08.627135+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:55:08.627135+0000","last_undegraded":"2024-09-15T21:55:08.627135+0000","last_fullsized":"2024-09-15T21:55:08.627135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595166+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:55:09.595166+0000","last_peered":"2024-09-15T21:55:09.595166+0000","last_clean":"2024-09-15T21:55:09.595166+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:55:09.595166+0000","last_undegraded":"2024-09-15T21:55:09.595166+0000","last_fullsized":"2024-09-15T21:55:09.595166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.562416+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:55:37.562416+0000","last_peered":"2024-09-15T21:55:37.562416+0000","last_clean":"2024-09-15T21:55:37.562416+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:55:37.562416+0000","last_undegraded":"2024-09-15T21:55:37.562416+0000","last_fullsized":"2024-09-15T21:55:37.562416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465946+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:55:08.465946+0000","last_peered":"2024-09-15T21:55:08.465946+0000","last_clean":"2024-09-15T21:55:08.465946+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:55:08.465946+0000","last_undegraded":"2024-09-15T21:55:08.465946+0000","last_fullsized":"2024-09-15T21:55:08.465946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465123+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:55:08.465123+0000","last_peered":"2024-09-15T21:55:08.465123+0000","last_clean":"2024-09-15T21:55:08.465123+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:55:08.465123+0000","last_undegraded":"2024-09-15T21:55:08.465123+0000","last_fullsized":"2024-09-15T21:55:08.465123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464769+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:55:08.464769+0000","last_peered":"2024-09-15T21:55:08.464769+0000","last_clean":"2024-09-15T21:55:08.464769+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:55:08.464769+0000","last_undegraded":"2024-09-15T21:55:08.464769+0000","last_fullsized":"2024-09-15T21:55:08.464769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803549+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:55:08.803549+0000","last_peered":"2024-09-15T21:55:08.803549+0000","last_clean":"2024-09-15T21:55:08.803549+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:55:08.803549+0000","last_undegraded":"2024-09-15T21:55:08.803549+0000","last_fullsized":"2024-09-15T21:55:08.803549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465116+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:55:08.465116+0000","last_peered":"2024-09-15T21:55:08.465116+0000","last_clean":"2024-09-15T21:55:08.465116+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:55:08.465116+0000","last_undegraded":"2024-09-15T21:55:08.465116+0000","last_fullsized":"2024-09-15T21:55:08.465116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:51.587776+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:55:51.587776+0000","last_peered":"2024-09-15T21:55:51.587776+0000","last_clean":"2024-09-15T21:55:51.587776+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:55:51.587776+0000","last_undegraded":"2024-09-15T21:55:51.587776+0000","last_fullsized":"2024-09-15T21:55:51.587776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595051+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:55:09.595051+0000","last_peered":"2024-09-15T21:55:09.595051+0000","last_clean":"2024-09-15T21:55:09.595051+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:55:09.595051+0000","last_undegraded":"2024-09-15T21:55:09.595051+0000","last_fullsized":"2024-09-15T21:55:09.595051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803791+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:55:08.803791+0000","last_peered":"2024-09-15T21:55:08.803791+0000","last_clean":"2024-09-15T21:55:08.803791+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:55:08.803791+0000","last_undegraded":"2024-09-15T21:55:08.803791+0000","last_fullsized":"2024-09-15T21:55:08.803791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:50.589406+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:55:50.589406+0000","last_peered":"2024-09-15T21:55:50.589406+0000","last_clean":"2024-09-15T21:55:50.589406+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:55:50.589406+0000","last_undegraded":"2024-09-15T21:55:50.589406+0000","last_fullsized":"2024-09-15T21:55:50.589406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595204+0000","last_change":"2024-09-15T21:55:05.496449+0000","last_active":"2024-09-15T21:55:09.595204+0000","last_peered":"2024-09-15T21:55:09.595204+0000","last_clean":"2024-09-15T21:55:09.595204+0000","last_became_active":"2024-09-15T21:55:05.496018+0000","last_became_peered":"2024-09-15T21:55:05.496018+0000","last_unstale":"2024-09-15T21:55:09.595204+0000","last_undegraded":"2024-09-15T21:55:09.595204+0000","last_fullsized":"2024-09-15T21:55:09.595204+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:18:44.250045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594596+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:55:09.594596+0000","last_peered":"2024-09-15T21:55:09.594596+0000","last_clean":"2024-09-15T21:55:09.594596+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:55:09.594596+0000","last_undegraded":"2024-09-15T21:55:09.594596+0000","last_fullsized":"2024-09-15T21:55:09.594596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:35.542432+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:55:35.542432+0000","last_peered":"2024-09-15T21:55:35.542432+0000","last_clean":"2024-09-15T21:55:35.542432+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:55:35.542432+0000","last_undegraded":"2024-09-15T21:55:35.542432+0000","last_fullsized":"2024-09-15T21:55:35.542432+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805007+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:55:08.805007+0000","last_peered":"2024-09-15T21:55:08.805007+0000","last_clean":"2024-09-15T21:55:08.805007+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:55:08.805007+0000","last_undegraded":"2024-09-15T21:55:08.805007+0000","last_fullsized":"2024-09-15T21:55:08.805007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627400+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:55:08.627400+0000","last_peered":"2024-09-15T21:55:08.627400+0000","last_clean":"2024-09-15T21:55:08.627400+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:55:08.627400+0000","last_undegraded":"2024-09-15T21:55:08.627400+0000","last_fullsized":"2024-09-15T21:55:08.627400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:38.567038+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:55:38.567038+0000","last_peered":"2024-09-15T21:55:38.567038+0000","last_clean":"2024-09-15T21:55:38.567038+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:55:38.567038+0000","last_undegraded":"2024-09-15T21:55:38.567038+0000","last_fullsized":"2024-09-15T21:55:38.567038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804812+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:55:08.804812+0000","last_peered":"2024-09-15T21:55:08.804812+0000","last_clean":"2024-09-15T21:55:08.804812+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:55:08.804812+0000","last_undegraded":"2024-09-15T21:55:08.804812+0000","last_fullsized":"2024-09-15T21:55:08.804812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804670+0000","last_change":"2024-09-15T21:55:05.468725+0000","last_active":"2024-09-15T21:55:08.804670+0000","last_peered":"2024-09-15T21:55:08.804670+0000","last_clean":"2024-09-15T21:55:08.804670+0000","last_became_active":"2024-09-15T21:55:05.468356+0000","last_became_peered":"2024-09-15T21:55:05.468356+0000","last_unstale":"2024-09-15T21:55:08.804670+0000","last_undegraded":"2024-09-15T21:55:08.804670+0000","last_fullsized":"2024-09-15T21:55:08.804670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:04:49.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804855+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:55:08.804855+0000","last_peered":"2024-09-15T21:55:08.804855+0000","last_clean":"2024-09-15T21:55:08.804855+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:55:08.804855+0000","last_undegraded":"2024-09-15T21:55:08.804855+0000","last_fullsized":"2024-09-15T21:55:08.804855+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'6","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:51.589391+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:55:51.589391+0000","last_peered":"2024-09-15T21:55:51.589391+0000","last_clean":"2024-09-15T21:55:51.589391+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:55:51.589391+0000","last_undegraded":"2024-09-15T21:55:51.589391+0000","last_fullsized":"2024-09-15T21:55:51.589391+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803813+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:55:08.803813+0000","last_peered":"2024-09-15T21:55:08.803813+0000","last_clean":"2024-09-15T21:55:08.803813+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:55:08.803813+0000","last_undegraded":"2024-09-15T21:55:08.803813+0000","last_fullsized":"2024-09-15T21:55:08.803813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804052+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:55:08.804052+0000","last_peered":"2024-09-15T21:55:08.804052+0000","last_clean":"2024-09-15T21:55:08.804052+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:55:08.804052+0000","last_undegraded":"2024-09-15T21:55:08.804052+0000","last_fullsized":"2024-09-15T21:55:08.804052+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561599+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:55:37.561599+0000","last_peered":"2024-09-15T21:55:37.561599+0000","last_clean":"2024-09-15T21:55:37.561599+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:55:37.561599+0000","last_undegraded":"2024-09-15T21:55:37.561599+0000","last_fullsized":"2024-09-15T21:55:37.561599+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803554+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:55:08.803554+0000","last_peered":"2024-09-15T21:55:08.803554+0000","last_clean":"2024-09-15T21:55:08.803554+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:55:08.803554+0000","last_undegraded":"2024-09-15T21:55:08.803554+0000","last_fullsized":"2024-09-15T21:55:08.803554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804410+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:55:08.804410+0000","last_peered":"2024-09-15T21:55:08.804410+0000","last_clean":"2024-09-15T21:55:08.804410+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:55:08.804410+0000","last_undegraded":"2024-09-15T21:55:08.804410+0000","last_fullsized":"2024-09-15T21:55:08.804410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561644+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:55:37.561644+0000","last_peered":"2024-09-15T21:55:37.561644+0000","last_clean":"2024-09-15T21:55:37.561644+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:55:37.561644+0000","last_undegraded":"2024-09-15T21:55:37.561644+0000","last_fullsized":"2024-09-15T21:55:37.561644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803480+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:55:08.803480+0000","last_peered":"2024-09-15T21:55:08.803480+0000","last_clean":"2024-09-15T21:55:08.803480+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:55:08.803480+0000","last_undegraded":"2024-09-15T21:55:08.803480+0000","last_fullsized":"2024-09-15T21:55:08.803480+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465949+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:55:08.465949+0000","last_peered":"2024-09-15T21:55:08.465949+0000","last_clean":"2024-09-15T21:55:08.465949+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:55:08.465949+0000","last_undegraded":"2024-09-15T21:55:08.465949+0000","last_fullsized":"2024-09-15T21:55:08.465949+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.539254+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:55:37.539254+0000","last_peered":"2024-09-15T21:55:37.539254+0000","last_clean":"2024-09-15T21:55:37.539254+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:55:37.539254+0000","last_undegraded":"2024-09-15T21:55:37.539254+0000","last_fullsized":"2024-09-15T21:55:37.539254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803913+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:55:08.803913+0000","last_peered":"2024-09-15T21:55:08.803913+0000","last_clean":"2024-09-15T21:55:08.803913+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:55:08.803913+0000","last_undegraded":"2024-09-15T21:55:08.803913+0000","last_fullsized":"2024-09-15T21:55:08.803913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804182+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:55:08.804182+0000","last_peered":"2024-09-15T21:55:08.804182+0000","last_clean":"2024-09-15T21:55:08.804182+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:55:08.804182+0000","last_undegraded":"2024-09-15T21:55:08.804182+0000","last_fullsized":"2024-09-15T21:55:08.804182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561639+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:55:37.561639+0000","last_peered":"2024-09-15T21:55:37.561639+0000","last_clean":"2024-09-15T21:55:37.561639+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:55:37.561639+0000","last_undegraded":"2024-09-15T21:55:37.561639+0000","last_fullsized":"2024-09-15T21:55:37.561639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595242+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:55:09.595242+0000","last_peered":"2024-09-15T21:55:09.595242+0000","last_clean":"2024-09-15T21:55:09.595242+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:55:09.595242+0000","last_undegraded":"2024-09-15T21:55:09.595242+0000","last_fullsized":"2024-09-15T21:55:09.595242+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804684+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:55:08.804684+0000","last_peered":"2024-09-15T21:55:08.804684+0000","last_clean":"2024-09-15T21:55:08.804684+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:55:08.804684+0000","last_undegraded":"2024-09-15T21:55:08.804684+0000","last_fullsized":"2024-09-15T21:55:08.804684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.587319+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:55:48.587319+0000","last_peered":"2024-09-15T21:55:48.587319+0000","last_clean":"2024-09-15T21:55:48.587319+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:55:48.587319+0000","last_undegraded":"2024-09-15T21:55:48.587319+0000","last_fullsized":"2024-09-15T21:55:48.587319+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464945+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:55:08.464945+0000","last_peered":"2024-09-15T21:55:08.464945+0000","last_clean":"2024-09-15T21:55:08.464945+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:55:08.464945+0000","last_undegraded":"2024-09-15T21:55:08.464945+0000","last_fullsized":"2024-09-15T21:55:08.464945+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804532+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:55:08.804532+0000","last_peered":"2024-09-15T21:55:08.804532+0000","last_clean":"2024-09-15T21:55:08.804532+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:55:08.804532+0000","last_undegraded":"2024-09-15T21:55:08.804532+0000","last_fullsized":"2024-09-15T21:55:08.804532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561862+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:55:37.561862+0000","last_peered":"2024-09-15T21:55:37.561862+0000","last_clean":"2024-09-15T21:55:37.561862+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:55:37.561862+0000","last_undegraded":"2024-09-15T21:55:37.561862+0000","last_fullsized":"2024-09-15T21:55:37.561862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465823+0000","last_change":"2024-09-15T21:55:05.469602+0000","last_active":"2024-09-15T21:55:08.465823+0000","last_peered":"2024-09-15T21:55:08.465823+0000","last_clean":"2024-09-15T21:55:08.465823+0000","last_became_active":"2024-09-15T21:55:05.468858+0000","last_became_peered":"2024-09-15T21:55:05.468858+0000","last_unstale":"2024-09-15T21:55:08.465823+0000","last_undegraded":"2024-09-15T21:55:08.465823+0000","last_fullsized":"2024-09-15T21:55:08.465823+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:58:09.823131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594797+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:55:09.594797+0000","last_peered":"2024-09-15T21:55:09.594797+0000","last_clean":"2024-09-15T21:55:09.594797+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:55:09.594797+0000","last_undegraded":"2024-09-15T21:55:09.594797+0000","last_fullsized":"2024-09-15T21:55:09.594797+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:38.566755+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:55:38.566755+0000","last_peered":"2024-09-15T21:55:38.566755+0000","last_clean":"2024-09-15T21:55:38.566755+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:55:38.566755+0000","last_undegraded":"2024-09-15T21:55:38.566755+0000","last_fullsized":"2024-09-15T21:55:38.566755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805112+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:55:08.805112+0000","last_peered":"2024-09-15T21:55:08.805112+0000","last_clean":"2024-09-15T21:55:08.805112+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:55:08.805112+0000","last_undegraded":"2024-09-15T21:55:08.805112+0000","last_fullsized":"2024-09-15T21:55:08.805112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804208+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:55:08.804208+0000","last_peered":"2024-09-15T21:55:08.804208+0000","last_clean":"2024-09-15T21:55:08.804208+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:55:08.804208+0000","last_undegraded":"2024-09-15T21:55:08.804208+0000","last_fullsized":"2024-09-15T21:55:08.804208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.589953+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:55:48.589953+0000","last_peered":"2024-09-15T21:55:48.589953+0000","last_clean":"2024-09-15T21:55:48.589953+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:55:48.589953+0000","last_undegraded":"2024-09-15T21:55:48.589953+0000","last_fullsized":"2024-09-15T21:55:48.589953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52413,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":65,"num_write":33,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464513+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:55:08.464513+0000","last_peered":"2024-09-15T21:55:08.464513+0000","last_clean":"2024-09-15T21:55:08.464513+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:55:08.464513+0000","last_undegraded":"2024-09-15T21:55:08.464513+0000","last_fullsized":"2024-09-15T21:55:08.464513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.805179+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:55:08.805179+0000","last_peered":"2024-09-15T21:55:08.805179+0000","last_clean":"2024-09-15T21:55:08.805179+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:55:08.805179+0000","last_undegraded":"2024-09-15T21:55:08.805179+0000","last_fullsized":"2024-09-15T21:55:08.805179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:36.555627+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:55:36.555627+0000","last_peered":"2024-09-15T21:55:36.555627+0000","last_clean":"2024-09-15T21:55:36.555627+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:55:36.555627+0000","last_undegraded":"2024-09-15T21:55:36.555627+0000","last_fullsized":"2024-09-15T21:55:36.555627+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804764+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:55:08.804764+0000","last_peered":"2024-09-15T21:55:08.804764+0000","last_clean":"2024-09-15T21:55:08.804764+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:55:08.804764+0000","last_undegraded":"2024-09-15T21:55:08.804764+0000","last_fullsized":"2024-09-15T21:55:08.804764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803890+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:55:08.803890+0000","last_peered":"2024-09-15T21:55:08.803890+0000","last_clean":"2024-09-15T21:55:08.803890+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:55:08.803890+0000","last_undegraded":"2024-09-15T21:55:08.803890+0000","last_fullsized":"2024-09-15T21:55:08.803890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561367+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:55:37.561367+0000","last_peered":"2024-09-15T21:55:37.561367+0000","last_clean":"2024-09-15T21:55:37.561367+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:55:37.561367+0000","last_undegraded":"2024-09-15T21:55:37.561367+0000","last_fullsized":"2024-09-15T21:55:37.561367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464287+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:55:08.464287+0000","last_peered":"2024-09-15T21:55:08.464287+0000","last_clean":"2024-09-15T21:55:08.464287+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:55:08.464287+0000","last_undegraded":"2024-09-15T21:55:08.464287+0000","last_fullsized":"2024-09-15T21:55:08.464287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803824+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:55:08.803824+0000","last_peered":"2024-09-15T21:55:08.803824+0000","last_clean":"2024-09-15T21:55:08.803824+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:55:08.803824+0000","last_undegraded":"2024-09-15T21:55:08.803824+0000","last_fullsized":"2024-09-15T21:55:08.803824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":56,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:48.584791+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:55:48.584791+0000","last_peered":"2024-09-15T21:55:48.584791+0000","last_clean":"2024-09-15T21:55:48.584791+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:55:48.584791+0000","last_undegraded":"2024-09-15T21:55:48.584791+0000","last_fullsized":"2024-09-15T21:55:48.584791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627307+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:55:08.627307+0000","last_peered":"2024-09-15T21:55:08.627307+0000","last_clean":"2024-09-15T21:55:08.627307+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:55:08.627307+0000","last_undegraded":"2024-09-15T21:55:08.627307+0000","last_fullsized":"2024-09-15T21:55:08.627307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803692+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:55:08.803692+0000","last_peered":"2024-09-15T21:55:08.803692+0000","last_clean":"2024-09-15T21:55:08.803692+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:55:08.803692+0000","last_undegraded":"2024-09-15T21:55:08.803692+0000","last_fullsized":"2024-09-15T21:55:08.803692+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464256+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:55:08.464256+0000","last_peered":"2024-09-15T21:55:08.464256+0000","last_clean":"2024-09-15T21:55:08.464256+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:55:08.464256+0000","last_undegraded":"2024-09-15T21:55:08.464256+0000","last_fullsized":"2024-09-15T21:55:08.464256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:37.561779+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:55:37.561779+0000","last_peered":"2024-09-15T21:55:37.561779+0000","last_clean":"2024-09-15T21:55:37.561779+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:55:37.561779+0000","last_undegraded":"2024-09-15T21:55:37.561779+0000","last_fullsized":"2024-09-15T21:55:37.561779+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466096+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:55:08.466096+0000","last_peered":"2024-09-15T21:55:08.466096+0000","last_clean":"2024-09-15T21:55:08.466096+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:55:08.466096+0000","last_undegraded":"2024-09-15T21:55:08.466096+0000","last_fullsized":"2024-09-15T21:55:08.466096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627287+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:55:08.627287+0000","last_peered":"2024-09-15T21:55:08.627287+0000","last_clean":"2024-09-15T21:55:08.627287+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:55:08.627287+0000","last_undegraded":"2024-09-15T21:55:08.627287+0000","last_fullsized":"2024-09-15T21:55:08.627287+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:35.543528+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:55:35.543528+0000","last_peered":"2024-09-15T21:55:35.543528+0000","last_clean":"2024-09-15T21:55:35.543528+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:55:35.543528+0000","last_undegraded":"2024-09-15T21:55:35.543528+0000","last_fullsized":"2024-09-15T21:55:35.543528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:36.556668+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:55:36.556668+0000","last_peered":"2024-09-15T21:55:36.556668+0000","last_clean":"2024-09-15T21:55:36.556668+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:55:36.556668+0000","last_undegraded":"2024-09-15T21:55:36.556668+0000","last_fullsized":"2024-09-15T21:55:36.556668+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804632+0000","last_change":"2024-09-15T21:54:58.444763+0000","last_active":"2024-09-15T21:55:08.804632+0000","last_peered":"2024-09-15T21:55:08.804632+0000","last_clean":"2024-09-15T21:55:08.804632+0000","last_became_active":"2024-09-15T21:54:58.443771+0000","last_became_peered":"2024-09-15T21:54:58.443771+0000","last_unstale":"2024-09-15T21:55:08.804632+0000","last_undegraded":"2024-09-15T21:55:08.804632+0000","last_fullsized":"2024-09-15T21:55:08.804632+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:49:25.519374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804183+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:55:08.804183+0000","last_peered":"2024-09-15T21:55:08.804183+0000","last_clean":"2024-09-15T21:55:08.804183+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:55:08.804183+0000","last_undegraded":"2024-09-15T21:55:08.804183+0000","last_fullsized":"2024-09-15T21:55:08.804183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:49.586838+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:55:49.586838+0000","last_peered":"2024-09-15T21:55:49.586838+0000","last_clean":"2024-09-15T21:55:49.586838+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:55:49.586838+0000","last_undegraded":"2024-09-15T21:55:49.586838+0000","last_fullsized":"2024-09-15T21:55:49.586838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803977+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:55:08.803977+0000","last_peered":"2024-09-15T21:55:08.803977+0000","last_clean":"2024-09-15T21:55:08.803977+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:55:08.803977+0000","last_undegraded":"2024-09-15T21:55:08.803977+0000","last_fullsized":"2024-09-15T21:55:08.803977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804801+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:55:08.804801+0000","last_peered":"2024-09-15T21:55:08.804801+0000","last_clean":"2024-09-15T21:55:08.804801+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:55:08.804801+0000","last_undegraded":"2024-09-15T21:55:08.804801+0000","last_fullsized":"2024-09-15T21:55:08.804801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465700+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:55:08.465700+0000","last_peered":"2024-09-15T21:55:08.465700+0000","last_clean":"2024-09-15T21:55:08.465700+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:55:08.465700+0000","last_undegraded":"2024-09-15T21:55:08.465700+0000","last_fullsized":"2024-09-15T21:55:08.465700+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804058+0000","last_change":"2024-09-15T21:54:58.445335+0000","last_active":"2024-09-15T21:55:08.804058+0000","last_peered":"2024-09-15T21:55:08.804058+0000","last_clean":"2024-09-15T21:55:08.804058+0000","last_became_active":"2024-09-15T21:54:58.444947+0000","last_became_peered":"2024-09-15T21:54:58.444947+0000","last_unstale":"2024-09-15T21:55:08.804058+0000","last_undegraded":"2024-09-15T21:55:08.804058+0000","last_fullsized":"2024-09-15T21:55:08.804058+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:12:55.939425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804355+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:55:08.804355+0000","last_peered":"2024-09-15T21:55:08.804355+0000","last_clean":"2024-09-15T21:55:08.804355+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:55:08.804355+0000","last_undegraded":"2024-09-15T21:55:08.804355+0000","last_fullsized":"2024-09-15T21:55:08.804355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803417+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:55:08.803417+0000","last_peered":"2024-09-15T21:55:08.803417+0000","last_clean":"2024-09-15T21:55:08.803417+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:55:08.803417+0000","last_undegraded":"2024-09-15T21:55:08.803417+0000","last_fullsized":"2024-09-15T21:55:08.803417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.595101+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:55:09.595101+0000","last_peered":"2024-09-15T21:55:09.595101+0000","last_clean":"2024-09-15T21:55:09.595101+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:55:09.595101+0000","last_undegraded":"2024-09-15T21:55:09.595101+0000","last_fullsized":"2024-09-15T21:55:09.595101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466034+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:55:08.466034+0000","last_peered":"2024-09-15T21:55:08.466034+0000","last_clean":"2024-09-15T21:55:08.466034+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:55:08.466034+0000","last_undegraded":"2024-09-15T21:55:08.466034+0000","last_fullsized":"2024-09-15T21:55:08.466034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804603+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:55:08.804603+0000","last_peered":"2024-09-15T21:55:08.804603+0000","last_clean":"2024-09-15T21:55:08.804603+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:55:08.804603+0000","last_undegraded":"2024-09-15T21:55:08.804603+0000","last_fullsized":"2024-09-15T21:55:08.804603+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627638+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:55:08.627638+0000","last_peered":"2024-09-15T21:55:08.627638+0000","last_clean":"2024-09-15T21:55:08.627638+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:55:08.627638+0000","last_undegraded":"2024-09-15T21:55:08.627638+0000","last_fullsized":"2024-09-15T21:55:08.627638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804801+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:55:08.804801+0000","last_peered":"2024-09-15T21:55:08.804801+0000","last_clean":"2024-09-15T21:55:08.804801+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:55:08.804801+0000","last_undegraded":"2024-09-15T21:55:08.804801+0000","last_fullsized":"2024-09-15T21:55:08.804801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627282+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:55:08.627282+0000","last_peered":"2024-09-15T21:55:08.627282+0000","last_clean":"2024-09-15T21:55:08.627282+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:55:08.627282+0000","last_undegraded":"2024-09-15T21:55:08.627282+0000","last_fullsized":"2024-09-15T21:55:08.627282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804683+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:55:08.804683+0000","last_peered":"2024-09-15T21:55:08.804683+0000","last_clean":"2024-09-15T21:55:08.804683+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:55:08.804683+0000","last_undegraded":"2024-09-15T21:55:08.804683+0000","last_fullsized":"2024-09-15T21:55:08.804683+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594882+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:55:09.594882+0000","last_peered":"2024-09-15T21:55:09.594882+0000","last_clean":"2024-09-15T21:55:09.594882+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:55:09.594882+0000","last_undegraded":"2024-09-15T21:55:09.594882+0000","last_fullsized":"2024-09-15T21:55:09.594882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465681+0000","last_change":"2024-09-15T21:54:58.454311+0000","last_active":"2024-09-15T21:55:08.465681+0000","last_peered":"2024-09-15T21:55:08.465681+0000","last_clean":"2024-09-15T21:55:08.465681+0000","last_became_active":"2024-09-15T21:54:58.453714+0000","last_became_peered":"2024-09-15T21:54:58.453714+0000","last_unstale":"2024-09-15T21:55:08.465681+0000","last_undegraded":"2024-09-15T21:55:08.465681+0000","last_fullsized":"2024-09-15T21:55:08.465681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:42.307149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804964+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:55:08.804964+0000","last_peered":"2024-09-15T21:55:08.804964+0000","last_clean":"2024-09-15T21:55:08.804964+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:55:08.804964+0000","last_undegraded":"2024-09-15T21:55:08.804964+0000","last_fullsized":"2024-09-15T21:55:08.804964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.464918+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:55:08.464918+0000","last_peered":"2024-09-15T21:55:08.464918+0000","last_clean":"2024-09-15T21:55:08.464918+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:55:08.464918+0000","last_undegraded":"2024-09-15T21:55:08.464918+0000","last_fullsized":"2024-09-15T21:55:08.464918+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466245+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:55:08.466245+0000","last_peered":"2024-09-15T21:55:08.466245+0000","last_clean":"2024-09-15T21:55:08.466245+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:55:08.466245+0000","last_undegraded":"2024-09-15T21:55:08.466245+0000","last_fullsized":"2024-09-15T21:55:08.466245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803407+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:55:08.803407+0000","last_peered":"2024-09-15T21:55:08.803407+0000","last_clean":"2024-09-15T21:55:08.803407+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:55:08.803407+0000","last_undegraded":"2024-09-15T21:55:08.803407+0000","last_fullsized":"2024-09-15T21:55:08.803407+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803319+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:55:08.803319+0000","last_peered":"2024-09-15T21:55:08.803319+0000","last_clean":"2024-09-15T21:55:08.803319+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:55:08.803319+0000","last_undegraded":"2024-09-15T21:55:08.803319+0000","last_fullsized":"2024-09-15T21:55:08.803319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803300+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:55:08.803300+0000","last_peered":"2024-09-15T21:55:08.803300+0000","last_clean":"2024-09-15T21:55:08.803300+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:55:08.803300+0000","last_undegraded":"2024-09-15T21:55:08.803300+0000","last_fullsized":"2024-09-15T21:55:08.803300+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:09.594874+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:55:09.594874+0000","last_peered":"2024-09-15T21:55:09.594874+0000","last_clean":"2024-09-15T21:55:09.594874+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:55:09.594874+0000","last_undegraded":"2024-09-15T21:55:09.594874+0000","last_fullsized":"2024-09-15T21:55:09.594874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804677+0000","last_change":"2024-09-15T21:54:58.443632+0000","last_active":"2024-09-15T21:55:08.804677+0000","last_peered":"2024-09-15T21:55:08.804677+0000","last_clean":"2024-09-15T21:55:08.804677+0000","last_became_active":"2024-09-15T21:54:58.443273+0000","last_became_peered":"2024-09-15T21:54:58.443273+0000","last_unstale":"2024-09-15T21:55:08.804677+0000","last_undegraded":"2024-09-15T21:55:08.804677+0000","last_fullsized":"2024-09-15T21:55:08.804677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:40.732569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804729+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:55:08.804729+0000","last_peered":"2024-09-15T21:55:08.804729+0000","last_clean":"2024-09-15T21:55:08.804729+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:55:08.804729+0000","last_undegraded":"2024-09-15T21:55:08.804729+0000","last_fullsized":"2024-09-15T21:55:08.804729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465520+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:55:08.465520+0000","last_peered":"2024-09-15T21:55:08.465520+0000","last_clean":"2024-09-15T21:55:08.465520+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:55:08.465520+0000","last_undegraded":"2024-09-15T21:55:08.465520+0000","last_fullsized":"2024-09-15T21:55:08.465520+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804312+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:55:08.804312+0000","last_peered":"2024-09-15T21:55:08.804312+0000","last_clean":"2024-09-15T21:55:08.804312+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:55:08.804312+0000","last_undegraded":"2024-09-15T21:55:08.804312+0000","last_fullsized":"2024-09-15T21:55:08.804312+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.627164+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:55:08.627164+0000","last_peered":"2024-09-15T21:55:08.627164+0000","last_clean":"2024-09-15T21:55:08.627164+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:55:08.627164+0000","last_undegraded":"2024-09-15T21:55:08.627164+0000","last_fullsized":"2024-09-15T21:55:08.627164+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804242+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:55:08.804242+0000","last_peered":"2024-09-15T21:55:08.804242+0000","last_clean":"2024-09-15T21:55:08.804242+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:55:08.804242+0000","last_undegraded":"2024-09-15T21:55:08.804242+0000","last_fullsized":"2024-09-15T21:55:08.804242+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.465389+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:55:08.465389+0000","last_peered":"2024-09-15T21:55:08.465389+0000","last_clean":"2024-09-15T21:55:08.465389+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:55:08.465389+0000","last_undegraded":"2024-09-15T21:55:08.465389+0000","last_fullsized":"2024-09-15T21:55:08.465389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803822+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:55:08.803822+0000","last_peered":"2024-09-15T21:55:08.803822+0000","last_clean":"2024-09-15T21:55:08.803822+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:55:08.803822+0000","last_undegraded":"2024-09-15T21:55:08.803822+0000","last_fullsized":"2024-09-15T21:55:08.803822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.804571+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:55:08.804571+0000","last_peered":"2024-09-15T21:55:08.804571+0000","last_clean":"2024-09-15T21:55:08.804571+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:55:08.804571+0000","last_undegraded":"2024-09-15T21:55:08.804571+0000","last_fullsized":"2024-09-15T21:55:08.804571+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803544+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:55:08.803544+0000","last_peered":"2024-09-15T21:55:08.803544+0000","last_clean":"2024-09-15T21:55:08.803544+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:55:08.803544+0000","last_undegraded":"2024-09-15T21:55:08.803544+0000","last_fullsized":"2024-09-15T21:55:08.803544+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.803696+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:55:08.803696+0000","last_peered":"2024-09-15T21:55:08.803696+0000","last_clean":"2024-09-15T21:55:08.803696+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:55:08.803696+0000","last_undegraded":"2024-09-15T21:55:08.803696+0000","last_fullsized":"2024-09-15T21:55:08.803696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T21:55:08.466206+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:55:08.466206+0000","last_peered":"2024-09-15T21:55:08.466206+0000","last_clean":"2024-09-15T21:55:08.466206+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:55:08.466206+0000","last_undegraded":"2024-09-15T21:55:08.466206+0000","last_fullsized":"2024-09-15T21:55:08.466206+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59621,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":164,"num_read_kb":145,"num_write":244,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":187393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514520,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":487584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776155,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1679360,"data_stored":1126420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939682,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":491217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168614,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":487499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397549,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1613824,"data_stored":1083998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561072,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":482625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1622016,"data_stored":1079580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":486699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":482333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":537074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":536068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":484474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":54165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":54149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:55:54.832 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T21:55:54.832 INFO:tasks.cephadm:Setting up client nodes... 2024-09-15T21:55:54.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-15T21:55:54.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:54 smithi203 ceph-mon[35546]: pgmap v316: 129 pgs: 129 active+clean; 635 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-15T21:55:54.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:54 smithi203 ceph-mon[35546]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:55:55.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:54 smithi174 ceph-mon[35244]: pgmap v316: 129 pgs: 129 active+clean; 635 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-15T21:55:55.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:54 smithi174 ceph-mon[35244]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:55:55.130 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.647+0000 7f54c03e7640 1 -- 172.21.15.28:0/97134226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b81037c0 msgr2=0x7f54b810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.647+0000 7f54c03e7640 1 --2- 172.21.15.28:0/97134226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b81037c0 0x7f54b810a050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009ca0 tx=0x7f54b002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 -- 172.21.15.28:0/97134226 shutdown_connections 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 --2- 172.21.15.28:0/97134226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b81037c0 0x7f54b810a050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 --2- 172.21.15.28:0/97134226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54b8102e00 0x7f54b8103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 --2- 172.21.15.28:0/97134226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54b8101c00 0x7f54b8102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 -- 172.21.15.28:0/97134226 >> 172.21.15.28:0/97134226 conn(0x7f54b80fd390 msgr2=0x7f54b80ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 -- 172.21.15.28:0/97134226 shutdown_connections 2024-09-15T21:55:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 -- 172.21.15.28:0/97134226 wait complete. 2024-09-15T21:55:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 Processor -- start 2024-09-15T21:55:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.648+0000 7f54c03e7640 1 -- start start 2024-09-15T21:55:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54c03e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54b8101c00 0x7f54b8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54c03e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b8102e00 0x7f54b8198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54c03e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54b81037c0 0x7f54b819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54c03e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54b810d8e0 con 0x7f54b8102e00 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54c03e7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54b810d760 con 0x7f54b81037c0 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54c03e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54b810da60 con 0x7f54b8101c00 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b8102e00 0x7f54b8198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b8102e00 0x7f54b8198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51488/0 (socket says 172.21.15.28:51488) 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 -- 172.21.15.28:0/762074445 learned_addr learned my addr 172.21.15.28:0/762074445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54be984640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54b81037c0 0x7f54b819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54be183640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54b8101c00 0x7f54b8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 -- 172.21.15.28:0/762074445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54b8101c00 msgr2=0x7f54b8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54b8101c00 0x7f54b8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 -- 172.21.15.28:0/762074445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54b81037c0 msgr2=0x7f54b819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54b81037c0 0x7f54b819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 -- 172.21.15.28:0/762074445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b819fb30 con 0x7f54b8102e00 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.649+0000 7f54bd982640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b8102e00 0x7f54b8198440 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f54a800b4a0 tx=0x7f54a800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.650+0000 7f54af7fe640 1 -- 172.21.15.28:0/762074445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8014070 con 0x7f54b8102e00 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.650+0000 7f54af7fe640 1 -- 172.21.15.28:0/762074445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a8010af0 con 0x7f54b8102e00 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.650+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b819fe20 con 0x7f54b8102e00 2024-09-15T21:55:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.650+0000 7f54af7fe640 1 -- 172.21.15.28:0/762074445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8017d10 con 0x7f54b8102e00 2024-09-15T21:55:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.650+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b81a0350 con 0x7f54b8102e00 2024-09-15T21:55:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.651+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b8109000 con 0x7f54b8102e00 2024-09-15T21:55:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.652+0000 7f54af7fe640 1 -- 172.21.15.28:0/762074445 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f54a8017070 con 0x7f54b8102e00 2024-09-15T21:55:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.652+0000 7f54af7fe640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54a40a3300 0x7f54a40a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.653+0000 7f54be183640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54a40a3300 0x7f54a40a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.653+0000 7f54af7fe640 1 -- 172.21.15.28:0/762074445 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f54a80ceea0 con 0x7f54b8102e00 2024-09-15T21:55:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.653+0000 7f54be183640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54a40a3300 0x7f54a40a57c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f54b8102c60 tx=0x7f54b400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:55.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.657+0000 7f54af7fe640 1 -- 172.21.15.28:0/762074445 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f54a809b280 con 0x7f54b8102e00 2024-09-15T21:55:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:55 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:55 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:55 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:55 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.934+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f54b802d9c0 con 0x7f54b8102e00 2024-09-15T21:55:55.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.937+0000 7f54af7fe640 1 -- 172.21.15.28:0/762074445 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f54a809f580 con 0x7f54b8102e00 2024-09-15T21:55:55.939 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-15T21:55:55.940 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDrV+dmSGbMNxAAtVE4nMDZ1fB02XjcmHcgPQ== 2024-09-15T21:55:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.939+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54a40a3300 msgr2=0x7f54a40a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.939+0000 7f54c03e7640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54a40a3300 0x7f54a40a57c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f54b8102c60 tx=0x7f54b400b390 comp rx=0 tx=0).stop 2024-09-15T21:55:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.939+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b8102e00 msgr2=0x7f54b8198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.939+0000 7f54c03e7640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b8102e00 0x7f54b8198440 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f54a800b4a0 tx=0x7f54a800b960 comp rx=0 tx=0).stop 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 shutdown_connections 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54a40a3300 0x7f54a40a57c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54b81037c0 0x7f54b819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b8102e00 0x7f54b8198440 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 --2- 172.21.15.28:0/762074445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54b8101c00 0x7f54b8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 >> 172.21.15.28:0/762074445 conn(0x7f54b80fd390 msgr2=0x7f54b81061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 shutdown_connections 2024-09-15T21:55:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:55.940+0000 7f54c03e7640 1 -- 172.21.15.28:0/762074445 wait complete. 2024-09-15T21:55:55.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:55 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:55.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:55 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:55.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:55 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:55.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:55 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:56.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:55 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:56.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:55 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:56.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:55 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:56.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:55 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:56.559 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:55:56.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-15T21:55:56.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T21:55:56.599 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T21:55:56.599 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-15T21:55:56.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-15T21:55:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-15T21:55:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/762074445' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T21:55:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/762074445' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T21:55:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:55:56.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:55:56.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-15T21:55:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/762074445' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T21:55:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/762074445' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T21:55:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:55:57.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:55:57.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-09-15T21:55:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/762074445' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T21:55:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/762074445' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T21:55:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:57.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:57.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:55:57.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:55:57.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:55:57.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 -- 172.21.15.28:0/2934127857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06501008f0 msgr2=0x7f0650100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:57.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 --2- 172.21.15.28:0/2934127857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06501008f0 0x7f0650100d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0644009930 tx=0x7f064402a420 comp rx=0 tx=0).stop 2024-09-15T21:55:57.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 -- 172.21.15.28:0/2934127857 shutdown_connections 2024-09-15T21:55:57.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 --2- 172.21.15.28:0/2934127857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0650101290 0x7f065010d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 --2- 172.21.15.28:0/2934127857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06501008f0 0x7f0650100d50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 --2- 172.21.15.28:0/2934127857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06501068f0 0x7f0650106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 -- 172.21.15.28:0/2934127857 >> 172.21.15.28:0/2934127857 conn(0x7f06500fc630 msgr2=0x7f06500fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 -- 172.21.15.28:0/2934127857 shutdown_connections 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 -- 172.21.15.28:0/2934127857 wait complete. 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.365+0000 7f0655802640 1 Processor -- start 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f0655802640 1 -- start start 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f0655802640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06501008f0 0x7f0650198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f0655802640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0650101290 0x7f06501987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f0655802640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06501068f0 0x7f065019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f0655802640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0650110cf0 con 0x7f0650101290 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f0655802640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0650110b70 con 0x7f06501008f0 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f0655802640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0650110e70 con 0x7f06501068f0 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0650101290 0x7f06501987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:57.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0650101290 0x7f06501987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51512/0 (socket says 172.21.15.28:51512) 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064e7fc640 1 -- 172.21.15.28:0/3613449429 learned_addr learned my addr 172.21.15.28:0/3613449429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064f7fe640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06501068f0 0x7f065019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064effd640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06501008f0 0x7f0650198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064e7fc640 1 -- 172.21.15.28:0/3613449429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06501068f0 msgr2=0x7f065019cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064e7fc640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06501068f0 0x7f065019cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.366+0000 7f064e7fc640 1 -- 172.21.15.28:0/3613449429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06501008f0 msgr2=0x7f0650198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f064e7fc640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06501008f0 0x7f0650198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f064e7fc640 1 -- 172.21.15.28:0/3613449429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065019d220 con 0x7f0650101290 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f064e7fc640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0650101290 0x7f06501987b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0644009230 tx=0x7f0644004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f0654800640 1 -- 172.21.15.28:0/3613449429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064403f030 con 0x7f0650101290 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f0654800640 1 -- 172.21.15.28:0/3613449429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0644003ad0 con 0x7f0650101290 2024-09-15T21:55:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065019d4b0 con 0x7f0650101290 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f0654800640 1 -- 172.21.15.28:0/3613449429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644035070 con 0x7f0650101290 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.367+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06501a4df0 con 0x7f0650101290 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.368+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0650102030 con 0x7f0650101290 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.369+0000 7f0654800640 1 -- 172.21.15.28:0/3613449429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0644042030 con 0x7f0650101290 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.370+0000 7f0654800640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06380a32d0 0x7f06380a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.370+0000 7f064effd640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06380a32d0 0x7f06380a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.370+0000 7f0654800640 1 -- 172.21.15.28:0/3613449429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f06440f0120 con 0x7f0650101290 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.370+0000 7f064effd640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06380a32d0 0x7f06380a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f064000bb60 tx=0x7f064000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.374+0000 7f0654800640 1 -- 172.21.15.28:0/3613449429 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f06440bc450 con 0x7f0650101290 2024-09-15T21:55:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.598+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f0650109e70 con 0x7f0650101290 2024-09-15T21:55:57.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.602+0000 7f0654800640 1 -- 172.21.15.28:0/3613449429 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264048 (secure 0 0 0) 0x7f06440c0750 con 0x7f0650101290 2024-09-15T21:55:57.607 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:57.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06380a32d0 msgr2=0x7f06380a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:57.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06380a32d0 0x7f06380a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f064000bb60 tx=0x7f064000b390 comp rx=0 tx=0).stop 2024-09-15T21:55:57.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0650101290 msgr2=0x7f06501987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:57.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0650101290 0x7f06501987b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0644009230 tx=0x7f0644004270 comp rx=0 tx=0).stop 2024-09-15T21:55:57.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 shutdown_connections 2024-09-15T21:55:57.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06380a32d0 0x7f06380a5790 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06501068f0 0x7f065019cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0650101290 0x7f06501987b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 --2- 172.21.15.28:0/3613449429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06501008f0 0x7f0650198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:57.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 >> 172.21.15.28:0/3613449429 conn(0x7f06500fc630 msgr2=0x7f065006a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:57.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 shutdown_connections 2024-09-15T21:55:57.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:57.607+0000 7f0655802640 1 -- 172.21.15.28:0/3613449429 wait complete. 2024-09-15T21:55:57.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-09-15T21:55:57.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-09-15T21:55:57.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:57 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3613449429' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T21:55:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-09-15T21:55:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-09-15T21:55:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-09-15T21:55:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-09-15T21:55:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-09-15T21:55:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-09-15T21:55:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-09-15T21:55:58.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-09-15T21:55:58.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-09-15T21:55:58.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-09-15T21:55:58.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:57 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3613449429' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-09-15T21:55:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-09-15T21:55:58.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-09-15T21:55:58.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:57 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3613449429' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T21:55:58.308 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1670367809},{"type":"v1","addr":"172.21.15.28:6801","nonce":1670367809}]},"active_addr":"172.21.15.28:6801/1670367809","active_change":"2024-09-15T21:47:57.453564+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14217,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3676950981}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":4057362541}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":955882315}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":972903625}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3979829443}]}]} 2024-09-15T21:55:58.314 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-15T21:55:58.314 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-15T21:55:58.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:55:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:55:59.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-15T21:55:59.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:58 smithi028 ceph-mon[30969]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T21:55:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 -- 172.21.15.28:0/1456467352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa381036e0 msgr2=0x7ffa38107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1456467352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa381036e0 0x7ffa38107b90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009930 tx=0x7ffa2002a420 comp rx=0 tx=0).stop 2024-09-15T21:55:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 -- 172.21.15.28:0/1456467352 shutdown_connections 2024-09-15T21:55:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1456467352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa381082a0 0x7ffa3810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1456467352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa381036e0 0x7ffa38107b90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1456467352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa38102d30 0x7ffa38103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 -- 172.21.15.28:0/1456467352 >> 172.21.15.28:0/1456467352 conn(0x7ffa380fc7d0 msgr2=0x7ffa380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 -- 172.21.15.28:0/1456467352 shutdown_connections 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 -- 172.21.15.28:0/1456467352 wait complete. 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.130+0000 7ffa3d735640 1 Processor -- start 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa3d735640 1 -- start start 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa3d735640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa38102d30 0x7ffa38198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa3d735640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa381036e0 0x7ffa38198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa3d735640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa381082a0 0x7ffa3819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa3d735640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa381120b0 con 0x7ffa38102d30 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa3d735640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa38111f30 con 0x7ffa381082a0 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa3d735640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffa38112230 con 0x7ffa381036e0 2024-09-15T21:55:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa38102d30 0x7ffa38198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa38102d30 0x7ffa38198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51524/0 (socket says 172.21.15.28:51524) 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 -- 172.21.15.28:0/1042108568 learned_addr learned my addr 172.21.15.28:0/1042108568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa367fc640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa381036e0 0x7ffa38198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa377fe640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa381082a0 0x7ffa3819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 -- 172.21.15.28:0/1042108568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa381036e0 msgr2=0x7ffa38198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa381036e0 0x7ffa38198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 -- 172.21.15.28:0/1042108568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa381082a0 msgr2=0x7ffa3819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa381082a0 0x7ffa3819ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.131+0000 7ffa36ffd640 1 -- 172.21.15.28:0/1042108568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3819d4f0 con 0x7ffa38102d30 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.132+0000 7ffa36ffd640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa38102d30 0x7ffa38198540 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c920 tx=0x7ffa3000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.132+0000 7ffa2ffff640 1 -- 172.21.15.28:0/1042108568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30007c70 con 0x7ffa38102d30 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.132+0000 7ffa2ffff640 1 -- 172.21.15.28:0/1042108568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa30007e10 con 0x7ffa38102d30 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.132+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3819d7e0 con 0x7ffa38102d30 2024-09-15T21:55:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.132+0000 7ffa2ffff640 1 -- 172.21.15.28:0/1042108568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30018890 con 0x7ffa38102d30 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.132+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa381a50c0 con 0x7ffa38102d30 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.133+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa38103110 con 0x7ffa38102d30 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.134+0000 7ffa2ffff640 1 -- 172.21.15.28:0/1042108568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ffa30018a70 con 0x7ffa38102d30 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.135+0000 7ffa2ffff640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ffa240a32d0 0x7ffa240a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.135+0000 7ffa367fc640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ffa240a32d0 0x7ffa240a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.135+0000 7ffa2ffff640 1 -- 172.21.15.28:0/1042108568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7ffa300cff50 con 0x7ffa38102d30 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.135+0000 7ffa367fc640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ffa240a32d0 0x7ffa240a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa2002a940 tx=0x7ffa20005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:55:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.139+0000 7ffa2ffff640 1 -- 172.21.15.28:0/1042108568 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffa30014030 con 0x7ffa38102d30 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-09-15T21:55:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-09-15T21:55:59.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-09-15T21:55:59.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-15T21:55:59.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:58 smithi203 ceph-mon[35546]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T21:55:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-15T21:55:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-09-15T21:55:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-09-15T21:55:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-09-15T21:55:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-09-15T21:55:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-09-15T21:55:59.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-09-15T21:55:59.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-09-15T21:55:59.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-09-15T21:55:59.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-09-15T21:55:59.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-15T21:55:59.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:58 smithi174 ceph-mon[35244]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T21:55:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.316+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa3810a2b0 con 0x7ffa38102d30 2024-09-15T21:55:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.318+0000 7ffa2ffff640 1 -- 172.21.15.28:0/1042108568 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24715 (secure 0 0 0) 0x7ffa3009c7b0 con 0x7ffa38102d30 2024-09-15T21:55:59.320 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:55:59.320 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:58.788704+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:55:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ffa240a32d0 msgr2=0x7ffa240a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ffa240a32d0 0x7ffa240a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa2002a940 tx=0x7ffa20005a40 comp rx=0 tx=0).stop 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa38102d30 msgr2=0x7ffa38198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa38102d30 0x7ffa38198540 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c920 tx=0x7ffa3000cde0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 shutdown_connections 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7ffa240a32d0 0x7ffa240a5790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa381082a0 0x7ffa3819ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa381036e0 0x7ffa38198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 --2- 172.21.15.28:0/1042108568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa38102d30 0x7ffa38198540 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 >> 172.21.15.28:0/1042108568 conn(0x7ffa380fc7d0 msgr2=0x7ffa38106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 shutdown_connections 2024-09-15T21:55:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:55:59.320+0000 7ffa3d735640 1 -- 172.21.15.28:0/1042108568 wait complete. 2024-09-15T21:55:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:59 smithi028 ceph-mon[30969]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T21:55:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:55:59 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1042108568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:55:59.960 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-15T21:55:59.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:56:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:59 smithi203 ceph-mon[35546]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T21:56:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:55:59 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1042108568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:00.219 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:59 smithi174 ceph-mon[35244]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T21:56:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:55:59 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1042108568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 -- 172.21.15.28:0/2184139217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 msgr2=0x7f171c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 --2- 172.21.15.28:0/2184139217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c10ea90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1714009ca0 tx=0x7f171402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 -- 172.21.15.28:0/2184139217 shutdown_connections 2024-09-15T21:56:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 --2- 172.21.15.28:0/2184139217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c10ea90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 --2- 172.21.15.28:0/2184139217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f171c102790 0x7f171c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 --2- 172.21.15.28:0/2184139217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c101de0 0x7f171c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 -- 172.21.15.28:0/2184139217 >> 172.21.15.28:0/2184139217 conn(0x7f171c0fd8c0 msgr2=0x7f171c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 -- 172.21.15.28:0/2184139217 shutdown_connections 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.705+0000 7f17247df640 1 -- 172.21.15.28:0/2184139217 wait complete. 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 Processor -- start 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 -- start start 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c101de0 0x7f171c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f171c102790 0x7f171c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f171c111f90 con 0x7f171c106f10 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f171c111e10 con 0x7f171c101de0 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f17247df640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f171c112110 con 0x7f171c102790 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f1722d7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f1722d7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37394/0 (socket says 172.21.15.28:37394) 2024-09-15T21:56:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f1722d7c640 1 -- 172.21.15.28:0/1501378659 learned_addr learned my addr 172.21.15.28:0/1501378659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f1721d7a640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f171c102790 0x7f171c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.706+0000 7f172257b640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c101de0 0x7f171c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f1722d7c640 1 -- 172.21.15.28:0/1501378659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f171c102790 msgr2=0x7f171c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f1722d7c640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f171c102790 0x7f171c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f1722d7c640 1 -- 172.21.15.28:0/1501378659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c101de0 msgr2=0x7f171c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f1722d7c640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c101de0 0x7f171c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f1722d7c640 1 -- 172.21.15.28:0/1501378659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c1a6930 con 0x7f171c106f10 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f1722d7c640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c074f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1714031a20 tx=0x7f1714031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f17137fe640 1 -- 172.21.15.28:0/1501378659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714003b70 con 0x7f171c106f10 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f17137fe640 1 -- 172.21.15.28:0/1501378659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1714003d10 con 0x7f171c106f10 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171c1a6bc0 con 0x7f171c106f10 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f17137fe640 1 -- 172.21.15.28:0/1501378659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171402fc80 con 0x7f171c106f10 2024-09-15T21:56:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.707+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171c1a7100 con 0x7f171c106f10 2024-09-15T21:56:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.708+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f171c1021c0 con 0x7f171c106f10 2024-09-15T21:56:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.709+0000 7f17137fe640 1 -- 172.21.15.28:0/1501378659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f17140046a0 con 0x7f171c106f10 2024-09-15T21:56:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.710+0000 7f17137fe640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f17080a32d0 0x7f17080a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.710+0000 7f172257b640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f17080a32d0 0x7f17080a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.711+0000 7f17137fe640 1 -- 172.21.15.28:0/1501378659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f17140f1120 con 0x7f171c106f10 2024-09-15T21:56:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.711+0000 7f172257b640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f17080a32d0 0x7f17080a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1718005fd0 tx=0x7f171800b570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:00.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.715+0000 7f17137fe640 1 -- 172.21.15.28:0/1501378659 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f17140bdb20 con 0x7f171c106f10 2024-09-15T21:56:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.893+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f171c109b60 con 0x7f171c106f10 2024-09-15T21:56:00.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:00 smithi028 ceph-mon[30969]: pgmap v321: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-15T21:56:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.895+0000 7f17137fe640 1 -- 172.21.15.28:0/1501378659 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24715 (secure 0 0 0) 0x7f17140bd420 con 0x7f171c106f10 2024-09-15T21:56:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:00.897 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:58.788704+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:56:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f17080a32d0 msgr2=0x7f17080a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f17080a32d0 0x7f17080a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1718005fd0 tx=0x7f171800b570 comp rx=0 tx=0).stop 2024-09-15T21:56:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 msgr2=0x7f171c074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c074f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1714031a20 tx=0x7f1714031a50 comp rx=0 tx=0).stop 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 shutdown_connections 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f17080a32d0 0x7f17080a5790 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c106f10 0x7f171c074f70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f171c102790 0x7f171c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 --2- 172.21.15.28:0/1501378659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c101de0 0x7f171c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 >> 172.21.15.28:0/1501378659 conn(0x7f171c0fd8c0 msgr2=0x7f171c0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 shutdown_connections 2024-09-15T21:56:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:00.898+0000 7f17247df640 1 -- 172.21.15.28:0/1501378659 wait complete. 2024-09-15T21:56:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:00 smithi203 ceph-mon[35546]: pgmap v321: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-15T21:56:01.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:00 smithi174 ceph-mon[35244]: pgmap v321: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-15T21:56:01.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-15T21:56:01.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-15T21:56:01.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-15T21:56:01.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-15T21:56:01.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-15T21:56:01.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-15T21:56:01.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-15T21:56:01.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-15T21:56:01.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-15T21:56:01.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-15T21:56:01.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-15T21:56:01.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-15T21:56:01.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:01 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1501378659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:02.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:01 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1501378659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:02.242 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.248 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:01 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1501378659' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:02.357 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.357 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.358 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.359 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.361 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.361 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.363 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.366 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/4168707846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb8103660 msgr2=0x7f7cb810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 --2- 172.21.15.28:0/4168707846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb8103660 0x7f7cb810bb50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009900 tx=0x7f7ca002a480 comp rx=0 tx=0).stop 2024-09-15T21:56:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/4168707846 shutdown_connections 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 --2- 172.21.15.28:0/4168707846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cb810c170 0x7f7cb810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 --2- 172.21.15.28:0/4168707846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb8103660 0x7f7cb810bb50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 --2- 172.21.15.28:0/4168707846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb8102d40 0x7f7cb8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/4168707846 >> 172.21.15.28:0/4168707846 conn(0x7f7cb80fc7a0 msgr2=0x7f7cb80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/4168707846 shutdown_connections 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/4168707846 wait complete. 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.970+0000 7f7cbd2e3640 1 Processor -- start 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cbd2e3640 1 -- start start 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cbd2e3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cb8102d40 0x7f7cb8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cbd2e3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb8103660 0x7f7cb8198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cbd2e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb810c170 0x7f7cb819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cbd2e3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb8111f70 con 0x7f7cb810c170 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cbd2e3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb8111df0 con 0x7f7cb8102d40 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cbd2e3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb81120f0 con 0x7f7cb8103660 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb810c170 0x7f7cb819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb810c170 0x7f7cb819cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37418/0 (socket says 172.21.15.28:37418) 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 -- 172.21.15.28:0/3966352897 learned_addr learned my addr 172.21.15.28:0/3966352897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb6ffd640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cb8102d40 0x7f7cb8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cae7fc640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb8103660 0x7f7cb8198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 -- 172.21.15.28:0/3966352897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb8103660 msgr2=0x7f7cb8198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cb8103660 0x7f7cb8198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 -- 172.21.15.28:0/3966352897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cb8102d40 msgr2=0x7f7cb8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cb8102d40 0x7f7cb8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 -- 172.21.15.28:0/3966352897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb819d400 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.971+0000 7f7cb77fe640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb810c170 0x7f7cb819cd20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000eaa0 tx=0x7f7cb000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.972+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb000cdb0 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.972+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cb0005020 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.972+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb819d6f0 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.972+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb00116b0 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.972+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb81a4fd0 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.973+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7cb8062980 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.974+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7cb0005560 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.974+0000 7f7cb4ff9640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ca40a32d0 0x7f7ca40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cb6ffd640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ca40a32d0 0x7f7ca40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f7cb00cffc0 con 0x7f7cb810c170 2024-09-15T21:56:02.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cb4ff9640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] conn(0x7f7ca40ae4a0 0x7f7ca40b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 --> [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cb0018ec0 con 0x7f7ca40ae4a0 2024-09-15T21:56:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f7cb00d25e0 con 0x7f7cb810c170 2024-09-15T21:56:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cae7fc640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] conn(0x7f7ca40ae4a0 0x7f7ca40b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cb6ffd640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ca40a32d0 0x7f7ca40a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800bbe0 tx=0x7f7ca800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.975+0000 7f7cae7fc640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] conn(0x7f7ca40ae4a0 0x7f7ca40b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:02.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.976+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== osd.9 v2:172.21.15.28:6826/306652678 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7cb0018ec0 con 0x7f7ca40ae4a0 2024-09-15T21:56:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.989+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 --> [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7cb8062e30 con 0x7f7ca40ae4a0 2024-09-15T21:56:02.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cb4ff9640 1 -- 172.21.15.28:0/3966352897 <== osd.9 v2:172.21.15.28:6826/306652678 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7cb8062e30 con 0x7f7ca40ae4a0 2024-09-15T21:56:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] conn(0x7f7ca40ae4a0 msgr2=0x7f7ca40b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] conn(0x7f7ca40ae4a0 0x7f7ca40b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ca40a32d0 msgr2=0x7f7ca40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7ca40a32d0 0x7f7ca40a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800bbe0 tx=0x7f7ca800b390 comp rx=0 tx=0).stop 2024-09-15T21:56:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb810c170 msgr2=0x7f7cb819cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:02.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 --2- 172.21.15.28:0/3966352897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb810c170 0x7f7cb819cd20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000eaa0 tx=0x7f7cb000ef60 comp rx=0 tx=0).stop 2024-09-15T21:56:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cb77fe640 1 -- 172.21.15.28:0/3966352897 reap_dead start 2024-09-15T21:56:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 shutdown_connections 2024-09-15T21:56:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.990+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 >> 172.21.15.28:0/3966352897 conn(0x7f7cb80fc7a0 msgr2=0x7f7cb80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.991+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 shutdown_connections 2024-09-15T21:56:02.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:02.991+0000 7f7cbd2e3640 1 -- 172.21.15.28:0/3966352897 wait complete. 2024-09-15T21:56:03.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:02 smithi028 ceph-mon[30969]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2024-09-15T21:56:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 -- 172.21.15.28:0/82087393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 msgr2=0x7f1240101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 --2- 172.21.15.28:0/82087393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 0x7f1240101200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1230009900 tx=0x7f123002a480 comp rx=0 tx=0).stop 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 -- 172.21.15.28:0/82087393 shutdown_connections 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 --2- 172.21.15.28:0/82087393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1240102ef0 0x7f124010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 --2- 172.21.15.28:0/82087393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1240102570 0x7f12401029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 --2- 172.21.15.28:0/82087393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 0x7f1240101200 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 -- 172.21.15.28:0/82087393 >> 172.21.15.28:0/82087393 conn(0x7f12400fc7d0 msgr2=0x7f12400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 -- 172.21.15.28:0/82087393 shutdown_connections 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 -- 172.21.15.28:0/82087393 wait complete. 2024-09-15T21:56:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 Processor -- start 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.154+0000 7f1246302640 1 -- start start 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f1246302640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 0x7f1240199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f1246302640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1240102570 0x7f1240199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f1246302640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1240102ef0 0x7f1240199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f1246302640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1240110fd0 con 0x7f1240100e20 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f1246302640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1240110e50 con 0x7f1240102ef0 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f1246302640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1240111150 con 0x7f1240102570 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 0x7f1240199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 0x7f1240199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37430/0 (socket says 172.21.15.28:37430) 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 -- 172.21.15.28:0/1030496001 learned_addr learned my addr 172.21.15.28:0/1030496001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f124489f640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1240102ef0 0x7f1240199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123f7fe640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1240102570 0x7f1240199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 -- 172.21.15.28:0/1030496001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1240102570 msgr2=0x7f1240199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1240102570 0x7f1240199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 -- 172.21.15.28:0/1030496001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1240102ef0 msgr2=0x7f1240199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1240102ef0 0x7f1240199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f123ffff640 1 -- 172.21.15.28:0/1030496001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12401a4820 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.155+0000 7f124489f640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1240102ef0 0x7f1240199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.156+0000 7f123ffff640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 0x7f1240199420 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f123002aa70 tx=0x7f1230003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.156+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123003f070 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.156+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12401a4ab0 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.156+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1230031c30 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.156+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1230035030 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.156+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12401a4f90 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.157+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1240062900 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.158+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1230042070 con 0x7f1240100e20 2024-09-15T21:56:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f123d7fa640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12280a32d0 0x7f12280a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f123f7fe640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12280a32d0 0x7f12280a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f12300f0bf0 con 0x7f1240100e20 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f123d7fa640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] conn(0x7f12280ae4a0 0x7f12280b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 --> [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1230038e90 con 0x7f12280ae4a0 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f12300f1040 con 0x7f1240100e20 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f124489f640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] conn(0x7f12280ae4a0 0x7f12280b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.159+0000 7f123f7fe640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12280a32d0 0x7f12280a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1234005fd0 tx=0x7f1234005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.160+0000 7f124489f640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] conn(0x7f12280ae4a0 0x7f12280b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.160+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== osd.6 v2:172.21.15.28:6818/908626673 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1230038e90 con 0x7f12280ae4a0 2024-09-15T21:56:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.174+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 --> [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1240062d80 con 0x7f12280ae4a0 2024-09-15T21:56:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.174+0000 7f123d7fa640 1 -- 172.21.15.28:0/1030496001 <== osd.6 v2:172.21.15.28:6818/908626673 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1240062d80 con 0x7f12280ae4a0 2024-09-15T21:56:03.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.174+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] conn(0x7f12280ae4a0 msgr2=0x7f12280b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.174+0000 7f1246302640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] conn(0x7f12280ae4a0 0x7f12280b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12280a32d0 msgr2=0x7f12280a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f12280a32d0 0x7f12280a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1234005fd0 tx=0x7f1234005850 comp rx=0 tx=0).stop 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 msgr2=0x7f1240199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 --2- 172.21.15.28:0/1030496001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1240100e20 0x7f1240199420 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f123002aa70 tx=0x7f1230003c10 comp rx=0 tx=0).stop 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f124489f640 1 -- 172.21.15.28:0/1030496001 reap_dead start 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 shutdown_connections 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 >> 172.21.15.28:0/1030496001 conn(0x7f12400fc7d0 msgr2=0x7f12401068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 shutdown_connections 2024-09-15T21:56:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.175+0000 7f1246302640 1 -- 172.21.15.28:0/1030496001 wait complete. 2024-09-15T21:56:03.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:02 smithi203 ceph-mon[35546]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2024-09-15T21:56:03.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:02 smithi174 ceph-mon[35244]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 11 op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2024-09-15T21:56:04.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 -- 172.21.15.28:0/2332031600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e01027a0 msgr2=0x7f06e01067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 --2- 172.21.15.28:0/2332031600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e01027a0 0x7f06e01067f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009900 tx=0x7f06c802a3f0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 -- 172.21.15.28:0/2332031600 shutdown_connections 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 --2- 172.21.15.28:0/2332031600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e0106f20 0x7f06e010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 --2- 172.21.15.28:0/2332031600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e01027a0 0x7f06e01067f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 --2- 172.21.15.28:0/2332031600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e0101df0 0x7f06e01021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 -- 172.21.15.28:0/2332031600 >> 172.21.15.28:0/2332031600 conn(0x7f06e00fd8f0 msgr2=0x7f06e00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 -- 172.21.15.28:0/2332031600 shutdown_connections 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 -- 172.21.15.28:0/2332031600 wait complete. 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.600+0000 7f06e48a7640 1 Processor -- start 2024-09-15T21:56:04.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06e48a7640 1 -- start start 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06e48a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e0101df0 0x7f06e0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06e48a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e01027a0 0x7f06e0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06e48a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e0106f20 0x7f06e019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06e48a7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0111f80 con 0x7f06e0106f20 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06e48a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0111e00 con 0x7f06e01027a0 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06e48a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0112100 con 0x7f06e0101df0 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e0106f20 0x7f06e019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e0106f20 0x7f06e019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37480/0 (socket says 172.21.15.28:37480) 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ddd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e01027a0 0x7f06e0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 -- 172.21.15.28:0/1636830246 learned_addr learned my addr 172.21.15.28:0/1636830246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06de59c640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e0101df0 0x7f06e0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 -- 172.21.15.28:0/1636830246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e0101df0 msgr2=0x7f06e0198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06e0101df0 0x7f06e0198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 -- 172.21.15.28:0/1636830246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e01027a0 msgr2=0x7f06e0198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06e01027a0 0x7f06e0198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 -- 172.21.15.28:0/1636830246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e019d3b0 con 0x7f06e0106f20 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.601+0000 7f06ded9d640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e0106f20 0x7f06e019ccd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f06d000cce0 tx=0x7f06d000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.602+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d000f450 con 0x7f06e0106f20 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.602+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d000fad0 con 0x7f06e0106f20 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.602+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e019d6a0 con 0x7f06e0106f20 2024-09-15T21:56:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.602+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0018c10 con 0x7f06e0106f20 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.602+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e01a4f80 con 0x7f06e0106f20 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.603+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f06e0062940 con 0x7f06e0106f20 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.604+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f06d000f5f0 con 0x7f06e0106f20 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06d77fe640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06cc0a3350 0x7f06cc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f06d00d05b0 con 0x7f06e0106f20 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06de59c640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06cc0a3350 0x7f06cc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06d77fe640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] conn(0x7f06cc0ae520 0x7f06cc0b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 --> [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06d000c120 con 0x7f06cc0ae520 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f06d00d0a00 con 0x7f06e0106f20 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06de59c640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06cc0a3350 0x7f06cc0a5810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f06d800bbe0 tx=0x7f06d800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.605+0000 7f06ddd9b640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] conn(0x7f06cc0ae520 0x7f06cc0b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.606+0000 7f06ddd9b640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] conn(0x7f06cc0ae520 0x7f06cc0b09c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.607+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== osd.4 v2:172.21.15.174:6808/3085836897 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d000c120 con 0x7f06cc0ae520 2024-09-15T21:56:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.621+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 --> [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f06e0062dc0 con 0x7f06cc0ae520 2024-09-15T21:56:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.621+0000 7f06d77fe640 1 -- 172.21.15.28:0/1636830246 <== osd.4 v2:172.21.15.174:6808/3085836897 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f06e0062dc0 con 0x7f06cc0ae520 2024-09-15T21:56:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 >> [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] conn(0x7f06cc0ae520 msgr2=0x7f06cc0b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] conn(0x7f06cc0ae520 0x7f06cc0b09c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06cc0a3350 msgr2=0x7f06cc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f06cc0a3350 0x7f06cc0a5810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f06d800bbe0 tx=0x7f06d800b390 comp rx=0 tx=0).stop 2024-09-15T21:56:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e0106f20 msgr2=0x7f06e019ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 --2- 172.21.15.28:0/1636830246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06e0106f20 0x7f06e019ccd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f06d000cce0 tx=0x7f06d000a620 comp rx=0 tx=0).stop 2024-09-15T21:56:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06ded9d640 1 -- 172.21.15.28:0/1636830246 reap_dead start 2024-09-15T21:56:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 shutdown_connections 2024-09-15T21:56:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 >> 172.21.15.28:0/1636830246 conn(0x7f06e00fd8f0 msgr2=0x7f06e00fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 shutdown_connections 2024-09-15T21:56:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:04.622+0000 7f06e48a7640 1 -- 172.21.15.28:0/1636830246 wait complete. 2024-09-15T21:56:04.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 -- 172.21.15.28:0/375875986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 msgr2=0x7f284c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 --2- 172.21.15.28:0/375875986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 0x7f284c10e610 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2844009ca0 tx=0x7f284402a950 comp rx=0 tx=0).stop 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 -- 172.21.15.28:0/375875986 shutdown_connections 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 --2- 172.21.15.28:0/375875986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 0x7f284c10e610 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 --2- 172.21.15.28:0/375875986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f284c103660 0x7f284c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 --2- 172.21.15.28:0/375875986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f284c102d40 0x7f284c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 -- 172.21.15.28:0/375875986 >> 172.21.15.28:0/375875986 conn(0x7f284c0fc7a0 msgr2=0x7f284c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 -- 172.21.15.28:0/375875986 shutdown_connections 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.777+0000 7f2851e56640 1 -- 172.21.15.28:0/375875986 wait complete. 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 Processor -- start 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 -- start start 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f284c102d40 0x7f284c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f284c103660 0x7f284c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 0x7f284c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f284c111f30 con 0x7f284c10c170 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f284c111db0 con 0x7f284c103660 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.778+0000 7f2851e56640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f284c1120b0 con 0x7f284c102d40 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 0x7f284c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 0x7f284c101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37450/0 (socket says 172.21.15.28:37450) 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f284c103660 0x7f284c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 -- 172.21.15.28:0/4189424013 learned_addr learned my addr 172.21.15.28:0/4189424013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284b7fe640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f284c102d40 0x7f284c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 -- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f284c102d40 msgr2=0x7f284c1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f284c102d40 0x7f284c1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 -- 172.21.15.28:0/4189424013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f284c103660 msgr2=0x7f284c100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f284c103660 0x7f284c100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 -- 172.21.15.28:0/4189424013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c1a6820 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284b7fe640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f284c102d40 0x7f284c1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f284bfff640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 0x7f284c101420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2844004af0 tx=0x7f284402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844034040 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f284c1a6ab0 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2844030920 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28440047b0 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.779+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f284c1a6f90 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.780+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f284c0fed90 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.782+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2844037020 con 0x7f284c10c170 2024-09-15T21:56:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.782+0000 7f2848ff9640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f28340a32d0 0x7f28340a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.782+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f28440f0c30 con 0x7f284c10c170 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.782+0000 7f284b7fe640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f28340a32d0 0x7f28340a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.783+0000 7f2848ff9640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] conn(0x7f28340ae4a0 0x7f28340b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.783+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 --> [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28340b1010 con 0x7f28340ae4a0 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.783+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f28440f1080 con 0x7f284c10c170 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.783+0000 7f284b7fe640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f28340a32d0 0x7f28340a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f283c00abe0 tx=0x7f283c00a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.783+0000 7f284affd640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] conn(0x7f28340ae4a0 0x7f28340b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.783+0000 7f284affd640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] conn(0x7f28340ae4a0 0x7f28340b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.784+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== osd.11 v2:172.21.15.203:6824/2779316485 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f28340b1010 con 0x7f28340ae4a0 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.798+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 --> [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f284c06b070 con 0x7f28340ae4a0 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.798+0000 7f2848ff9640 1 -- 172.21.15.28:0/4189424013 <== osd.11 v2:172.21.15.203:6824/2779316485 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f284c06b070 con 0x7f28340ae4a0 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] conn(0x7f28340ae4a0 msgr2=0x7f28340b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] conn(0x7f28340ae4a0 0x7f28340b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f28340a32d0 msgr2=0x7f28340a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f28340a32d0 0x7f28340a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f283c00abe0 tx=0x7f283c00a5c0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 msgr2=0x7f284c101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 --2- 172.21.15.28:0/4189424013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f284c10c170 0x7f284c101420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2844004af0 tx=0x7f284402f5b0 comp rx=0 tx=0).stop 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f284bfff640 1 -- 172.21.15.28:0/4189424013 reap_dead start 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 shutdown_connections 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 >> 172.21.15.28:0/4189424013 conn(0x7f284c0fc7a0 msgr2=0x7f284c0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 shutdown_connections 2024-09-15T21:56:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:03.799+0000 7f2851e56640 1 -- 172.21.15.28:0/4189424013 wait complete. 2024-09-15T21:56:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:05.161 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:04 smithi028 ceph-mon[30969]: pgmap v323: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 383 B/s wr, 9 op/s; 101 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:04 smithi203 ceph-mon[35546]: pgmap v323: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 383 B/s wr, 9 op/s; 101 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:04 smithi174 ceph-mon[35244]: pgmap v323: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 383 B/s wr, 9 op/s; 101 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:05.743 INFO:teuthology.orchestra.run.smithi028.stdout:270582939685 2024-09-15T21:56:05.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-15T21:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 -- 172.21.15.28:0/2282783716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 msgr2=0x7fb888103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 --2- 172.21.15.28:0/2282783716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 0x7fb888103170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb878009930 tx=0x7fb87802a420 comp rx=0 tx=0).stop 2024-09-15T21:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 -- 172.21.15.28:0/2282783716 shutdown_connections 2024-09-15T21:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 --2- 172.21.15.28:0/2282783716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb888108290 0x7fb88810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 --2- 172.21.15.28:0/2282783716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb888103740 0x7fb888107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 --2- 172.21.15.28:0/2282783716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 0x7fb888103170 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 -- 172.21.15.28:0/2282783716 >> 172.21.15.28:0/2282783716 conn(0x7fb8880fc7d0 msgr2=0x7fb8880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 -- 172.21.15.28:0/2282783716 shutdown_connections 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.923+0000 7fb88cb09640 1 -- 172.21.15.28:0/2282783716 wait complete. 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 Processor -- start 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 -- start start 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 0x7fb8881009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb888103740 0x7fb888100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb888108290 0x7fb888101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb888112090 con 0x7fb888102d90 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb888111f10 con 0x7fb888108290 2024-09-15T21:56:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.924+0000 7fb88cb09640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb888112210 con 0x7fb888103740 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 0x7fb8881009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 0x7fb8881009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37500/0 (socket says 172.21.15.28:37500) 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 -- 172.21.15.28:0/1446764049 learned_addr learned my addr 172.21.15.28:0/1446764049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb885d9b640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb888103740 0x7fb888100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb886d9d640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb888108290 0x7fb888101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 -- 172.21.15.28:0/1446764049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb888103740 msgr2=0x7fb888100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb888103740 0x7fb888100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 -- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb888108290 msgr2=0x7fb888101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb888108290 0x7fb888101470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 -- 172.21.15.28:0/1446764049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8881a69d0 con 0x7fb888102d90 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88659c640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 0x7fb8881009f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb878009230 tx=0x7fb878004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:05.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87803f030 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8881a6c60 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb878003ad0 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878035070 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.925+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8881a7190 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.926+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb8880628c0 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.929+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb878042030 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.929+0000 7fb8777fe640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8700a3350 0x7fb8700a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.929+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fb8780f0730 con 0x7fb888102d90 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.929+0000 7fb885d9b640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8700a3350 0x7fb8700a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.929+0000 7fb8777fe640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] conn(0x7fb8700ae520 0x7fb8700b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.929+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 --> [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8700b1090 con 0x7fb8700ae520 2024-09-15T21:56:05.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.929+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb8780f0b80 con 0x7fb888102d90 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.930+0000 7fb886d9d640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] conn(0x7fb8700ae520 0x7fb8700b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.930+0000 7fb885d9b640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8700a3350 0x7fb8700a5810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb888102480 tx=0x7fb87c005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.930+0000 7fb886d9d640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] conn(0x7fb8700ae520 0x7fb8700b09c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.931+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== osd.7 v2:172.21.15.174:6816/1679850033 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb8700b1090 con 0x7fb8700ae520 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.945+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 --> [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb888062cc0 con 0x7fb8700ae520 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.946+0000 7fb8777fe640 1 -- 172.21.15.28:0/1446764049 <== osd.7 v2:172.21.15.174:6816/1679850033 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb888062cc0 con 0x7fb8700ae520 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.946+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] conn(0x7fb8700ae520 msgr2=0x7fb8700b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.946+0000 7fb88cb09640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] conn(0x7fb8700ae520 0x7fb8700b09c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.946+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8700a3350 msgr2=0x7fb8700a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.946+0000 7fb88cb09640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8700a3350 0x7fb8700a5810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb888102480 tx=0x7fb87c005cc0 comp rx=0 tx=0).stop 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.946+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 msgr2=0x7fb8881009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.946+0000 7fb88cb09640 1 --2- 172.21.15.28:0/1446764049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb888102d90 0x7fb8881009f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb878009230 tx=0x7fb878004270 comp rx=0 tx=0).stop 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.947+0000 7fb886d9d640 1 -- 172.21.15.28:0/1446764049 reap_dead start 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.947+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 shutdown_connections 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.947+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 >> 172.21.15.28:0/1446764049 conn(0x7fb8880fc7d0 msgr2=0x7fb888106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.947+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 shutdown_connections 2024-09-15T21:56:05.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:05.947+0000 7fb88cb09640 1 -- 172.21.15.28:0/1446764049 wait complete. 2024-09-15T21:56:06.658 INFO:teuthology.orchestra.run.smithi028.stdout:188978561075 2024-09-15T21:56:06.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-15T21:56:06.887 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:06 smithi028 ceph-mon[30969]: pgmap v324: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 11 op/s; 101 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:07.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:06 smithi203 ceph-mon[35546]: pgmap v324: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 11 op/s; 101 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:07.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:06 smithi174 ceph-mon[35244]: pgmap v324: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 11 op/s; 101 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:08.135 INFO:teuthology.orchestra.run.smithi028.stdout:210453397552 2024-09-15T21:56:08.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-15T21:56:08.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:08 smithi028 ceph-mon[30969]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 314 B/s wr, 10 op/s; 83 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 -- 172.21.15.28:0/3536557101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 msgr2=0x7f4c08106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 --2- 172.21.15.28:0/3536557101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 0x7f4c08106f80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009900 tx=0x7f4c0402a410 comp rx=0 tx=0).stop 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 -- 172.21.15.28:0/3536557101 shutdown_connections 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 --2- 172.21.15.28:0/3536557101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c08100a70 0x7f4c0810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 --2- 172.21.15.28:0/3536557101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c081000d0 0x7f4c08100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 --2- 172.21.15.28:0/3536557101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 0x7f4c08106f80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 -- 172.21.15.28:0/3536557101 >> 172.21.15.28:0/3536557101 conn(0x7f4c08077210 msgr2=0x7f4c080fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 -- 172.21.15.28:0/3536557101 shutdown_connections 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.065+0000 7f4c10234640 1 -- 172.21.15.28:0/3536557101 wait complete. 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 Processor -- start 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 -- start start 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c081000d0 0x7f4c081012b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c08100a70 0x7f4c081017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 0x7f4c08101d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081120b0 con 0x7f4c08106ba0 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c08111f30 con 0x7f4c081000d0 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.066+0000 7f4c10234640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4c08112230 con 0x7f4c08100a70 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 0x7f4c08101d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 0x7f4c08101d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37584/0 (socket says 172.21.15.28:37584) 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 -- 172.21.15.28:0/967611307 learned_addr learned my addr 172.21.15.28:0/967611307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0dfd0640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c081000d0 0x7f4c081012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0d7cf640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c08100a70 0x7f4c081017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 -- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c08100a70 msgr2=0x7f4c081017f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c08100a70 0x7f4c081017f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 -- 172.21.15.28:0/967611307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c081000d0 msgr2=0x7f4c081012b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c081000d0 0x7f4c081012b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 -- 172.21.15.28:0/967611307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c081a69f0 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0d7cf640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c08100a70 0x7f4c081017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0dfd0640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c081000d0 0x7f4c081012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c0e7d1640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 0x7f4c08101d30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000b970 tx=0x7f4c0000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00019070 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c081a6ce0 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c0000cb40 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00007650 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.067+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c081a71c0 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.068+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4c08062880 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.069+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4c0001c070 con 0x7f4c08106ba0 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.070+0000 7f4bfeffd640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bf40a32d0 0x7f4bf40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.070+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f4c000cfbd0 con 0x7f4c08106ba0 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.070+0000 7f4c0dfd0640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bf40a32d0 0x7f4bf40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.070+0000 7f4bfeffd640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] conn(0x7f4bf40ae4a0 0x7f4bf40b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.070+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 --> [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c000047d0 con 0x7f4bf40ae4a0 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.070+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f4c000cff80 con 0x7f4c08106ba0 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.071+0000 7f4c0dfd0640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bf40a32d0 0x7f4bf40a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4c04005ab0 tx=0x7f4c04005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.071+0000 7f4c0d7cf640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] conn(0x7f4bf40ae4a0 0x7f4bf40b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.071+0000 7f4c0d7cf640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] conn(0x7f4bf40ae4a0 0x7f4bf40b0940 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.072+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== osd.2 v2:172.21.15.203:6800/2523083594 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c000047d0 con 0x7f4bf40ae4a0 2024-09-15T21:56:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.086+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 --> [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4c08062c80 con 0x7f4bf40ae4a0 2024-09-15T21:56:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.086+0000 7f4bfeffd640 1 -- 172.21.15.28:0/967611307 <== osd.2 v2:172.21.15.203:6800/2523083594 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4c08062c80 con 0x7f4bf40ae4a0 2024-09-15T21:56:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] conn(0x7f4bf40ae4a0 msgr2=0x7f4bf40b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] conn(0x7f4bf40ae4a0 0x7f4bf40b0940 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bf40a32d0 msgr2=0x7f4bf40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bf40a32d0 0x7f4bf40a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4c04005ab0 tx=0x7f4c04005a20 comp rx=0 tx=0).stop 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 msgr2=0x7f4c08101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 --2- 172.21.15.28:0/967611307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c08106ba0 0x7f4c08101d30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000b970 tx=0x7f4c0000be30 comp rx=0 tx=0).stop 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c0e7d1640 1 -- 172.21.15.28:0/967611307 reap_dead start 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 shutdown_connections 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 >> 172.21.15.28:0/967611307 conn(0x7f4c08077210 msgr2=0x7f4c08104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 shutdown_connections 2024-09-15T21:56:09.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:09.087+0000 7f4c10234640 1 -- 172.21.15.28:0/967611307 wait complete. 2024-09-15T21:56:09.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:08 smithi203 ceph-mon[35546]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 314 B/s wr, 10 op/s; 83 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:09.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:08 smithi174 ceph-mon[35244]: pgmap v325: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 314 B/s wr, 10 op/s; 83 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T21:56:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 -- 172.21.15.28:0/404608707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4106a80 msgr2=0x7fcef4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 --2- 172.21.15.28:0/404608707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4106a80 0x7fcef4106e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009930 tx=0x7fcedc02a420 comp rx=0 tx=0).stop 2024-09-15T21:56:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 -- 172.21.15.28:0/404608707 shutdown_connections 2024-09-15T21:56:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 --2- 172.21.15.28:0/404608707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcef4101420 0x7fcef410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 --2- 172.21.15.28:0/404608707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcef4100a80 0x7fcef4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 --2- 172.21.15.28:0/404608707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4106a80 0x7fcef4106e60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 -- 172.21.15.28:0/404608707 >> 172.21.15.28:0/404608707 conn(0x7fcef40fc7a0 msgr2=0x7fcef40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 -- 172.21.15.28:0/404608707 shutdown_connections 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.947+0000 7fcefae22640 1 -- 172.21.15.28:0/404608707 wait complete. 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 Processor -- start 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 -- start start 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100a80 0x7fcef4198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcef4101420 0x7fcef41989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcef4106a80 0x7fcef419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4110e80 con 0x7fcef4100a80 2024-09-15T21:56:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4110d00 con 0x7fcef4106a80 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcefae22640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4111000 con 0x7fcef4101420 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcef8bbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100a80 0x7fcef4198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcef8bbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100a80 0x7fcef4198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37566/0 (socket says 172.21.15.28:37566) 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcef8bbe640 1 -- 172.21.15.28:0/1845971742 learned_addr learned my addr 172.21.15.28:0/1845971742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcef3fff640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcef4101420 0x7fcef41989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.948+0000 7fcef93bf640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcef4106a80 0x7fcef419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef8bbe640 1 -- 172.21.15.28:0/1845971742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcef4101420 msgr2=0x7fcef41989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef8bbe640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcef4101420 0x7fcef41989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef8bbe640 1 -- 172.21.15.28:0/1845971742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcef4106a80 msgr2=0x7fcef419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef8bbe640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcef4106a80 0x7fcef419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef8bbe640 1 -- 172.21.15.28:0/1845971742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef419d440 con 0x7fcef4100a80 2024-09-15T21:56:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef8bbe640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100a80 0x7fcef4198490 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009230 tx=0x7fcedc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc03f030 con 0x7fcef4100a80 2024-09-15T21:56:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef419d6d0 con 0x7fcef4100a80 2024-09-15T21:56:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcedc003ad0 con 0x7fcef4100a80 2024-09-15T21:56:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc035070 con 0x7fcef4100a80 2024-09-15T21:56:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.949+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef41a5010 con 0x7fcef4100a80 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.950+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcef4105af0 con 0x7fcef4100a80 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.951+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fcedc042030 con 0x7fcef4100a80 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.952+0000 7fcef1ffb640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcee00a32d0 0x7fcee00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.952+0000 7fcef3fff640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcee00a32d0 0x7fcee00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.952+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fcedc0f0940 con 0x7fcef4100a80 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.952+0000 7fcef1ffb640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] conn(0x7fcee00ae4a0 0x7fcee00b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.952+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 --> [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcee00b1010 con 0x7fcee00ae4a0 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.952+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fcedc0f0cf0 con 0x7fcef4100a80 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.953+0000 7fcef93bf640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] conn(0x7fcee00ae4a0 0x7fcee00b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.953+0000 7fcef3fff640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcee00a32d0 0x7fcee00a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcef40fed70 tx=0x7fcee4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.953+0000 7fcef93bf640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] conn(0x7fcee00ae4a0 0x7fcee00b0940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.954+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== osd.0 v2:172.21.15.28:6802/896252218 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcee00b1010 con 0x7fcee00ae4a0 2024-09-15T21:56:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.967+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 --> [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcef41021c0 con 0x7fcee00ae4a0 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.967+0000 7fcef1ffb640 1 -- 172.21.15.28:0/1845971742 <== osd.0 v2:172.21.15.28:6802/896252218 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcef41021c0 con 0x7fcee00ae4a0 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] conn(0x7fcee00ae4a0 msgr2=0x7fcee00b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] conn(0x7fcee00ae4a0 0x7fcee00b0940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcee00a32d0 msgr2=0x7fcee00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcee00a32d0 0x7fcee00a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcef40fed70 tx=0x7fcee4009500 comp rx=0 tx=0).stop 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100a80 msgr2=0x7fcef4198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 --2- 172.21.15.28:0/1845971742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100a80 0x7fcef4198490 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009230 tx=0x7fcedc004270 comp rx=0 tx=0).stop 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcef93bf640 1 -- 172.21.15.28:0/1845971742 reap_dead start 2024-09-15T21:56:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 shutdown_connections 2024-09-15T21:56:09.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 >> 172.21.15.28:0/1845971742 conn(0x7fcef40fc7a0 msgr2=0x7fcef406a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:09.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 shutdown_connections 2024-09-15T21:56:09.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:08.968+0000 7fcefae22640 1 -- 172.21.15.28:0/1845971742 wait complete. 2024-09-15T21:56:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.504+0000 7f3c78772640 1 -- 172.21.15.28:0/689489480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 msgr2=0x7f3c70103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.504+0000 7f3c78772640 1 --2- 172.21.15.28:0/689489480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 0x7f3c70103190 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009930 tx=0x7f3c6c02a420 comp rx=0 tx=0).stop 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.504+0000 7f3c78772640 1 -- 172.21.15.28:0/689489480 shutdown_connections 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.504+0000 7f3c78772640 1 --2- 172.21.15.28:0/689489480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c701082e0 0x7f3c7010ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.504+0000 7f3c78772640 1 --2- 172.21.15.28:0/689489480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70103760 0x7f3c70107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.504+0000 7f3c78772640 1 --2- 172.21.15.28:0/689489480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 0x7f3c70103190 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.504+0000 7f3c78772640 1 -- 172.21.15.28:0/689489480 >> 172.21.15.28:0/689489480 conn(0x7f3c700fc7d0 msgr2=0x7f3c700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.505+0000 7f3c78772640 1 -- 172.21.15.28:0/689489480 shutdown_connections 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.505+0000 7f3c78772640 1 -- 172.21.15.28:0/689489480 wait complete. 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 Processor -- start 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 -- start start 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 0x7f3c70100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c70103760 0x7f3c70100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c701082e0 0x7f3c70101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c701120b0 con 0x7f3c70102db0 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c70111f30 con 0x7f3c70103760 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c78772640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3c70112230 con 0x7f3c701082e0 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c7650e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 0x7f3c70100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c7650e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 0x7f3c70100a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37526/0 (socket says 172.21.15.28:37526) 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.506+0000 7f3c7650e640 1 -- 172.21.15.28:0/2473587612 learned_addr learned my addr 172.21.15.28:0/2473587612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c75d0d640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c70103760 0x7f3c70100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c76d0f640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c701082e0 0x7f3c70101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c7650e640 1 -- 172.21.15.28:0/2473587612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c701082e0 msgr2=0x7f3c70101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c7650e640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c701082e0 0x7f3c70101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c7650e640 1 -- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c70103760 msgr2=0x7f3c70100f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c7650e640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c70103760 0x7f3c70100f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c7650e640 1 -- 172.21.15.28:0/2473587612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c701a6b40 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c75d0d640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c70103760 0x7f3c70100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c7650e640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 0x7f3c70100a10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009230 tx=0x7f3c6c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c03f070 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c701a6dd0 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c6c031e30 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.507+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c035030 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.508+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c701a7270 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.508+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c70062900 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.510+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3c6c042070 con 0x7f3c70102db0 2024-09-15T21:56:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.511+0000 7f3c637fe640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3c5c0a3350 0x7f3c5c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.511+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f3c6c0f0b30 con 0x7f3c70102db0 2024-09-15T21:56:09.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.511+0000 7f3c75d0d640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3c5c0a3350 0x7f3c5c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.511+0000 7f3c637fe640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] conn(0x7f3c5c0ae520 0x7f3c5c0b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:09.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.511+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 --> [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c5c0b1090 con 0x7f3c5c0ae520 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.511+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f3c6c0f0f80 con 0x7f3c70102db0 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.512+0000 7f3c75d0d640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3c5c0a3350 0x7f3c5c0a5810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3c700feeb0 tx=0x7f3c6400a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.512+0000 7f3c76d0f640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] conn(0x7f3c5c0ae520 0x7f3c5c0b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.512+0000 7f3c76d0f640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] conn(0x7f3c5c0ae520 0x7f3c5c0b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.513+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== osd.1 v2:172.21.15.174:6800/4110616785 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c5c0b1090 con 0x7f3c5c0ae520 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.527+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 --> [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c70062d00 con 0x7f3c5c0ae520 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c637fe640 1 -- 172.21.15.28:0/2473587612 <== osd.1 v2:172.21.15.174:6800/4110616785 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3c70062d00 con 0x7f3c5c0ae520 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] conn(0x7f3c5c0ae520 msgr2=0x7f3c5c0b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] conn(0x7f3c5c0ae520 0x7f3c5c0b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3c5c0a3350 msgr2=0x7f3c5c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3c5c0a3350 0x7f3c5c0a5810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3c700feeb0 tx=0x7f3c6400a250 comp rx=0 tx=0).stop 2024-09-15T21:56:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 msgr2=0x7f3c70100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 --2- 172.21.15.28:0/2473587612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c70102db0 0x7f3c70100a10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009230 tx=0x7f3c6c003c20 comp rx=0 tx=0).stop 2024-09-15T21:56:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c76d0f640 1 -- 172.21.15.28:0/2473587612 reap_dead start 2024-09-15T21:56:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 shutdown_connections 2024-09-15T21:56:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 >> 172.21.15.28:0/2473587612 conn(0x7f3c700fc7d0 msgr2=0x7f3c70105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.528+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 shutdown_connections 2024-09-15T21:56:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.529+0000 7f3c78772640 1 -- 172.21.15.28:0/2473587612 wait complete. 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.234+0000 7efc7dcae640 1 -- 172.21.15.28:0/1651777500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 msgr2=0x7efc7810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.234+0000 7efc7dcae640 1 --2- 172.21.15.28:0/1651777500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 0x7efc7810ea90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efc70009ca0 tx=0x7efc7002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.234+0000 7efc7dcae640 1 -- 172.21.15.28:0/1651777500 shutdown_connections 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.234+0000 7efc7dcae640 1 --2- 172.21.15.28:0/1651777500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 0x7efc7810ea90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.234+0000 7efc7dcae640 1 --2- 172.21.15.28:0/1651777500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc78102790 0x7efc781067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.234+0000 7efc7dcae640 1 --2- 172.21.15.28:0/1651777500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc78101de0 0x7efc781021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.234+0000 7efc7dcae640 1 -- 172.21.15.28:0/1651777500 >> 172.21.15.28:0/1651777500 conn(0x7efc780fd8c0 msgr2=0x7efc780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.235+0000 7efc7dcae640 1 -- 172.21.15.28:0/1651777500 shutdown_connections 2024-09-15T21:56:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.235+0000 7efc7dcae640 1 -- 172.21.15.28:0/1651777500 wait complete. 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.235+0000 7efc7dcae640 1 Processor -- start 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.235+0000 7efc7dcae640 1 -- start start 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc7dcae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc78101de0 0x7efc78198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc7dcae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc78102790 0x7efc78198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc7dcae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 0x7efc7819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc7dcae640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc78111f70 con 0x7efc78106f10 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc7dcae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efc78111df0 con 0x7efc78102790 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc7dcae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc781120f0 con 0x7efc78101de0 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 0x7efc7819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 0x7efc7819cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37602/0 (socket says 172.21.15.28:37602) 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 -- 172.21.15.28:0/1750608342 learned_addr learned my addr 172.21.15.28:0/1750608342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc777fe640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc78101de0 0x7efc78198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc76ffd640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc78102790 0x7efc78198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 -- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc78101de0 msgr2=0x7efc78198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc78101de0 0x7efc78198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 -- 172.21.15.28:0/1750608342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc78102790 msgr2=0x7efc78198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc78102790 0x7efc78198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 -- 172.21.15.28:0/1750608342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7819d3e0 con 0x7efc78106f10 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.236+0000 7efc77fff640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 0x7efc7819cd00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efc70031a20 tx=0x7efc70031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.237+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70003b70 con 0x7efc78106f10 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.237+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc70003d10 con 0x7efc78106f10 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.237+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7819d670 con 0x7efc78106f10 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.237+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7002fbd0 con 0x7efc78106f10 2024-09-15T21:56:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.237+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc781a4fb0 con 0x7efc78106f10 2024-09-15T21:56:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.238+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc780fee70 con 0x7efc78106f10 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.239+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7efc7002fd70 con 0x7efc78106f10 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.240+0000 7efc74ff9640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7efc600a3350 0x7efc600a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.240+0000 7efc777fe640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7efc600a3350 0x7efc600a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.240+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7efc700f0fe0 con 0x7efc78106f10 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.240+0000 7efc74ff9640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] conn(0x7efc600ae520 0x7efc600b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.240+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 --> [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc600b1090 con 0x7efc600ae520 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.240+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7efc700f1390 con 0x7efc78106f10 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.241+0000 7efc777fe640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7efc600a3350 0x7efc600a5810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efc6800abe0 tx=0x7efc6800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.241+0000 7efc76ffd640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] conn(0x7efc600ae520 0x7efc600b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.241+0000 7efc76ffd640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] conn(0x7efc600ae520 0x7efc600b09c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.242+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== osd.8 v2:172.21.15.203:6816/3629574 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efc600b1090 con 0x7efc600ae520 2024-09-15T21:56:10.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.256+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 --> [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efc781032c0 con 0x7efc600ae520 2024-09-15T21:56:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.256+0000 7efc74ff9640 1 -- 172.21.15.28:0/1750608342 <== osd.8 v2:172.21.15.203:6816/3629574 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efc781032c0 con 0x7efc600ae520 2024-09-15T21:56:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] conn(0x7efc600ae520 msgr2=0x7efc600b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] conn(0x7efc600ae520 0x7efc600b09c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7efc600a3350 msgr2=0x7efc600a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7efc600a3350 0x7efc600a5810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efc6800abe0 tx=0x7efc6800a2d0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 msgr2=0x7efc7819cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 --2- 172.21.15.28:0/1750608342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc78106f10 0x7efc7819cd00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efc70031a20 tx=0x7efc70031a50 comp rx=0 tx=0).stop 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc77fff640 1 -- 172.21.15.28:0/1750608342 reap_dead start 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 shutdown_connections 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 >> 172.21.15.28:0/1750608342 conn(0x7efc780fd8c0 msgr2=0x7efc780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 shutdown_connections 2024-09-15T21:56:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:10.257+0000 7efc7dcae640 1 -- 172.21.15.28:0/1750608342 wait complete. 2024-09-15T21:56:10.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:10 smithi028 ceph-mon[30969]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 285 B/s wr, 11 op/s; 75 B/s, 0 keys/s, 1 objects/s recovering 2024-09-15T21:56:10.874 INFO:teuthology.orchestra.run.smithi028.stdout:240518168619 2024-09-15T21:56:10.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 -- 172.21.15.28:0/3345864992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14102570 msgr2=0x7fbc141029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 --2- 172.21.15.28:0/3345864992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14102570 0x7fbc141029b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009900 tx=0x7fbc0402a480 comp rx=0 tx=0).stop 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 -- 172.21.15.28:0/3345864992 shutdown_connections 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 --2- 172.21.15.28:0/3345864992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc14102ef0 0x7fbc1410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 --2- 172.21.15.28:0/3345864992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14102570 0x7fbc141029b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 --2- 172.21.15.28:0/3345864992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc14100e20 0x7fbc14101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 -- 172.21.15.28:0/3345864992 >> 172.21.15.28:0/3345864992 conn(0x7fbc140fc7d0 msgr2=0x7fbc140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 -- 172.21.15.28:0/3345864992 shutdown_connections 2024-09-15T21:56:10.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.727+0000 7fbc19efa640 1 -- 172.21.15.28:0/3345864992 wait complete. 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 Processor -- start 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 -- start start 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14100e20 0x7fbc1419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc14102570 0x7fbc1419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc14102ef0 0x7fbc1419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14110f50 con 0x7fbc14100e20 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14110dd0 con 0x7fbc14102ef0 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.728+0000 7fbc19efa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc141110d0 con 0x7fbc14102570 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc14102570 0x7fbc1419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc14102570 0x7fbc1419ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:55872/0 (socket says 172.21.15.28:55872) 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 -- 172.21.15.28:0/2819626381 learned_addr learned my addr 172.21.15.28:0/2819626381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:10.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc137fe640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14100e20 0x7fbc1419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 -- 172.21.15.28:0/2819626381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc14102ef0 msgr2=0x7fbc1419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc14102ef0 0x7fbc1419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 -- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14100e20 msgr2=0x7fbc1419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14100e20 0x7fbc1419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc12ffd640 1 -- 172.21.15.28:0/2819626381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1419f6e0 con 0x7fbc14102570 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.729+0000 7fbc137fe640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc14100e20 0x7fbc1419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.730+0000 7fbc12ffd640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc14102570 0x7fbc1419ac70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc04006010 tx=0x7fbc04003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.730+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0403f070 con 0x7fbc14102570 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.730+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1419f970 con 0x7fbc14102570 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.730+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc04031c30 con 0x7fbc14102570 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.730+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04035030 con 0x7fbc14102570 2024-09-15T21:56:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.730+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc141a72b0 con 0x7fbc14102570 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.732+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbc14106a40 con 0x7fbc14102570 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.734+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbc04042070 con 0x7fbc14102570 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.735+0000 7fbc10ff9640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbc000a3350 0x7fbc000a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.736+0000 7fbc137fe640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbc000a3350 0x7fbc000a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.736+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fbc040f0fe0 con 0x7fbc14102570 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.736+0000 7fbc10ff9640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] conn(0x7fbc000ae520 0x7fbc000b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.736+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 --> [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc04037ec0 con 0x7fbc000ae520 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.736+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbc040f1430 con 0x7fbc14102570 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.736+0000 7fbc137fe640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbc000a3350 0x7fbc000a5810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc005d00 tx=0x7fbbfc009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.736+0000 7fbc13fff640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] conn(0x7fbc000ae520 0x7fbc000b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.737+0000 7fbc13fff640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] conn(0x7fbc000ae520 0x7fbc000b09c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.738+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== osd.10 v2:172.21.15.174:6824/2504623164 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc04037ec0 con 0x7fbc000ae520 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.751+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 --> [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbc141021b0 con 0x7fbc000ae520 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.752+0000 7fbc10ff9640 1 -- 172.21.15.28:0/2819626381 <== osd.10 v2:172.21.15.174:6824/2504623164 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbc141021b0 con 0x7fbc000ae520 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.752+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 >> [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] conn(0x7fbc000ae520 msgr2=0x7fbc000b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.752+0000 7fbc19efa640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] conn(0x7fbc000ae520 0x7fbc000b09c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.752+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbc000a3350 msgr2=0x7fbc000a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc19efa640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbc000a3350 0x7fbc000a5810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc005d00 tx=0x7fbbfc009550 comp rx=0 tx=0).stop 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc14102570 msgr2=0x7fbc1419ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc19efa640 1 --2- 172.21.15.28:0/2819626381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc14102570 0x7fbc1419ac70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc04006010 tx=0x7fbc04003c10 comp rx=0 tx=0).stop 2024-09-15T21:56:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc13fff640 1 -- 172.21.15.28:0/2819626381 reap_dead start 2024-09-15T21:56:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 shutdown_connections 2024-09-15T21:56:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 >> 172.21.15.28:0/2819626381 conn(0x7fbc140fc7d0 msgr2=0x7fbc1406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 shutdown_connections 2024-09-15T21:56:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:07.753+0000 7fbc19efa640 1 -- 172.21.15.28:0/2819626381 wait complete. 2024-09-15T21:56:11.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:10 smithi203 ceph-mon[35546]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 285 B/s wr, 11 op/s; 75 B/s, 0 keys/s, 1 objects/s recovering 2024-09-15T21:56:11.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:10 smithi174 ceph-mon[35244]: pgmap v326: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 285 B/s wr, 11 op/s; 75 B/s, 0 keys/s, 1 objects/s recovering 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 -- 172.21.15.28:0/1302314668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc102790 msgr2=0x7f16dc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 --2- 172.21.15.28:0/1302314668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc102790 0x7f16dc1067e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f16d0009900 tx=0x7f16d002a410 comp rx=0 tx=0).stop 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 -- 172.21.15.28:0/1302314668 shutdown_connections 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 --2- 172.21.15.28:0/1302314668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16dc106f10 0x7f16dc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 --2- 172.21.15.28:0/1302314668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc102790 0x7f16dc1067e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 --2- 172.21.15.28:0/1302314668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16dc101de0 0x7f16dc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 -- 172.21.15.28:0/1302314668 >> 172.21.15.28:0/1302314668 conn(0x7f16dc0fd8c0 msgr2=0x7f16dc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 -- 172.21.15.28:0/1302314668 shutdown_connections 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.388+0000 7f16e3b99640 1 -- 172.21.15.28:0/1302314668 wait complete. 2024-09-15T21:56:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 Processor -- start 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 -- start start 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc101de0 0x7f16dc1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16dc102790 0x7f16dc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16dc106f10 0x7f16dc19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc111f70 con 0x7f16dc101de0 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc111df0 con 0x7f16dc102790 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e3b99640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc1120f0 con 0x7f16dc106f10 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc101de0 0x7f16dc1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc101de0 0x7f16dc1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47446/0 (socket says 172.21.15.28:47446) 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 -- 172.21.15.28:0/2199496724 learned_addr learned my addr 172.21.15.28:0/2199496724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e2136640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16dc106f10 0x7f16dc19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:11.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1134640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16dc102790 0x7f16dc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 -- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16dc106f10 msgr2=0x7f16dc19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16dc106f10 0x7f16dc19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 -- 172.21.15.28:0/2199496724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16dc102790 msgr2=0x7f16dc198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16dc102790 0x7f16dc198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e1935640 1 -- 172.21.15.28:0/2199496724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16dc19d380 con 0x7f16dc101de0 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.389+0000 7f16e2136640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16dc106f10 0x7f16dc19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.390+0000 7f16e1935640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc101de0 0x7f16dc1983d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f16d400d770 tx=0x7f16d400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.390+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4017070 con 0x7f16dc101de0 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.390+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16d4005500 con 0x7f16dc101de0 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.390+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16dc19d670 con 0x7f16dc101de0 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.390+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4004e00 con 0x7f16dc101de0 2024-09-15T21:56:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.390+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16dc1a4f50 con 0x7f16dc101de0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.391+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16dc062980 con 0x7f16dc101de0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.392+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f16d4004fa0 con 0x7f16dc101de0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.393+0000 7f16daffd640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f16c40a3350 0x7f16c40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.393+0000 7f16e1134640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f16c40a3350 0x7f16c40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.393+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f16d40cfc70 con 0x7f16dc101de0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.393+0000 7f16daffd640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] conn(0x7f16c40ae520 0x7f16c40b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.393+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 --> [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16d4001d30 con 0x7f16c40ae520 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.393+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f16d40d00c0 con 0x7f16dc101de0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.393+0000 7f16e1134640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f16c40a3350 0x7f16c40a5810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16dc1088e0 tx=0x7f16d0005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.394+0000 7f16e2136640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] conn(0x7f16c40ae520 0x7f16c40b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.394+0000 7f16e2136640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] conn(0x7f16c40ae520 0x7f16c40b09c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.395+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== osd.5 v2:172.21.15.203:6808/3676469409 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16d4001d30 con 0x7f16c40ae520 2024-09-15T21:56:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.409+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 --> [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f16dc062e00 con 0x7f16c40ae520 2024-09-15T21:56:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.409+0000 7f16daffd640 1 -- 172.21.15.28:0/2199496724 <== osd.5 v2:172.21.15.203:6808/3676469409 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f16dc062e00 con 0x7f16c40ae520 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] conn(0x7f16c40ae520 msgr2=0x7f16c40b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] conn(0x7f16c40ae520 0x7f16c40b09c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f16c40a3350 msgr2=0x7f16c40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f16c40a3350 0x7f16c40a5810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16dc1088e0 tx=0x7f16d0005a20 comp rx=0 tx=0).stop 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc101de0 msgr2=0x7f16dc1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 --2- 172.21.15.28:0/2199496724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16dc101de0 0x7f16dc1983d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f16d400d770 tx=0x7f16d400dc30 comp rx=0 tx=0).stop 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e2136640 1 -- 172.21.15.28:0/2199496724 reap_dead start 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 shutdown_connections 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 >> 172.21.15.28:0/2199496724 conn(0x7f16dc0fd8c0 msgr2=0x7f16dc0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 shutdown_connections 2024-09-15T21:56:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:11.410+0000 7f16e3b99640 1 -- 172.21.15.28:0/2199496724 wait complete. 2024-09-15T21:56:12.173 INFO:teuthology.orchestra.run.smithi028.stdout:137438953535 2024-09-15T21:56:12.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-15T21:56:12.577 INFO:teuthology.orchestra.run.smithi028.stdout:326417514523 2024-09-15T21:56:12.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-15T21:56:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:55834574927 2024-09-15T21:56:13.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-15T21:56:13.119 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:12 smithi028 ceph-mon[30969]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 67 B/s, 0 objects/s recovering 2024-09-15T21:56:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 -- 172.21.15.28:0/2377878820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700102570 msgr2=0x7f37001029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 --2- 172.21.15.28:0/2377878820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700102570 0x7f37001029b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009900 tx=0x7f36ec02a480 comp rx=0 tx=0).stop 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 -- 172.21.15.28:0/2377878820 shutdown_connections 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 --2- 172.21.15.28:0/2377878820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3700102ef0 0x7f370010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 --2- 172.21.15.28:0/2377878820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700102570 0x7f37001029b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 --2- 172.21.15.28:0/2377878820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3700100e20 0x7f3700101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 -- 172.21.15.28:0/2377878820 >> 172.21.15.28:0/2377878820 conn(0x7f37000fc7d0 msgr2=0x7f37000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 -- 172.21.15.28:0/2377878820 shutdown_connections 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.201+0000 7f3706d52640 1 -- 172.21.15.28:0/2377878820 wait complete. 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 Processor -- start 2024-09-15T21:56:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 -- start start 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700100e20 0x7f37000fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3700102570 0x7f37000fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3700102ef0 0x7f37000fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3700110f80 con 0x7f3700100e20 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3700110e00 con 0x7f3700102ef0 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3706d52640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3700111100 con 0x7f3700102570 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3704aee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700100e20 0x7f37000fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f36fffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3700102570 0x7f37000fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3704aee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700100e20 0x7f37000fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47468/0 (socket says 172.21.15.28:47468) 2024-09-15T21:56:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3704aee640 1 -- 172.21.15.28:0/2650010044 learned_addr learned my addr 172.21.15.28:0/2650010044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f37052ef640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3700102ef0 0x7f37000fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3704aee640 1 -- 172.21.15.28:0/2650010044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3700102570 msgr2=0x7f37000fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3704aee640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3700102570 0x7f37000fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3704aee640 1 -- 172.21.15.28:0/2650010044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3700102ef0 msgr2=0x7f37000fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.202+0000 7f3704aee640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3700102ef0 0x7f37000fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.203+0000 7f3704aee640 1 -- 172.21.15.28:0/2650010044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37001a6910 con 0x7f3700100e20 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.203+0000 7f3704aee640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700100e20 0x7f37000fcc80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f36f400d950 tx=0x7f36f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.203+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4017070 con 0x7f3700100e20 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.203+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36f4009c30 con 0x7f3700100e20 2024-09-15T21:56:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.203+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37001a6c00 con 0x7f3700100e20 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.203+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f400f6b0 con 0x7f3700100e20 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.203+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37001a7160 con 0x7f3700100e20 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.204+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3700107220 con 0x7f3700100e20 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.206+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f36f40109d0 con 0x7f3700100e20 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.207+0000 7f36fdffb640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f36f00a3350 0x7f36f00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.207+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f36f40cfd80 con 0x7f3700100e20 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.207+0000 7f36fffff640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f36f00a3350 0x7f36f00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.207+0000 7f36fdffb640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] conn(0x7f36f00ae520 0x7f36f00b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.207+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 --> [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36f401ce50 con 0x7f36f00ae520 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.207+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f36f40d0130 con 0x7f3700100e20 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.207+0000 7f37052ef640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] conn(0x7f36f00ae520 0x7f36f00b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.208+0000 7f36fffff640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f36f00a3350 0x7f36f00a5810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009230 tx=0x7f36ec0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.208+0000 7f37052ef640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] conn(0x7f36f00ae520 0x7f36f00b09c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.208+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== osd.3 v2:172.21.15.28:6810/1752867147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36f401ce50 con 0x7f36f00ae520 2024-09-15T21:56:13.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:12 smithi203 ceph-mon[35546]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 67 B/s, 0 objects/s recovering 2024-09-15T21:56:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.223+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 --> [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37001021b0 con 0x7f36f00ae520 2024-09-15T21:56:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f36fdffb640 1 -- 172.21.15.28:0/2650010044 <== osd.3 v2:172.21.15.28:6810/1752867147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f37001021b0 con 0x7f36f00ae520 2024-09-15T21:56:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] conn(0x7f36f00ae520 msgr2=0x7f36f00b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] conn(0x7f36f00ae520 0x7f36f00b09c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f36f00a3350 msgr2=0x7f36f00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f36f00a3350 0x7f36f00a5810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009230 tx=0x7f36ec0058d0 comp rx=0 tx=0).stop 2024-09-15T21:56:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700100e20 msgr2=0x7f37000fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 --2- 172.21.15.28:0/2650010044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700100e20 0x7f37000fcc80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f36f400d950 tx=0x7f36f400de10 comp rx=0 tx=0).stop 2024-09-15T21:56:13.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f37052ef640 1 -- 172.21.15.28:0/2650010044 reap_dead start 2024-09-15T21:56:13.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 shutdown_connections 2024-09-15T21:56:13.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.224+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 >> 172.21.15.28:0/2650010044 conn(0x7f37000fc7d0 msgr2=0x7f3700069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:13.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.225+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 shutdown_connections 2024-09-15T21:56:13.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:13.225+0000 7f3706d52640 1 -- 172.21.15.28:0/2650010044 wait complete. 2024-09-15T21:56:13.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:12 smithi174 ceph-mon[35244]: pgmap v327: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 67 B/s, 0 objects/s recovering 2024-09-15T21:56:13.807 INFO:teuthology.orchestra.run.smithi028.stdout:77309411403 2024-09-15T21:56:13.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-15T21:56:14.200 INFO:teuthology.orchestra.run.smithi028.stdout:292057776159 2024-09-15T21:56:14.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-15T21:56:14.660 INFO:teuthology.orchestra.run.smithi028.stdout:158913790012 2024-09-15T21:56:14.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-15T21:56:14.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:14 smithi028 ceph-mon[30969]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 58 B/s, 0 objects/s recovering 2024-09-15T21:56:15.012 INFO:teuthology.orchestra.run.smithi028.stdout:34359738451 2024-09-15T21:56:15.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-15T21:56:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:14 smithi203 ceph-mon[35546]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 58 B/s, 0 objects/s recovering 2024-09-15T21:56:15.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:14 smithi174 ceph-mon[35244]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 58 B/s, 0 objects/s recovering 2024-09-15T21:56:15.704 INFO:teuthology.orchestra.run.smithi028.stdout:107374182470 2024-09-15T21:56:15.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-15T21:56:15.753 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:15.763 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:16.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 -- 172.21.15.28:0/1470674049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58106a80 msgr2=0x7f3f58106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 --2- 172.21.15.28:0/1470674049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58106a80 0x7f3f58106e60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009930 tx=0x7f3f4802a420 comp rx=0 tx=0).stop 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 -- 172.21.15.28:0/1470674049 shutdown_connections 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 --2- 172.21.15.28:0/1470674049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f58101420 0x7f3f5810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 --2- 172.21.15.28:0/1470674049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f58100a80 0x7f3f58100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 --2- 172.21.15.28:0/1470674049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58106a80 0x7f3f58106e60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 -- 172.21.15.28:0/1470674049 >> 172.21.15.28:0/1470674049 conn(0x7f3f580fc7a0 msgr2=0x7f3f580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 -- 172.21.15.28:0/1470674049 shutdown_connections 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.266+0000 7f3f60481640 1 -- 172.21.15.28:0/1470674049 wait complete. 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 Processor -- start 2024-09-15T21:56:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 -- start start 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58100a80 0x7f3f581984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f58101420 0x7f3f58198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f58106a80 0x7f3f5819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58110ea0 con 0x7f3f58100a80 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58110d20 con 0x7f3f58101420 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f60481640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58111020 con 0x7f3f58106a80 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58100a80 0x7f3f581984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58100a80 0x7f3f581984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47486/0 (socket says 172.21.15.28:47486) 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 -- 172.21.15.28:0/2400285717 learned_addr learned my addr 172.21.15.28:0/2400285717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 -- 172.21.15.28:0/2400285717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f58106a80 msgr2=0x7f3f5819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f58106a80 0x7f3f5819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 -- 172.21.15.28:0/2400285717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f58101420 msgr2=0x7f3f58198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f58101420 0x7f3f58198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.267+0000 7f3f5e21d640 1 -- 172.21.15.28:0/2400285717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5819d480 con 0x7f3f58100a80 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.268+0000 7f3f5e21d640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58100a80 0x7f3f581984d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009230 tx=0x7f3f48004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.268+0000 7f3f577fe640 1 -- 172.21.15.28:0/2400285717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4803f030 con 0x7f3f58100a80 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.268+0000 7f3f577fe640 1 -- 172.21.15.28:0/2400285717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f48003ad0 con 0x7f3f58100a80 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.268+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5819d710 con 0x7f3f58100a80 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.268+0000 7f3f577fe640 1 -- 172.21.15.28:0/2400285717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48035070 con 0x7f3f58100a80 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.268+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f581a5050 con 0x7f3f58100a80 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.269+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f581021c0 con 0x7f3f58100a80 2024-09-15T21:56:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.270+0000 7f3f577fe640 1 -- 172.21.15.28:0/2400285717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3f48042030 con 0x7f3f58100a80 2024-09-15T21:56:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.271+0000 7f3f577fe640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3f400a32d0 0x7f3f400a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.271+0000 7f3f577fe640 1 -- 172.21.15.28:0/2400285717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f3f480f0380 con 0x7f3f58100a80 2024-09-15T21:56:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.271+0000 7f3f5da1c640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3f400a32d0 0x7f3f400a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.271+0000 7f3f5da1c640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3f400a32d0 0x7f3f400a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f58105170 tx=0x7f3f4c009250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.275+0000 7f3f577fe640 1 -- 172.21.15.28:0/2400285717 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3f480bcbe0 con 0x7f3f58100a80 2024-09-15T21:56:16.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.455+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3f5810a060 con 0x7f3f58100a80 2024-09-15T21:56:16.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.456+0000 7f3f577fe640 1 -- 172.21.15.28:0/2400285717 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3f480bc4e0 con 0x7f3f58100a80 2024-09-15T21:56:16.459 INFO:teuthology.orchestra.run.smithi028.stdout:137438953538 2024-09-15T21:56:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3f400a32d0 msgr2=0x7f3f400a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3f400a32d0 0x7f3f400a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f58105170 tx=0x7f3f4c009250 comp rx=0 tx=0).stop 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58100a80 msgr2=0x7f3f581984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58100a80 0x7f3f581984d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009230 tx=0x7f3f48004270 comp rx=0 tx=0).stop 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 shutdown_connections 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f3f400a32d0 0x7f3f400a5790 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f58106a80 0x7f3f5819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f58101420 0x7f3f58198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 --2- 172.21.15.28:0/2400285717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f58100a80 0x7f3f581984d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 >> 172.21.15.28:0/2400285717 conn(0x7f3f580fc7a0 msgr2=0x7f3f5806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 shutdown_connections 2024-09-15T21:56:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:16.458+0000 7f3f60481640 1 -- 172.21.15.28:0/2400285717 wait complete. 2024-09-15T21:56:17.023 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:16 smithi028 ceph-mon[30969]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-09-15T21:56:17.024 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:16 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2400285717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 -- 172.21.15.28:0/2332458676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 msgr2=0x7f1aec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 --2- 172.21.15.28:0/2332458676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec1067e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009900 tx=0x7f1ae002a410 comp rx=0 tx=0).stop 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 -- 172.21.15.28:0/2332458676 shutdown_connections 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 --2- 172.21.15.28:0/2332458676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec106f10 0x7f1aec10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 --2- 172.21.15.28:0/2332458676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec1067e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 --2- 172.21.15.28:0/2332458676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aec101de0 0x7f1aec1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 -- 172.21.15.28:0/2332458676 >> 172.21.15.28:0/2332458676 conn(0x7f1aec0fd8c0 msgr2=0x7f1aec0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 -- 172.21.15.28:0/2332458676 shutdown_connections 2024-09-15T21:56:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.183+0000 7f1af2064640 1 -- 172.21.15.28:0/2332458676 wait complete. 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 Processor -- start 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 -- start start 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec101de0 0x7f1aec1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aec106f10 0x7f1aec19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec111f90 con 0x7f1aec102790 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec111e10 con 0x7f1aec101de0 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1af2064640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec112110 con 0x7f1aec106f10 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.184+0000 7f1aeaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47504/0 (socket says 172.21.15.28:47504) 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 -- 172.21.15.28:0/635430675 learned_addr learned my addr 172.21.15.28:0/635430675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aebfff640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aec106f10 0x7f1aec19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeb7fe640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec101de0 0x7f1aec1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 -- 172.21.15.28:0/635430675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aec106f10 msgr2=0x7f1aec19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aec106f10 0x7f1aec19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 -- 172.21.15.28:0/635430675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec101de0 msgr2=0x7f1aec1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec101de0 0x7f1aec1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 -- 172.21.15.28:0/635430675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec19d480 con 0x7f1aec102790 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1aeaffd640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec198a10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0005ab0 tx=0x7f1ae00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1ae8ff9640 1 -- 172.21.15.28:0/635430675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae003f030 con 0x7f1aec102790 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1ae8ff9640 1 -- 172.21.15.28:0/635430675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae0003ce0 con 0x7f1aec102790 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aec19d710 con 0x7f1aec102790 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1ae8ff9640 1 -- 172.21.15.28:0/635430675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0035070 con 0x7f1aec102790 2024-09-15T21:56:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.185+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aec1a5050 con 0x7f1aec102790 2024-09-15T21:56:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.186+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aec1032c0 con 0x7f1aec102790 2024-09-15T21:56:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.187+0000 7f1ae8ff9640 1 -- 172.21.15.28:0/635430675 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1ae0042030 con 0x7f1aec102790 2024-09-15T21:56:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.188+0000 7f1ae8ff9640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f1ad40a32d0 0x7f1ad40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.189+0000 7f1aeb7fe640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f1ad40a32d0 0x7f1ad40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.189+0000 7f1ae8ff9640 1 -- 172.21.15.28:0/635430675 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f1ae00f0800 con 0x7f1aec102790 2024-09-15T21:56:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.189+0000 7f1aeb7fe640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f1ad40a32d0 0x7f1ad40a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1adc005e10 tx=0x7f1adc005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.193+0000 7f1ae8ff9640 1 -- 172.21.15.28:0/635430675 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1ae00bd000 con 0x7f1aec102790 2024-09-15T21:56:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:16 smithi203 ceph-mon[35546]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-09-15T21:56:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:16 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2400285717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T21:56:17.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:16 smithi174 ceph-mon[35244]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-09-15T21:56:17.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:16 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2400285717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T21:56:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.379+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1aec10a0f0 con 0x7f1aec102790 2024-09-15T21:56:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.379+0000 7f1ae8ff9640 1 -- 172.21.15.28:0/635430675 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1ae00bc900 con 0x7f1aec102790 2024-09-15T21:56:17.382 INFO:teuthology.orchestra.run.smithi028.stdout:188978561077 2024-09-15T21:56:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.381+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f1ad40a32d0 msgr2=0x7f1ad40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.381+0000 7f1af2064640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f1ad40a32d0 0x7f1ad40a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1adc005e10 tx=0x7f1adc005da0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.381+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 msgr2=0x7f1aec198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.381+0000 7f1af2064640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec198a10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0005ab0 tx=0x7f1ae00042f0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 shutdown_connections 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f1ad40a32d0 0x7f1ad40a5790 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1aec106f10 0x7f1aec19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aec102790 0x7f1aec198a10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 --2- 172.21.15.28:0/635430675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec101de0 0x7f1aec1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 >> 172.21.15.28:0/635430675 conn(0x7f1aec0fd8c0 msgr2=0x7f1aec0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 shutdown_connections 2024-09-15T21:56:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.382+0000 7f1af2064640 1 -- 172.21.15.28:0/635430675 wait complete. 2024-09-15T21:56:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.384+0000 7f7e06ded640 1 -- 172.21.15.28:0/3073475323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00100e20 msgr2=0x7f7e00101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 --2- 172.21.15.28:0/3073475323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00100e20 0x7f7e00101200 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009900 tx=0x7f7df402a480 comp rx=0 tx=0).stop 2024-09-15T21:56:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 -- 172.21.15.28:0/3073475323 shutdown_connections 2024-09-15T21:56:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 --2- 172.21.15.28:0/3073475323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e00102ef0 0x7f7e0010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 --2- 172.21.15.28:0/3073475323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00102570 0x7f7e001029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 --2- 172.21.15.28:0/3073475323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00100e20 0x7f7e00101200 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 -- 172.21.15.28:0/3073475323 >> 172.21.15.28:0/3073475323 conn(0x7f7e000fc7d0 msgr2=0x7f7e000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 -- 172.21.15.28:0/3073475323 shutdown_connections 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 -- 172.21.15.28:0/3073475323 wait complete. 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 Processor -- start 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.385+0000 7f7e06ded640 1 -- start start 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e06ded640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e00100e20 0x7f7e001973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e06ded640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00102570 0x7f7e00197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e06ded640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00102ef0 0x7f7e0019bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e06ded640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e00110f80 con 0x7f7e00102570 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e06ded640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e00110e00 con 0x7f7e00100e20 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e06ded640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7e00111100 con 0x7f7e00102ef0 2024-09-15T21:56:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00102570 0x7f7e00197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00102570 0x7f7e00197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47534/0 (socket says 172.21.15.28:47534) 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 -- 172.21.15.28:0/344170518 learned_addr learned my addr 172.21.15.28:0/344170518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e0538a640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00102ef0 0x7f7e0019bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7e04b89640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e00100e20 0x7f7e001973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 -- 172.21.15.28:0/344170518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00102ef0 msgr2=0x7f7e0019bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00102ef0 0x7f7e0019bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 -- 172.21.15.28:0/344170518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e00100e20 msgr2=0x7f7e001973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e00100e20 0x7f7e001973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 -- 172.21.15.28:0/344170518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0019c390 con 0x7f7e00102570 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.386+0000 7f7dfffff640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00102570 0x7f7e00197920 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00d950 tx=0x7f7dec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.387+0000 7f7dfdffb640 1 -- 172.21.15.28:0/344170518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec017070 con 0x7f7e00102570 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.387+0000 7f7dfdffb640 1 -- 172.21.15.28:0/344170518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec009c30 con 0x7f7e00102570 2024-09-15T21:56:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.387+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0019c680 con 0x7f7e00102570 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.387+0000 7f7dfdffb640 1 -- 172.21.15.28:0/344170518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec00f6b0 con 0x7f7e00102570 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.387+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e001a3f60 con 0x7f7e00102570 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.388+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e001021b0 con 0x7f7e00102570 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.389+0000 7f7dfdffb640 1 -- 172.21.15.28:0/344170518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7dec0109d0 con 0x7f7e00102570 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.390+0000 7f7dfdffb640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a32d0 0x7f7df00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.390+0000 7f7dfdffb640 1 -- 172.21.15.28:0/344170518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f7dec0cfa40 con 0x7f7e00102570 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.390+0000 7f7e04b89640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a32d0 0x7f7df00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.390+0000 7f7e04b89640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a32d0 0x7f7df00a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009230 tx=0x7f7df40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.394+0000 7f7dfdffb640 1 -- 172.21.15.28:0/344170518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7dec009de0 con 0x7f7e00102570 2024-09-15T21:56:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.586+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f7e00109090 con 0x7f7e00102570 2024-09-15T21:56:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.587+0000 7f7dfdffb640 1 -- 172.21.15.28:0/344170518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7dec09bbf0 con 0x7f7e00102570 2024-09-15T21:56:17.590 INFO:teuthology.orchestra.run.smithi028.stdout:292057776161 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a32d0 msgr2=0x7f7df00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a32d0 0x7f7df00a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009230 tx=0x7f7df40058d0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00102570 msgr2=0x7f7e00197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00102570 0x7f7e00197920 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00d950 tx=0x7f7dec00de10 comp rx=0 tx=0).stop 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 shutdown_connections 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f7df00a32d0 0x7f7df00a5790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e00102ef0 0x7f7e0019bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e00102570 0x7f7e00197920 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 --2- 172.21.15.28:0/344170518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e00100e20 0x7f7e001973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 >> 172.21.15.28:0/344170518 conn(0x7f7e000fc7d0 msgr2=0x7f7e000fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 shutdown_connections 2024-09-15T21:56:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:17.589+0000 7f7e06ded640 1 -- 172.21.15.28:0/344170518 wait complete. 2024-09-15T21:56:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:17.860 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:17.861 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:17.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:17 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/635430675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T21:56:17.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:17 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/344170518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T21:56:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:17 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/635430675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T21:56:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:17 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/344170518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T21:56:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:17 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/635430675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T21:56:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:17 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/344170518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T21:56:18.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953535 got 137438953538 for osd.4 2024-09-15T21:56:18.441 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:18.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:18 smithi028 ceph-mon[30969]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-15T21:56:18.915 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776159 got 292057776161 for osd.10 2024-09-15T21:56:18.916 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:18 smithi203 ceph-mon[35546]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-15T21:56:19.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:18 smithi174 ceph-mon[35244]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-09-15T21:56:19.564 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:19.592 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561075 got 188978561077 for osd.6 2024-09-15T21:56:19.592 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:20.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 -- 172.21.15.28:0/2802280077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 msgr2=0x7f45d01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 --2- 172.21.15.28:0/2802280077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d01029b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009900 tx=0x7f45b802a480 comp rx=0 tx=0).stop 2024-09-15T21:56:20.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 -- 172.21.15.28:0/2802280077 shutdown_connections 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 --2- 172.21.15.28:0/2802280077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d0102ef0 0x7f45d010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 --2- 172.21.15.28:0/2802280077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d01029b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 --2- 172.21.15.28:0/2802280077 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d0100e20 0x7f45d0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 -- 172.21.15.28:0/2802280077 >> 172.21.15.28:0/2802280077 conn(0x7f45d00fc7d0 msgr2=0x7f45d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 -- 172.21.15.28:0/2802280077 shutdown_connections 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.019+0000 7f45d656e640 1 -- 172.21.15.28:0/2802280077 wait complete. 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 Processor -- start 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 -- start start 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d0100e20 0x7f45d01973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d0197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d0102ef0 0x7f45d019bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0110f80 con 0x7f45d0102570 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0110e00 con 0x7f45d0100e20 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45d656e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0111100 con 0x7f45d0102ef0 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d0197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d0197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47552/0 (socket says 172.21.15.28:47552) 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.020+0000 7f45c7fff640 1 -- 172.21.15.28:0/323516227 learned_addr learned my addr 172.21.15.28:0/323516227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45cffff640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d0100e20 0x7f45d01973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45c7fff640 1 -- 172.21.15.28:0/323516227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d0102ef0 msgr2=0x7f45d019bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45c7fff640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d0102ef0 0x7f45d019bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45c7fff640 1 -- 172.21.15.28:0/323516227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d0100e20 msgr2=0x7f45d01973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45c7fff640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d0100e20 0x7f45d01973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45c7fff640 1 -- 172.21.15.28:0/323516227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d019c390 con 0x7f45d0102570 2024-09-15T21:56:20.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45cffff640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d0100e20 0x7f45d01973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45c7fff640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d0197920 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f45b8006010 tx=0x7f45b8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45cdffb640 1 -- 172.21.15.28:0/323516227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8035070 con 0x7f45d0102570 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45cdffb640 1 -- 172.21.15.28:0/323516227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45b8031a20 con 0x7f45d0102570 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d019c620 con 0x7f45d0102570 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45cdffb640 1 -- 172.21.15.28:0/323516227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8038bf0 con 0x7f45d0102570 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.021+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d0069f70 con 0x7f45d0102570 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.023+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d01021b0 con 0x7f45d0102570 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.024+0000 7f45cdffb640 1 -- 172.21.15.28:0/323516227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f45b8031540 con 0x7f45d0102570 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.024+0000 7f45cdffb640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f45bc0a32d0 0x7f45bc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.024+0000 7f45cffff640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f45bc0a32d0 0x7f45bc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.024+0000 7f45cdffb640 1 -- 172.21.15.28:0/323516227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f45b80f1f70 con 0x7f45d0102570 2024-09-15T21:56:20.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.025+0000 7f45cffff640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f45bc0a32d0 0x7f45bc0a5790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45c0005d00 tx=0x7f45c0005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:20.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.029+0000 7f45cdffb640 1 -- 172.21.15.28:0/323516227 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f45b80be7d0 con 0x7f45d0102570 2024-09-15T21:56:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.233+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f45d0198770 con 0x7f45d0102570 2024-09-15T21:56:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.234+0000 7f45cdffb640 1 -- 172.21.15.28:0/323516227 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f45d0198770 con 0x7f45d0102570 2024-09-15T21:56:20.236 INFO:teuthology.orchestra.run.smithi028.stdout:55834574930 2024-09-15T21:56:20.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f45bc0a32d0 msgr2=0x7f45bc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f45bc0a32d0 0x7f45bc0a5790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45c0005d00 tx=0x7f45c0005c70 comp rx=0 tx=0).stop 2024-09-15T21:56:20.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 msgr2=0x7f45d0197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d0197920 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f45b8006010 tx=0x7f45b8003b90 comp rx=0 tx=0).stop 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 shutdown_connections 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f45bc0a32d0 0x7f45bc0a5790 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d0102ef0 0x7f45d019bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0102570 0x7f45d0197920 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 --2- 172.21.15.28:0/323516227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d0100e20 0x7f45d01973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 >> 172.21.15.28:0/323516227 conn(0x7f45d00fc7d0 msgr2=0x7f45d01068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 shutdown_connections 2024-09-15T21:56:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.236+0000 7f45d656e640 1 -- 172.21.15.28:0/323516227 wait complete. 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 -- 172.21.15.28:0/2747288173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 msgr2=0x7f8ce410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 --2- 172.21.15.28:0/2747288173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce410d950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc009930 tx=0x7f8cdc02a420 comp rx=0 tx=0).stop 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 -- 172.21.15.28:0/2747288173 shutdown_connections 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 --2- 172.21.15.28:0/2747288173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce410d950 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 --2- 172.21.15.28:0/2747288173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ce4100a80 0x7f8ce4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 --2- 172.21.15.28:0/2747288173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ce4106a80 0x7f8ce4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 -- 172.21.15.28:0/2747288173 >> 172.21.15.28:0/2747288173 conn(0x7f8ce40fc7a0 msgr2=0x7f8ce40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 -- 172.21.15.28:0/2747288173 shutdown_connections 2024-09-15T21:56:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.651+0000 7f8cec366640 1 -- 172.21.15.28:0/2747288173 wait complete. 2024-09-15T21:56:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 Processor -- start 2024-09-15T21:56:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 -- start start 2024-09-15T21:56:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ce4100a80 0x7f8ce41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce4198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ce4106a80 0x7f8ce419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce4111fa0 con 0x7f8ce4101420 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce4111e20 con 0x7f8ce4100a80 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cec366640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce4112120 con 0x7f8ce4106a80 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce4198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce4198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55396/0 (socket says 172.21.15.28:55396) 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 -- 172.21.15.28:0/2660305226 learned_addr learned my addr 172.21.15.28:0/2660305226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cea102640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ce4100a80 0x7f8ce41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8cea903640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ce4106a80 0x7f8ce419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 -- 172.21.15.28:0/2660305226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ce4106a80 msgr2=0x7f8ce419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ce4106a80 0x7f8ce419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 -- 172.21.15.28:0/2660305226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ce4100a80 msgr2=0x7f8ce41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ce4100a80 0x7f8ce41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.652+0000 7f8ce9901640 1 -- 172.21.15.28:0/2660305226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce419d490 con 0x7f8ce4101420 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.653+0000 7f8ce9901640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce4198a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400d950 tx=0x7f8cd400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.653+0000 7f8cdb7fe640 1 -- 172.21.15.28:0/2660305226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4017030 con 0x7f8ce4101420 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.653+0000 7f8cdb7fe640 1 -- 172.21.15.28:0/2660305226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cd4004020 con 0x7f8ce4101420 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.653+0000 7f8cdb7fe640 1 -- 172.21.15.28:0/2660305226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd40047a0 con 0x7f8ce4101420 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.653+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce419d780 con 0x7f8ce4101420 2024-09-15T21:56:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.653+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ce41a5060 con 0x7f8ce4101420 2024-09-15T21:56:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.654+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ce4100ee0 con 0x7f8ce4101420 2024-09-15T21:56:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.655+0000 7f8cdb7fe640 1 -- 172.21.15.28:0/2660305226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8cd40041d0 con 0x7f8ce4101420 2024-09-15T21:56:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.656+0000 7f8cdb7fe640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8cd00a32d0 0x7f8cd00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.656+0000 7f8cdb7fe640 1 -- 172.21.15.28:0/2660305226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f8cd40cfe20 con 0x7f8ce4101420 2024-09-15T21:56:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.656+0000 7f8cea102640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8cd00a32d0 0x7f8cd00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.657+0000 7f8cea102640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8cd00a32d0 0x7f8cd00a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0005fd0 tx=0x7f8ce0009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.659+0000 7f8cdb7fe640 1 -- 172.21.15.28:0/2660305226 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8cd409c620 con 0x7f8ce4101420 2024-09-15T21:56:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.837+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8ce4109040 con 0x7f8ce4101420 2024-09-15T21:56:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.840+0000 7f8cdb7fe640 1 -- 172.21.15.28:0/2660305226 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8cd409bf20 con 0x7f8ce4101420 2024-09-15T21:56:20.843 INFO:teuthology.orchestra.run.smithi028.stdout:77309411405 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8cd00a32d0 msgr2=0x7f8cd00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8cd00a32d0 0x7f8cd00a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0005fd0 tx=0x7f8ce0009680 comp rx=0 tx=0).stop 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 msgr2=0x7f8ce4198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce4198a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400d950 tx=0x7f8cd400de10 comp rx=0 tx=0).stop 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 shutdown_connections 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f8cd00a32d0 0x7f8cd00a5790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ce4106a80 0x7f8ce419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ce4101420 0x7f8ce4198a20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 --2- 172.21.15.28:0/2660305226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ce4100a80 0x7f8ce41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 >> 172.21.15.28:0/2660305226 conn(0x7f8ce40fc7a0 msgr2=0x7f8ce40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 shutdown_connections 2024-09-15T21:56:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:20.842+0000 7f8cec366640 1 -- 172.21.15.28:0/2660305226 wait complete. 2024-09-15T21:56:21.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:20 smithi028 ceph-mon[30969]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:21.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:20 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/323516227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T21:56:21.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:20 smithi203 ceph-mon[35546]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:21.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:20 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/323516227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T21:56:21.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:20 smithi174 ceph-mon[35244]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:21.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:20 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/323516227' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T21:56:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 -- 172.21.15.28:0/3318347882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 msgr2=0x7fb900106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 --2- 172.21.15.28:0/3318347882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb900106840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009930 tx=0x7fb8f402a420 comp rx=0 tx=0).stop 2024-09-15T21:56:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 -- 172.21.15.28:0/3318347882 shutdown_connections 2024-09-15T21:56:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 --2- 172.21.15.28:0/3318347882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb900106f50 0x7fb90010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 --2- 172.21.15.28:0/3318347882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb900106840 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 --2- 172.21.15.28:0/3318347882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9001002f0 0x7fb9001006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 -- 172.21.15.28:0/3318347882 >> 172.21.15.28:0/3318347882 conn(0x7fb900077210 msgr2=0x7fb9000fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 -- 172.21.15.28:0/3318347882 shutdown_connections 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.283+0000 7fb905a72640 1 -- 172.21.15.28:0/3318347882 wait complete. 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 Processor -- start 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 -- start start 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9001002f0 0x7fb900101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb9001017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb900106f50 0x7fb900101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb900111fa0 con 0x7fb900100ca0 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb900111e20 con 0x7fb9001002f0 2024-09-15T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.284+0000 7fb905a72640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb900112120 con 0x7fb900106f50 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb9001017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb9001017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55416/0 (socket says 172.21.15.28:55416) 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 -- 172.21.15.28:0/2196028505 learned_addr learned my addr 172.21.15.28:0/2196028505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8ff7fe640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9001002f0 0x7fb900101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8fffff640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb900106f50 0x7fb900101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 -- 172.21.15.28:0/2196028505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb900106f50 msgr2=0x7fb900101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb900106f50 0x7fb900101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 -- 172.21.15.28:0/2196028505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9001002f0 msgr2=0x7fb900101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9001002f0 0x7fb900101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 -- 172.21.15.28:0/2196028505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9001a68c0 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8feffd640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb9001017b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4005940 tx=0x7fb8f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8fcff9640 1 -- 172.21.15.28:0/2196028505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f403f030 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8fcff9640 1 -- 172.21.15.28:0/2196028505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8f4003ce0 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9001a6b50 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb8fcff9640 1 -- 172.21.15.28:0/2196028505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4035070 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.285+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9001a7030 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.287+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9001006d0 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.288+0000 7fb8fcff9640 1 -- 172.21.15.28:0/2196028505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb8f4042030 con 0x7fb900100ca0 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.288+0000 7fb8fcff9640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8e80a32d0 0x7fb8e80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.288+0000 7fb8ff7fe640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8e80a32d0 0x7fb8e80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.288+0000 7fb8fcff9640 1 -- 172.21.15.28:0/2196028505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fb8f40f0660 con 0x7fb900100ca0 2024-09-15T21:56:21.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.289+0000 7fb8ff7fe640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8e80a32d0 0x7fb8e80a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0005e10 tx=0x7fb8f0005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:21.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.292+0000 7fb8fcff9640 1 -- 172.21.15.28:0/2196028505 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb8f40bcec0 con 0x7fb900100ca0 2024-09-15T21:56:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.474+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb9001027a0 con 0x7fb900100ca0 2024-09-15T21:56:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.474+0000 7fb8fcff9640 1 -- 172.21.15.28:0/2196028505 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb9001027a0 con 0x7fb900100ca0 2024-09-15T21:56:21.476 INFO:teuthology.orchestra.run.smithi028.stdout:34359738454 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8e80a32d0 msgr2=0x7fb8e80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8e80a32d0 0x7fb8e80a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0005e10 tx=0x7fb8f0005da0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 msgr2=0x7fb9001017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb9001017b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4005940 tx=0x7fb8f40042f0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 shutdown_connections 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb8e80a32d0 0x7fb8e80a5790 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb900106f50 0x7fb900101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900100ca0 0x7fb9001017b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 --2- 172.21.15.28:0/2196028505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9001002f0 0x7fb900101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.476+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 >> 172.21.15.28:0/2196028505 conn(0x7fb900077210 msgr2=0x7fb900105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.477+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 shutdown_connections 2024-09-15T21:56:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.477+0000 7fb905a72640 1 -- 172.21.15.28:0/2196028505 wait complete. 2024-09-15T21:56:21.757 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574930 for osd.1 2024-09-15T21:56:21.757 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 -- 172.21.15.28:0/842324964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8100e20 msgr2=0x7fa5c8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/842324964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8100e20 0x7fa5c8101200 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009900 tx=0x7fa5c402a480 comp rx=0 tx=0).stop 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 -- 172.21.15.28:0/842324964 shutdown_connections 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/842324964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5c8102ef0 0x7fa5c810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/842324964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5c8102570 0x7fa5c81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/842324964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8100e20 0x7fa5c8101200 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 -- 172.21.15.28:0/842324964 >> 172.21.15.28:0/842324964 conn(0x7fa5c80fc7d0 msgr2=0x7fa5c80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 -- 172.21.15.28:0/842324964 shutdown_connections 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.912+0000 7fa5cfedd640 1 -- 172.21.15.28:0/842324964 wait complete. 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 Processor -- start 2024-09-15T21:56:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 -- start start 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5c8100e20 0x7fa5c8103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8102570 0x7fa5c8103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5c8102ef0 0x7fa5c81041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c8110f80 con 0x7fa5c8102570 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c8110e00 con 0x7fa5c8100e20 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cfedd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c8111100 con 0x7fa5c8102ef0 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8102570 0x7fa5c8103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8102570 0x7fa5c8103c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55434/0 (socket says 172.21.15.28:55434) 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 -- 172.21.15.28:0/2605209658 learned_addr learned my addr 172.21.15.28:0/2605209658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cdc79640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5c8100e20 0x7fa5c8103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 -- 172.21.15.28:0/2605209658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5c8102ef0 msgr2=0x7fa5c81041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5ce47a640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5c8102ef0 0x7fa5c81041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5c8102ef0 0x7fa5c81041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 -- 172.21.15.28:0/2605209658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5c8100e20 msgr2=0x7fa5c8103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5c8100e20 0x7fa5c8103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cd478640 1 -- 172.21.15.28:0/2605209658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c81a6940 con 0x7fa5c8102570 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.913+0000 7fa5cdc79640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5c8100e20 0x7fa5c8103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.914+0000 7fa5cd478640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8102570 0x7fa5c8103c90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c970 tx=0x7fa5b800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.914+0000 7fa5beffd640 1 -- 172.21.15.28:0/2605209658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8007bb0 con 0x7fa5c8102570 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.914+0000 7fa5beffd640 1 -- 172.21.15.28:0/2605209658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5b8007d50 con 0x7fa5c8102570 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.914+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c81a6c30 con 0x7fa5c8102570 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.914+0000 7fa5beffd640 1 -- 172.21.15.28:0/2605209658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800f450 con 0x7fa5c8102570 2024-09-15T21:56:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.914+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c81a7120 con 0x7fa5c8102570 2024-09-15T21:56:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.915+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c8101200 con 0x7fa5c8102570 2024-09-15T21:56:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.916+0000 7fa5beffd640 1 -- 172.21.15.28:0/2605209658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa5b8005020 con 0x7fa5c8102570 2024-09-15T21:56:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.917+0000 7fa5beffd640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa5b40a32d0 0x7fa5b40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.917+0000 7fa5cdc79640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa5b40a32d0 0x7fa5b40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.917+0000 7fa5beffd640 1 -- 172.21.15.28:0/2605209658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fa5b80cfca0 con 0x7fa5c8102570 2024-09-15T21:56:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.917+0000 7fa5cdc79640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa5b40a32d0 0x7fa5b40a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4005ab0 tx=0x7fa5c4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:21.921+0000 7fa5beffd640 1 -- 172.21.15.28:0/2605209658 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa5b8014030 con 0x7fa5c8102570 2024-09-15T21:56:22.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:21 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2660305226' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T21:56:22.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:21 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2196028505' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T21:56:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.103+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa5c81a7380 con 0x7fa5c8102570 2024-09-15T21:56:22.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.103+0000 7fa5beffd640 1 -- 172.21.15.28:0/2605209658 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa5b809c500 con 0x7fa5c8102570 2024-09-15T21:56:22.105 INFO:teuthology.orchestra.run.smithi028.stdout:326417514526 2024-09-15T21:56:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa5b40a32d0 msgr2=0x7fa5b40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa5b40a32d0 0x7fa5b40a5790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4005ab0 tx=0x7fa5c4005a20 comp rx=0 tx=0).stop 2024-09-15T21:56:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8102570 msgr2=0x7fa5c8103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8102570 0x7fa5c8103c90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c970 tx=0x7fa5b800ce30 comp rx=0 tx=0).stop 2024-09-15T21:56:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 shutdown_connections 2024-09-15T21:56:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa5b40a32d0 0x7fa5b40a5790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5c8102ef0 0x7fa5c81041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8102570 0x7fa5c8103c90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 --2- 172.21.15.28:0/2605209658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5c8100e20 0x7fa5c8103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 >> 172.21.15.28:0/2605209658 conn(0x7fa5c80fc7d0 msgr2=0x7fa5c806a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 shutdown_connections 2024-09-15T21:56:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.105+0000 7fa5cfedd640 1 -- 172.21.15.28:0/2605209658 wait complete. 2024-09-15T21:56:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:21 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2660305226' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T21:56:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:21 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2196028505' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T21:56:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:21 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2660305226' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T21:56:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:21 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2196028505' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T21:56:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 -- 172.21.15.28:0/2537082159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088102790 msgr2=0x7f80881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 --2- 172.21.15.28:0/2537082159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088102790 0x7f80881067e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8078009900 tx=0x7f807802a410 comp rx=0 tx=0).stop 2024-09-15T21:56:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 -- 172.21.15.28:0/2537082159 shutdown_connections 2024-09-15T21:56:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 --2- 172.21.15.28:0/2537082159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8088106f10 0x7f808810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 --2- 172.21.15.28:0/2537082159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088102790 0x7f80881067e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 --2- 172.21.15.28:0/2537082159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8088101de0 0x7f80881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 -- 172.21.15.28:0/2537082159 >> 172.21.15.28:0/2537082159 conn(0x7f80880fd8c0 msgr2=0x7f80880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 -- 172.21.15.28:0/2537082159 shutdown_connections 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.576+0000 7f808f4ec640 1 -- 172.21.15.28:0/2537082159 wait complete. 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 Processor -- start 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 -- start start 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8088101de0 0x7f8088198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8088102790 0x7f8088198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088106f10 0x7f808819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8088111f70 con 0x7f8088106f10 2024-09-15T21:56:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8088111df0 con 0x7f8088101de0 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.577+0000 7f808f4ec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80881120f0 con 0x7f8088102790 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088106f10 0x7f808819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088106f10 0x7f808819cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55454/0 (socket says 172.21.15.28:55454) 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808ca87640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8088102790 0x7f8088198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 -- 172.21.15.28:0/1974932389 learned_addr learned my addr 172.21.15.28:0/1974932389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808d288640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8088101de0 0x7f8088198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 -- 172.21.15.28:0/1974932389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8088102790 msgr2=0x7f8088198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8088102790 0x7f8088198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 -- 172.21.15.28:0/1974932389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8088101de0 msgr2=0x7f8088198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8088101de0 0x7f8088198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 -- 172.21.15.28:0/1974932389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808819d3e0 con 0x7f8088106f10 2024-09-15T21:56:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808d288640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8088101de0 0x7f8088198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808da89640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088106f10 0x7f808819cd00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f808000d750 tx=0x7f808000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f807e7fc640 1 -- 172.21.15.28:0/1974932389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080017070 con 0x7f8088106f10 2024-09-15T21:56:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f807e7fc640 1 -- 172.21.15.28:0/1974932389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8080005500 con 0x7f8088106f10 2024-09-15T21:56:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808819d6d0 con 0x7f8088106f10 2024-09-15T21:56:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.578+0000 7f807e7fc640 1 -- 172.21.15.28:0/1974932389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080004d90 con 0x7f8088106f10 2024-09-15T21:56:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.579+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80881a4fb0 con 0x7f8088106f10 2024-09-15T21:56:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.580+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80881021c0 con 0x7f8088106f10 2024-09-15T21:56:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.581+0000 7f807e7fc640 1 -- 172.21.15.28:0/1974932389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8080004f30 con 0x7f8088106f10 2024-09-15T21:56:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.582+0000 7f807e7fc640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80740a3350 0x7f80740a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.582+0000 7f808d288640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80740a3350 0x7f80740a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.582+0000 7f807e7fc640 1 -- 172.21.15.28:0/1974932389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f80800cfb10 con 0x7f8088106f10 2024-09-15T21:56:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.582+0000 7f808d288640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80740a3350 0x7f80740a5810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8084005c40 tx=0x7f808400a430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.585+0000 7f807e7fc640 1 -- 172.21.15.28:0/1974932389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f808009c390 con 0x7f8088106f10 2024-09-15T21:56:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.762+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f8088109b60 con 0x7f8088106f10 2024-09-15T21:56:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.762+0000 7f807e7fc640 1 -- 172.21.15.28:0/1974932389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f808009bc90 con 0x7f8088106f10 2024-09-15T21:56:22.764 INFO:teuthology.orchestra.run.smithi028.stdout:158913790014 2024-09-15T21:56:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80740a3350 msgr2=0x7f80740a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80740a3350 0x7f80740a5810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8084005c40 tx=0x7f808400a430 comp rx=0 tx=0).stop 2024-09-15T21:56:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088106f10 msgr2=0x7f808819cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088106f10 0x7f808819cd00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f808000d750 tx=0x7f808000dc10 comp rx=0 tx=0).stop 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 shutdown_connections 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f80740a3350 0x7f80740a5810 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8088106f10 0x7f808819cd00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8088102790 0x7f8088198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 --2- 172.21.15.28:0/1974932389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8088101de0 0x7f8088198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 >> 172.21.15.28:0/1974932389 conn(0x7f80880fd8c0 msgr2=0x7f80880fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 shutdown_connections 2024-09-15T21:56:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:22.764+0000 7f808f4ec640 1 -- 172.21.15.28:0/1974932389 wait complete. 2024-09-15T21:56:23.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:22 smithi028 ceph-mon[30969]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:23.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:22 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2605209658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T21:56:23.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:22 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1974932389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T21:56:23.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:22 smithi203 ceph-mon[35546]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:23.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:22 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2605209658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T21:56:23.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:22 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1974932389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T21:56:23.247 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738454 for osd.0 2024-09-15T21:56:23.248 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:23.259 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:22 smithi174 ceph-mon[35244]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:23.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:22 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2605209658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T21:56:23.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:22 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1974932389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T21:56:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 -- 172.21.15.28:0/3649700394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c0074620 msgr2=0x7f35c0074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 --2- 172.21.15.28:0/3649700394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c0074620 0x7f35c0074a00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009900 tx=0x7f35b002a480 comp rx=0 tx=0).stop 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 -- 172.21.15.28:0/3649700394 shutdown_connections 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 --2- 172.21.15.28:0/3649700394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35c010e3d0 0x7f35c0110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 --2- 172.21.15.28:0/3649700394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35c0074f40 0x7f35c010de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 --2- 172.21.15.28:0/3649700394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c0074620 0x7f35c0074a00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 -- 172.21.15.28:0/3649700394 >> 172.21.15.28:0/3649700394 conn(0x7f35c00fd8f0 msgr2=0x7f35c00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 -- 172.21.15.28:0/3649700394 shutdown_connections 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.541+0000 7f35c5e08640 1 -- 172.21.15.28:0/3649700394 wait complete. 2024-09-15T21:56:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 Processor -- start 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 -- start start 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35c0074620 0x7f35c01a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35c0074f40 0x7f35c01a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c010e3d0 0x7f35c019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35c0114240 con 0x7f35c010e3d0 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f35c01140c0 con 0x7f35c0074620 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35c5e08640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35c01143c0 con 0x7f35c0074f40 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c010e3d0 0x7f35c019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c010e3d0 0x7f35c019a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55498/0 (socket says 172.21.15.28:55498) 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35bffff640 1 -- 172.21.15.28:0/2268700188 learned_addr learned my addr 172.21.15.28:0/2268700188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35bf7fe640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35c0074620 0x7f35c01a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.542+0000 7f35beffd640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35c0074f40 0x7f35c01a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bffff640 1 -- 172.21.15.28:0/2268700188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35c0074f40 msgr2=0x7f35c01a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bffff640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35c0074f40 0x7f35c01a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bffff640 1 -- 172.21.15.28:0/2268700188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35c0074620 msgr2=0x7f35c01a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bffff640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35c0074620 0x7f35c01a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bffff640 1 -- 172.21.15.28:0/2268700188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c019ad30 con 0x7f35c010e3d0 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bffff640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c010e3d0 0x7f35c019a600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f35b800b970 tx=0x7f35b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bcff9640 1 -- 172.21.15.28:0/2268700188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8019070 con 0x7f35c010e3d0 2024-09-15T21:56:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c019b020 con 0x7f35c010e3d0 2024-09-15T21:56:24.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bcff9640 1 -- 172.21.15.28:0/2268700188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b800cb40 con 0x7f35c010e3d0 2024-09-15T21:56:24.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35bcff9640 1 -- 172.21.15.28:0/2268700188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8007650 con 0x7f35c010e3d0 2024-09-15T21:56:24.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.543+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c01a7310 con 0x7f35c010e3d0 2024-09-15T21:56:24.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.544+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c0074a80 con 0x7f35c010e3d0 2024-09-15T21:56:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.546+0000 7f35bcff9640 1 -- 172.21.15.28:0/2268700188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f35b801c070 con 0x7f35c010e3d0 2024-09-15T21:56:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.546+0000 7f35bcff9640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f35a80a32d0 0x7f35a80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.546+0000 7f35bcff9640 1 -- 172.21.15.28:0/2268700188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f35b80cf790 con 0x7f35c010e3d0 2024-09-15T21:56:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.547+0000 7f35bf7fe640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f35a80a32d0 0x7f35a80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.547+0000 7f35bf7fe640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f35a80a32d0 0x7f35a80a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35b0000c00 tx=0x7f35b00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.549+0000 7f35bcff9640 1 -- 172.21.15.28:0/2268700188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f35b809bab0 con 0x7f35c010e3d0 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.452+0000 7f4bc654b640 1 -- 172.21.15.28:0/3611071558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0106a80 msgr2=0x7f4bc0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.452+0000 7f4bc654b640 1 --2- 172.21.15.28:0/3611071558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0106a80 0x7f4bc0106e60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0009930 tx=0x7f4bb002a420 comp rx=0 tx=0).stop 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 -- 172.21.15.28:0/3611071558 shutdown_connections 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 --2- 172.21.15.28:0/3611071558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc0101420 0x7f4bc010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 --2- 172.21.15.28:0/3611071558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bc0100a80 0x7f4bc0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 --2- 172.21.15.28:0/3611071558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0106a80 0x7f4bc0106e60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 -- 172.21.15.28:0/3611071558 >> 172.21.15.28:0/3611071558 conn(0x7f4bc00fc7a0 msgr2=0x7f4bc00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 -- 172.21.15.28:0/3611071558 shutdown_connections 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 -- 172.21.15.28:0/3611071558 wait complete. 2024-09-15T21:56:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 Processor -- start 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 -- start start 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0100a80 0x7f4bc01984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.453+0000 7f4bc654b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc0101420 0x7f4bc0198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bc654b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bc0106a80 0x7f4bc019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bc654b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0110ec0 con 0x7f4bc0100a80 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bc654b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0110d40 con 0x7f4bc0106a80 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bc654b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0111040 con 0x7f4bc0101420 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0100a80 0x7f4bc01984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0100a80 0x7f4bc01984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55466/0 (socket says 172.21.15.28:55466) 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 -- 172.21.15.28:0/695671186 learned_addr learned my addr 172.21.15.28:0/695671186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bb7fff640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc0101420 0x7f4bc0198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bc4ae8640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bc0106a80 0x7f4bc019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 -- 172.21.15.28:0/695671186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc0101420 msgr2=0x7f4bc0198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc0101420 0x7f4bc0198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 -- 172.21.15.28:0/695671186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bc0106a80 msgr2=0x7f4bc019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bc0106a80 0x7f4bc019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 -- 172.21.15.28:0/695671186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc019d490 con 0x7f4bc0100a80 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bc4ae8640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bc0106a80 0x7f4bc019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbffff640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0100a80 0x7f4bc01984e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0009230 tx=0x7f4bb0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbdffb640 1 -- 172.21.15.28:0/695671186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb003f030 con 0x7f4bc0100a80 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbdffb640 1 -- 172.21.15.28:0/695671186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bb0003ad0 con 0x7f4bc0100a80 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc019d720 con 0x7f4bc0100a80 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.454+0000 7f4bbdffb640 1 -- 172.21.15.28:0/695671186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0035070 con 0x7f4bc0100a80 2024-09-15T21:56:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.455+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc01a5060 con 0x7f4bc0100a80 2024-09-15T21:56:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.455+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc01021c0 con 0x7f4bc0100a80 2024-09-15T21:56:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.457+0000 7f4bbdffb640 1 -- 172.21.15.28:0/695671186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4bb0042030 con 0x7f4bc0100a80 2024-09-15T21:56:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.458+0000 7f4bbdffb640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bac0a32d0 0x7f4bac0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.458+0000 7f4bb7fff640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bac0a32d0 0x7f4bac0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.458+0000 7f4bbdffb640 1 -- 172.21.15.28:0/695671186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f4bb00f0750 con 0x7f4bc0100a80 2024-09-15T21:56:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.459+0000 7f4bb7fff640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bac0a32d0 0x7f4bac0a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0105200 tx=0x7f4ba8009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.461+0000 7f4bbdffb640 1 -- 172.21.15.28:0/695671186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4bb00bcfb0 con 0x7f4bc0100a80 2024-09-15T21:56:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.644+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f4bc010a000 con 0x7f4bc0100a80 2024-09-15T21:56:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.644+0000 7f4bbdffb640 1 -- 172.21.15.28:0/695671186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4bb00bc8b0 con 0x7f4bc0100a80 2024-09-15T21:56:24.646 INFO:teuthology.orchestra.run.smithi028.stdout:240518168621 2024-09-15T21:56:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bac0a32d0 msgr2=0x7f4bac0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bac0a32d0 0x7f4bac0a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0105200 tx=0x7f4ba8009110 comp rx=0 tx=0).stop 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0100a80 msgr2=0x7f4bc01984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0100a80 0x7f4bc01984e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0009230 tx=0x7f4bb0004270 comp rx=0 tx=0).stop 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 shutdown_connections 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f4bac0a32d0 0x7f4bac0a5790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bc0106a80 0x7f4bc019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bc0101420 0x7f4bc0198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 --2- 172.21.15.28:0/695671186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc0100a80 0x7f4bc01984e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 >> 172.21.15.28:0/695671186 conn(0x7f4bc00fc7a0 msgr2=0x7f4bc006aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 shutdown_connections 2024-09-15T21:56:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.647+0000 7f4bc654b640 1 -- 172.21.15.28:0/695671186 wait complete. 2024-09-15T21:56:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.728+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f35c0109550 con 0x7f35c010e3d0 2024-09-15T21:56:24.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.728+0000 7f35bcff9640 1 -- 172.21.15.28:0/2268700188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f35b809fdb0 con 0x7f35c010e3d0 2024-09-15T21:56:24.730 INFO:teuthology.orchestra.run.smithi028.stdout:270582939689 2024-09-15T21:56:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f35a80a32d0 msgr2=0x7f35a80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f35a80a32d0 0x7f35a80a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35b0000c00 tx=0x7f35b00058d0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c010e3d0 msgr2=0x7f35c019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c010e3d0 0x7f35c019a600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f35b800b970 tx=0x7f35b800be30 comp rx=0 tx=0).stop 2024-09-15T21:56:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 shutdown_connections 2024-09-15T21:56:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f35a80a32d0 0x7f35a80a5790 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c010e3d0 0x7f35c019a600 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35c0074f40 0x7f35c01a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 --2- 172.21.15.28:0/2268700188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35c0074620 0x7f35c01a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 >> 172.21.15.28:0/2268700188 conn(0x7f35c00fd8f0 msgr2=0x7f35c00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 shutdown_connections 2024-09-15T21:56:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:24.731+0000 7f35c5e08640 1 -- 172.21.15.28:0/2268700188 wait complete. 2024-09-15T21:56:25.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:24 smithi028 ceph-mon[30969]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:25.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:24 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/695671186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T21:56:25.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:24 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2268700188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T21:56:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:24 smithi203 ceph-mon[35546]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:24 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/695671186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T21:56:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:24 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2268700188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T21:56:25.232 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514523 got 326417514526 for osd.11 2024-09-15T21:56:25.232 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:25.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:24 smithi174 ceph-mon[35244]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:25.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:24 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/695671186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T21:56:25.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:24 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2268700188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T21:56:26.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411405 for osd.2 2024-09-15T21:56:26.045 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 -- 172.21.15.28:0/591492007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 msgr2=0x7fca1c10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 --2- 172.21.15.28:0/591492007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c10bbf0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009900 tx=0x7fca0c02a480 comp rx=0 tx=0).stop 2024-09-15T21:56:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 -- 172.21.15.28:0/591492007 shutdown_connections 2024-09-15T21:56:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 --2- 172.21.15.28:0/591492007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca1c10c180 0x7fca1c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 --2- 172.21.15.28:0/591492007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c10bbf0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 --2- 172.21.15.28:0/591492007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca1c102de0 0x7fca1c1031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 -- 172.21.15.28:0/591492007 >> 172.21.15.28:0/591492007 conn(0x7fca1c0fc780 msgr2=0x7fca1c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.225+0000 7fca2334c640 1 -- 172.21.15.28:0/591492007 shutdown_connections 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 -- 172.21.15.28:0/591492007 wait complete. 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 Processor -- start 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 -- start start 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca1c102de0 0x7fca1c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca1c10c180 0x7fca1c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c111f50 con 0x7fca1c103700 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c111dd0 con 0x7fca1c102de0 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.226+0000 7fca2334c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c1120d0 con 0x7fca1c10c180 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55514/0 (socket says 172.21.15.28:55514) 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 -- 172.21.15.28:0/2177142990 learned_addr learned my addr 172.21.15.28:0/2177142990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca210e8640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca1c102de0 0x7fca1c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca218e9640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca1c10c180 0x7fca1c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 -- 172.21.15.28:0/2177142990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca1c10c180 msgr2=0x7fca1c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca1c10c180 0x7fca1c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 -- 172.21.15.28:0/2177142990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca1c102de0 msgr2=0x7fca1c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca1c102de0 0x7fca1c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 -- 172.21.15.28:0/2177142990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1c19d440 con 0x7fca1c103700 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca210e8640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca1c102de0 0x7fca1c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca208e7640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c1989d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fca0c006010 tx=0x7fca0c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca127fc640 1 -- 172.21.15.28:0/2177142990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c03f030 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca127fc640 1 -- 172.21.15.28:0/2177142990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca0c003ce0 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca127fc640 1 -- 172.21.15.28:0/2177142990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c035070 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.227+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1c19d6d0 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.228+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca1c1a5010 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.229+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1c06afb0 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.230+0000 7fca127fc640 1 -- 172.21.15.28:0/2177142990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fca0c042030 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.230+0000 7fca127fc640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fca080a32d0 0x7fca080a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.231+0000 7fca210e8640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fca080a32d0 0x7fca080a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.231+0000 7fca127fc640 1 -- 172.21.15.28:0/2177142990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fca0c0f05d0 con 0x7fca1c103700 2024-09-15T21:56:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.231+0000 7fca210e8640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fca080a32d0 0x7fca080a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fca180058c0 tx=0x7fca18005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:26.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.235+0000 7fca127fc640 1 -- 172.21.15.28:0/2177142990 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fca0c0bce90 con 0x7fca1c103700 2024-09-15T21:56:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.417+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fca1c107200 con 0x7fca1c103700 2024-09-15T21:56:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.417+0000 7fca127fc640 1 -- 172.21.15.28:0/2177142990 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fca0c0bc790 con 0x7fca1c103700 2024-09-15T21:56:26.419 INFO:teuthology.orchestra.run.smithi028.stdout:210453397556 2024-09-15T21:56:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fca080a32d0 msgr2=0x7fca080a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fca080a32d0 0x7fca080a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fca180058c0 tx=0x7fca18005850 comp rx=0 tx=0).stop 2024-09-15T21:56:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 msgr2=0x7fca1c1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c1989d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fca0c006010 tx=0x7fca0c0042f0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 shutdown_connections 2024-09-15T21:56:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fca080a32d0 0x7fca080a5790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca1c10c180 0x7fca1c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1c103700 0x7fca1c1989d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 --2- 172.21.15.28:0/2177142990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca1c102de0 0x7fca1c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 >> 172.21.15.28:0/2177142990 conn(0x7fca1c0fc780 msgr2=0x7fca1c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 shutdown_connections 2024-09-15T21:56:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:26.419+0000 7fca2334c640 1 -- 172.21.15.28:0/2177142990 wait complete. 2024-09-15T21:56:26.721 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168619 got 240518168621 for osd.8 2024-09-15T21:56:26.721 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:26.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790012 got 158913790014 for osd.5 2024-09-15T21:56:26.825 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:26.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:26 smithi028 ceph-mon[30969]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:26.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:26 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2177142990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T21:56:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:26 smithi203 ceph-mon[35546]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:27.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:26 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2177142990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T21:56:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:26 smithi174 ceph-mon[35244]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:27.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:26 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2177142990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T21:56:27.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939685 got 270582939689 for osd.9 2024-09-15T21:56:27.306 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:27.570 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:28.007 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397552 got 210453397556 for osd.7 2024-09-15T21:56:28.007 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:28.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.527+0000 7f40454cf640 1 -- 172.21.15.28:0/4143532708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040102ef0 msgr2=0x7f404010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:28.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.527+0000 7f40454cf640 1 --2- 172.21.15.28:0/4143532708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040102ef0 0x7f404010da40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4038009ca0 tx=0x7f403802a950 comp rx=0 tx=0).stop 2024-09-15T21:56:28.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.527+0000 7f40454cf640 1 -- 172.21.15.28:0/4143532708 shutdown_connections 2024-09-15T21:56:28.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.527+0000 7f40454cf640 1 --2- 172.21.15.28:0/4143532708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040102ef0 0x7f404010da40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.527+0000 7f40454cf640 1 --2- 172.21.15.28:0/4143532708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4040102570 0x7f40401029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.527+0000 7f40454cf640 1 --2- 172.21.15.28:0/4143532708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4040100e20 0x7f4040101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.527+0000 7f40454cf640 1 -- 172.21.15.28:0/4143532708 >> 172.21.15.28:0/4143532708 conn(0x7f40400fc7d0 msgr2=0x7f40400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.528+0000 7f40454cf640 1 -- 172.21.15.28:0/4143532708 shutdown_connections 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.528+0000 7f40454cf640 1 -- 172.21.15.28:0/4143532708 wait complete. 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.528+0000 7f40454cf640 1 Processor -- start 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.528+0000 7f40454cf640 1 -- start start 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.528+0000 7f40454cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040100e20 0x7f4040197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f40454cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4040102570 0x7f4040197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:28.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f40454cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4040102ef0 0x7f4040197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f40454cf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4040110f80 con 0x7f4040100e20 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f40454cf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4040110e00 con 0x7f4040102570 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f40454cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4040111100 con 0x7f4040102ef0 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040100e20 0x7f4040197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040100e20 0x7f4040197000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55534/0 (socket says 172.21.15.28:55534) 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 -- 172.21.15.28:0/1785798532 learned_addr learned my addr 172.21.15.28:0/1785798532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403f7fe640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4040102ef0 0x7f4040197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403e7fc640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4040102570 0x7f4040197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 -- 172.21.15.28:0/1785798532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4040102ef0 msgr2=0x7f4040197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4040102ef0 0x7f4040197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 -- 172.21.15.28:0/1785798532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4040102570 msgr2=0x7f4040197540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4040102570 0x7f4040197540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 -- 172.21.15.28:0/1785798532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40401a6870 con 0x7f4040100e20 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403f7fe640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4040102ef0 0x7f4040197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403e7fc640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4040102570 0x7f4040197540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f403effd640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040100e20 0x7f4040197000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f403000c920 tx=0x7f403000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.529+0000 7f4027fff640 1 -- 172.21.15.28:0/1785798532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030007c70 con 0x7f4040100e20 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.530+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40401a6b60 con 0x7f4040100e20 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.530+0000 7f4027fff640 1 -- 172.21.15.28:0/1785798532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4030007e10 con 0x7f4040100e20 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.530+0000 7f4027fff640 1 -- 172.21.15.28:0/1785798532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030018890 con 0x7f4040100e20 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.530+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40401a70a0 con 0x7f4040100e20 2024-09-15T21:56:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.531+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4040101200 con 0x7f4040100e20 2024-09-15T21:56:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.532+0000 7f4027fff640 1 -- 172.21.15.28:0/1785798532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4030018a70 con 0x7f4040100e20 2024-09-15T21:56:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.533+0000 7f4027fff640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f40280a32d0 0x7f40280a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.533+0000 7f403e7fc640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f40280a32d0 0x7f40280a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.533+0000 7f4027fff640 1 -- 172.21.15.28:0/1785798532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f40300d0060 con 0x7f4040100e20 2024-09-15T21:56:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.533+0000 7f403e7fc640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f40280a32d0 0x7f40280a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f40340058c0 tx=0x7f4034005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.537+0000 7f4027fff640 1 -- 172.21.15.28:0/1785798532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4030014030 con 0x7f4040100e20 2024-09-15T21:56:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.713+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f40401a72d0 con 0x7f4040100e20 2024-09-15T21:56:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.713+0000 7f4027fff640 1 -- 172.21.15.28:0/1785798532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f403000de20 con 0x7f4040100e20 2024-09-15T21:56:28.715 INFO:teuthology.orchestra.run.smithi028.stdout:107374182473 2024-09-15T21:56:28.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f40280a32d0 msgr2=0x7f40280a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:28.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f40280a32d0 0x7f40280a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f40340058c0 tx=0x7f4034005850 comp rx=0 tx=0).stop 2024-09-15T21:56:28.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040100e20 msgr2=0x7f4040197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040100e20 0x7f4040197000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f403000c920 tx=0x7f403000cde0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 shutdown_connections 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f40280a32d0 0x7f40280a5790 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4040102ef0 0x7f4040197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4040102570 0x7f4040197540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.715+0000 7f40454cf640 1 --2- 172.21.15.28:0/1785798532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4040100e20 0x7f4040197000 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.716+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 >> 172.21.15.28:0/1785798532 conn(0x7f40400fc7d0 msgr2=0x7f40401067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.716+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 shutdown_connections 2024-09-15T21:56:28.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:28.716+0000 7f40454cf640 1 -- 172.21.15.28:0/1785798532 wait complete. 2024-09-15T21:56:29.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:28 smithi028 ceph-mon[30969]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:29.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:28 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1785798532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T21:56:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:28 smithi203 ceph-mon[35546]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:29.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:28 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1785798532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T21:56:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:28 smithi174 ceph-mon[35244]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:28 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1785798532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T21:56:29.407 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182470 got 107374182473 for osd.3 2024-09-15T21:56:29.407 DEBUG:teuthology.parallel:result is None 2024-09-15T21:56:29.407 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T21:56:29.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T21:56:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 -- 172.21.15.28:0/3762981182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d410c170 msgr2=0x7fc2d410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/3762981182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d410c170 0x7fc2d410e610 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009ca0 tx=0x7fc2cc02a950 comp rx=0 tx=0).stop 2024-09-15T21:56:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 -- 172.21.15.28:0/3762981182 shutdown_connections 2024-09-15T21:56:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/3762981182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d410c170 0x7fc2d410e610 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/3762981182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2d4103660 0x7fc2d410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/3762981182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2d4102d40 0x7fc2d4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 -- 172.21.15.28:0/3762981182 >> 172.21.15.28:0/3762981182 conn(0x7fc2d40fc7a0 msgr2=0x7fc2d40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 -- 172.21.15.28:0/3762981182 shutdown_connections 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.199+0000 7fc2dbf05640 1 -- 172.21.15.28:0/3762981182 wait complete. 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 Processor -- start 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 -- start start 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2d4102d40 0x7fc2d41983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d4103660 0x7fc2d4198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2d410c170 0x7fc2d419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4111f70 con 0x7fc2d4103660 2024-09-15T21:56:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d4111df0 con 0x7fc2d410c170 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2dbf05640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d41120f0 con 0x7fc2d4102d40 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2d94a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d4103660 0x7fc2d4198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2d94a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d4103660 0x7fc2d4198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55550/0 (socket says 172.21.15.28:55550) 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2d94a0640 1 -- 172.21.15.28:0/369669994 learned_addr learned my addr 172.21.15.28:0/369669994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.200+0000 7fc2da4a2640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2d410c170 0x7fc2d419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2d9ca1640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2d4102d40 0x7fc2d41983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2d94a0640 1 -- 172.21.15.28:0/369669994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2d4102d40 msgr2=0x7fc2d41983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2d94a0640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2d4102d40 0x7fc2d41983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2d94a0640 1 -- 172.21.15.28:0/369669994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2d410c170 msgr2=0x7fc2d419ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2d94a0640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2d410c170 0x7fc2d419ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2d94a0640 1 -- 172.21.15.28:0/369669994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d419d3a0 con 0x7fc2d4103660 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2d94a0640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d4103660 0x7fc2d4198930 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400c970 tx=0x7fc2c400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2caffd640 1 -- 172.21.15.28:0/369669994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4007bb0 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2caffd640 1 -- 172.21.15.28:0/369669994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c4007d50 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d419d690 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2caffd640 1 -- 172.21.15.28:0/369669994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c400f450 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.201+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d41a4f70 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.202+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d4103120 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.203+0000 7fc2caffd640 1 -- 172.21.15.28:0/369669994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc2c4004f40 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.204+0000 7fc2caffd640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc2c00a32d0 0x7fc2c00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.204+0000 7fc2d9ca1640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc2c00a32d0 0x7fc2c00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.205+0000 7fc2caffd640 1 -- 172.21.15.28:0/369669994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fc2c40cfec0 con 0x7fc2d4103660 2024-09-15T21:56:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.205+0000 7fc2d9ca1640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc2c00a32d0 0x7fc2c00a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00058c0 tx=0x7fc2d000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.208+0000 7fc2caffd640 1 -- 172.21.15.28:0/369669994 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc2c4014030 con 0x7fc2d4103660 2024-09-15T21:56:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.382+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2d4062630 con 0x7fc2c00a32d0 2024-09-15T21:56:30.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.393+0000 7fc2caffd640 1 -- 172.21.15.28:0/369669994 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340417 (secure 0 0 0) 0x7fc2d4062630 con 0x7fc2c00a32d0 2024-09-15T21:56:30.403 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:56:30.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc2c00a32d0 msgr2=0x7fc2c00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc2c00a32d0 0x7fc2c00a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00058c0 tx=0x7fc2d000a4c0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d4103660 msgr2=0x7fc2d4198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d4103660 0x7fc2d4198930 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400c970 tx=0x7fc2c400ce30 comp rx=0 tx=0).stop 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 shutdown_connections 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc2c00a32d0 0x7fc2c00a5790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2d410c170 0x7fc2d419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d4103660 0x7fc2d4198930 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 --2- 172.21.15.28:0/369669994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2d4102d40 0x7fc2d41983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.396+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 >> 172.21.15.28:0/369669994 conn(0x7fc2d40fc7a0 msgr2=0x7fc2d40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.397+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 shutdown_connections 2024-09-15T21:56:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:30.397+0000 7fc2dbf05640 1 -- 172.21.15.28:0/369669994 wait complete. 2024-09-15T21:56:31.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:30 smithi028 ceph-mon[30969]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:31.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:30 smithi028 ceph-mon[30969]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:56:31.115 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-15T21:56:29.561136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660741,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":671,"num_read_kb":464,"num_write":434,"num_write_kb":3082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":354,"ondisk_log_size":354,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363516,"kb_used_data":14192,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578308,"statfs":{"total":1151940427776,"available":1151568187392,"internally_reserved":0,"allocated":14532608,"data_stored":8024381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":130,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003154"},"pg_stats":[{"pgid":"2.1f","version":"88'13","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:23.632714+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:56:23.632714+0000","last_peered":"2024-09-15T21:56:23.632714+0000","last_clean":"2024-09-15T21:56:23.632714+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:56:23.632714+0000","last_undegraded":"2024-09-15T21:56:23.632714+0000","last_fullsized":"2024-09-15T21:56:23.632714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812304+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:55:58.812304+0000","last_peered":"2024-09-15T21:55:58.812304+0000","last_clean":"2024-09-15T21:55:58.812304+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:55:58.812304+0000","last_undegraded":"2024-09-15T21:55:58.812304+0000","last_fullsized":"2024-09-15T21:55:58.812304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795704+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:55:58.795704+0000","last_peered":"2024-09-15T21:55:58.795704+0000","last_clean":"2024-09-15T21:55:58.795704+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:55:58.795704+0000","last_undegraded":"2024-09-15T21:55:58.795704+0000","last_fullsized":"2024-09-15T21:55:58.795704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812493+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:55:58.812493+0000","last_peered":"2024-09-15T21:55:58.812493+0000","last_clean":"2024-09-15T21:55:58.812493+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:55:58.812493+0000","last_undegraded":"2024-09-15T21:55:58.812493+0000","last_fullsized":"2024-09-15T21:55:58.812493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811770+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:55:58.811770+0000","last_peered":"2024-09-15T21:55:58.811770+0000","last_clean":"2024-09-15T21:55:58.811770+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:55:58.811770+0000","last_undegraded":"2024-09-15T21:55:58.811770+0000","last_fullsized":"2024-09-15T21:55:58.811770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796959+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:55:58.796959+0000","last_peered":"2024-09-15T21:55:58.796959+0000","last_clean":"2024-09-15T21:55:58.796959+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:55:58.796959+0000","last_undegraded":"2024-09-15T21:55:58.796959+0000","last_fullsized":"2024-09-15T21:55:58.796959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.813316+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:55:58.813316+0000","last_peered":"2024-09-15T21:55:58.813316+0000","last_clean":"2024-09-15T21:55:58.813316+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:55:58.813316+0000","last_undegraded":"2024-09-15T21:55:58.813316+0000","last_fullsized":"2024-09-15T21:55:58.813316+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793109+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:55:58.793109+0000","last_peered":"2024-09-15T21:55:58.793109+0000","last_clean":"2024-09-15T21:55:58.793109+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:55:58.793109+0000","last_undegraded":"2024-09-15T21:55:58.793109+0000","last_fullsized":"2024-09-15T21:55:58.793109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.798113+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:55:58.798113+0000","last_peered":"2024-09-15T21:55:58.798113+0000","last_clean":"2024-09-15T21:55:58.798113+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:55:58.798113+0000","last_undegraded":"2024-09-15T21:55:58.798113+0000","last_fullsized":"2024-09-15T21:55:58.798113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.808786+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:55:58.808786+0000","last_peered":"2024-09-15T21:55:58.808786+0000","last_clean":"2024-09-15T21:55:58.808786+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:55:58.808786+0000","last_undegraded":"2024-09-15T21:55:58.808786+0000","last_fullsized":"2024-09-15T21:55:58.808786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793413+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:55:58.793413+0000","last_peered":"2024-09-15T21:55:58.793413+0000","last_clean":"2024-09-15T21:55:58.793413+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:55:58.793413+0000","last_undegraded":"2024-09-15T21:55:58.793413+0000","last_fullsized":"2024-09-15T21:55:58.793413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797100+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:55:58.797100+0000","last_peered":"2024-09-15T21:55:58.797100+0000","last_clean":"2024-09-15T21:55:58.797100+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:55:58.797100+0000","last_undegraded":"2024-09-15T21:55:58.797100+0000","last_fullsized":"2024-09-15T21:55:58.797100+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796067+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:55:58.796067+0000","last_peered":"2024-09-15T21:55:58.796067+0000","last_clean":"2024-09-15T21:55:58.796067+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:55:58.796067+0000","last_undegraded":"2024-09-15T21:55:58.796067+0000","last_fullsized":"2024-09-15T21:55:58.796067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801982+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:55:58.801982+0000","last_peered":"2024-09-15T21:55:58.801982+0000","last_clean":"2024-09-15T21:55:58.801982+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:55:58.801982+0000","last_undegraded":"2024-09-15T21:55:58.801982+0000","last_fullsized":"2024-09-15T21:55:58.801982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797273+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:55:58.797273+0000","last_peered":"2024-09-15T21:55:58.797273+0000","last_clean":"2024-09-15T21:55:58.797273+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:55:58.797273+0000","last_undegraded":"2024-09-15T21:55:58.797273+0000","last_fullsized":"2024-09-15T21:55:58.797273+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812085+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:55:58.812085+0000","last_peered":"2024-09-15T21:55:58.812085+0000","last_clean":"2024-09-15T21:55:58.812085+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:55:58.812085+0000","last_undegraded":"2024-09-15T21:55:58.812085+0000","last_fullsized":"2024-09-15T21:55:58.812085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795111+0000","last_change":"2024-09-15T21:54:56.343335+0000","last_active":"2024-09-15T21:55:58.795111+0000","last_peered":"2024-09-15T21:55:58.795111+0000","last_clean":"2024-09-15T21:55:58.795111+0000","last_became_active":"2024-09-15T21:54:56.343011+0000","last_became_peered":"2024-09-15T21:54:56.343011+0000","last_unstale":"2024-09-15T21:55:58.795111+0000","last_undegraded":"2024-09-15T21:55:58.795111+0000","last_fullsized":"2024-09-15T21:55:58.795111+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:57.013184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.820660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:55:58.820660+0000","last_peered":"2024-09-15T21:55:58.820660+0000","last_clean":"2024-09-15T21:55:58.820660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:55:58.820660+0000","last_undegraded":"2024-09-15T21:55:58.820660+0000","last_fullsized":"2024-09-15T21:55:58.820660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794505+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:55:58.794505+0000","last_peered":"2024-09-15T21:55:58.794505+0000","last_clean":"2024-09-15T21:55:58.794505+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:55:58.794505+0000","last_undegraded":"2024-09-15T21:55:58.794505+0000","last_fullsized":"2024-09-15T21:55:58.794505+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812038+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:55:58.812038+0000","last_peered":"2024-09-15T21:55:58.812038+0000","last_clean":"2024-09-15T21:55:58.812038+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:55:58.812038+0000","last_undegraded":"2024-09-15T21:55:58.812038+0000","last_fullsized":"2024-09-15T21:55:58.812038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795227+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:55:58.795227+0000","last_peered":"2024-09-15T21:55:58.795227+0000","last_clean":"2024-09-15T21:55:58.795227+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:55:58.795227+0000","last_undegraded":"2024-09-15T21:55:58.795227+0000","last_fullsized":"2024-09-15T21:55:58.795227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.800986+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:55:58.800986+0000","last_peered":"2024-09-15T21:55:58.800986+0000","last_clean":"2024-09-15T21:55:58.800986+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:55:58.800986+0000","last_undegraded":"2024-09-15T21:55:58.800986+0000","last_fullsized":"2024-09-15T21:55:58.800986+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792814+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:55:58.792814+0000","last_peered":"2024-09-15T21:55:58.792814+0000","last_clean":"2024-09-15T21:55:58.792814+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:55:58.792814+0000","last_undegraded":"2024-09-15T21:55:58.792814+0000","last_fullsized":"2024-09-15T21:55:58.792814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797185+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:55:58.797185+0000","last_peered":"2024-09-15T21:55:58.797185+0000","last_clean":"2024-09-15T21:55:58.797185+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:55:58.797185+0000","last_undegraded":"2024-09-15T21:55:58.797185+0000","last_fullsized":"2024-09-15T21:55:58.797185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811457+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:55:58.811457+0000","last_peered":"2024-09-15T21:55:58.811457+0000","last_clean":"2024-09-15T21:55:58.811457+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:55:58.811457+0000","last_undegraded":"2024-09-15T21:55:58.811457+0000","last_fullsized":"2024-09-15T21:55:58.811457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797641+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:55:58.797641+0000","last_peered":"2024-09-15T21:55:58.797641+0000","last_clean":"2024-09-15T21:55:58.797641+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:55:58.797641+0000","last_undegraded":"2024-09-15T21:55:58.797641+0000","last_fullsized":"2024-09-15T21:55:58.797641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796594+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:55:58.796594+0000","last_peered":"2024-09-15T21:55:58.796594+0000","last_clean":"2024-09-15T21:55:58.796594+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:55:58.796594+0000","last_undegraded":"2024-09-15T21:55:58.796594+0000","last_fullsized":"2024-09-15T21:55:58.796594+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812452+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:55:58.812452+0000","last_peered":"2024-09-15T21:55:58.812452+0000","last_clean":"2024-09-15T21:55:58.812452+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:55:58.812452+0000","last_undegraded":"2024-09-15T21:55:58.812452+0000","last_fullsized":"2024-09-15T21:55:58.812452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'17","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795207+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:55:58.795207+0000","last_peered":"2024-09-15T21:55:58.795207+0000","last_clean":"2024-09-15T21:55:58.795207+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:55:58.795207+0000","last_undegraded":"2024-09-15T21:55:58.795207+0000","last_fullsized":"2024-09-15T21:55:58.795207+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811578+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:55:58.811578+0000","last_peered":"2024-09-15T21:55:58.811578+0000","last_clean":"2024-09-15T21:55:58.811578+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:55:58.811578+0000","last_undegraded":"2024-09-15T21:55:58.811578+0000","last_fullsized":"2024-09-15T21:55:58.811578+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792295+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:55:58.792295+0000","last_peered":"2024-09-15T21:55:58.792295+0000","last_clean":"2024-09-15T21:55:58.792295+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:55:58.792295+0000","last_undegraded":"2024-09-15T21:55:58.792295+0000","last_fullsized":"2024-09-15T21:55:58.792295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802418+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:55:58.802418+0000","last_peered":"2024-09-15T21:55:58.802418+0000","last_clean":"2024-09-15T21:55:58.802418+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:55:58.802418+0000","last_undegraded":"2024-09-15T21:55:58.802418+0000","last_fullsized":"2024-09-15T21:55:58.802418+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796538+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:55:58.796538+0000","last_peered":"2024-09-15T21:55:58.796538+0000","last_clean":"2024-09-15T21:55:58.796538+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:55:58.796538+0000","last_undegraded":"2024-09-15T21:55:58.796538+0000","last_fullsized":"2024-09-15T21:55:58.796538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812387+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:55:58.812387+0000","last_peered":"2024-09-15T21:55:58.812387+0000","last_clean":"2024-09-15T21:55:58.812387+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:55:58.812387+0000","last_undegraded":"2024-09-15T21:55:58.812387+0000","last_fullsized":"2024-09-15T21:55:58.812387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795409+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:55:58.795409+0000","last_peered":"2024-09-15T21:55:58.795409+0000","last_clean":"2024-09-15T21:55:58.795409+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:55:58.795409+0000","last_undegraded":"2024-09-15T21:55:58.795409+0000","last_fullsized":"2024-09-15T21:55:58.795409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801128+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:55:58.801128+0000","last_peered":"2024-09-15T21:55:58.801128+0000","last_clean":"2024-09-15T21:55:58.801128+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:55:58.801128+0000","last_undegraded":"2024-09-15T21:55:58.801128+0000","last_fullsized":"2024-09-15T21:55:58.801128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812022+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:55:58.812022+0000","last_peered":"2024-09-15T21:55:58.812022+0000","last_clean":"2024-09-15T21:55:58.812022+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:55:58.812022+0000","last_undegraded":"2024-09-15T21:55:58.812022+0000","last_fullsized":"2024-09-15T21:55:58.812022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:26.637739+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:56:26.637739+0000","last_peered":"2024-09-15T21:56:26.637739+0000","last_clean":"2024-09-15T21:56:26.637739+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:56:26.637739+0000","last_undegraded":"2024-09-15T21:56:26.637739+0000","last_fullsized":"2024-09-15T21:56:26.637739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792164+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:55:58.792164+0000","last_peered":"2024-09-15T21:55:58.792164+0000","last_clean":"2024-09-15T21:55:58.792164+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:55:58.792164+0000","last_undegraded":"2024-09-15T21:55:58.792164+0000","last_fullsized":"2024-09-15T21:55:58.792164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792705+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:55:58.792705+0000","last_peered":"2024-09-15T21:55:58.792705+0000","last_clean":"2024-09-15T21:55:58.792705+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:55:58.792705+0000","last_undegraded":"2024-09-15T21:55:58.792705+0000","last_fullsized":"2024-09-15T21:55:58.792705+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:25.636410+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:56:25.636410+0000","last_peered":"2024-09-15T21:56:25.636410+0000","last_clean":"2024-09-15T21:56:25.636410+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:56:25.636410+0000","last_undegraded":"2024-09-15T21:56:25.636410+0000","last_fullsized":"2024-09-15T21:56:25.636410+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.813389+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:55:58.813389+0000","last_peered":"2024-09-15T21:55:58.813389+0000","last_clean":"2024-09-15T21:55:58.813389+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:55:58.813389+0000","last_undegraded":"2024-09-15T21:55:58.813389+0000","last_fullsized":"2024-09-15T21:55:58.813389+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791729+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:55:58.791729+0000","last_peered":"2024-09-15T21:55:58.791729+0000","last_clean":"2024-09-15T21:55:58.791729+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:55:58.791729+0000","last_undegraded":"2024-09-15T21:55:58.791729+0000","last_fullsized":"2024-09-15T21:55:58.791729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796412+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:55:58.796412+0000","last_peered":"2024-09-15T21:55:58.796412+0000","last_clean":"2024-09-15T21:55:58.796412+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:55:58.796412+0000","last_undegraded":"2024-09-15T21:55:58.796412+0000","last_fullsized":"2024-09-15T21:55:58.796412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793036+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:55:58.793036+0000","last_peered":"2024-09-15T21:55:58.793036+0000","last_clean":"2024-09-15T21:55:58.793036+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:55:58.793036+0000","last_undegraded":"2024-09-15T21:55:58.793036+0000","last_fullsized":"2024-09-15T21:55:58.793036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812161+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:55:58.812161+0000","last_peered":"2024-09-15T21:55:58.812161+0000","last_clean":"2024-09-15T21:55:58.812161+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:55:58.812161+0000","last_undegraded":"2024-09-15T21:55:58.812161+0000","last_fullsized":"2024-09-15T21:55:58.812161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.798507+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:55:58.798507+0000","last_peered":"2024-09-15T21:55:58.798507+0000","last_clean":"2024-09-15T21:55:58.798507+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:55:58.798507+0000","last_undegraded":"2024-09-15T21:55:58.798507+0000","last_fullsized":"2024-09-15T21:55:58.798507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797065+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:55:58.797065+0000","last_peered":"2024-09-15T21:55:58.797065+0000","last_clean":"2024-09-15T21:55:58.797065+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:55:58.797065+0000","last_undegraded":"2024-09-15T21:55:58.797065+0000","last_fullsized":"2024-09-15T21:55:58.797065+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.818314+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:55:58.818314+0000","last_peered":"2024-09-15T21:55:58.818314+0000","last_clean":"2024-09-15T21:55:58.818314+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:55:58.818314+0000","last_undegraded":"2024-09-15T21:55:58.818314+0000","last_fullsized":"2024-09-15T21:55:58.818314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797531+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:55:58.797531+0000","last_peered":"2024-09-15T21:55:58.797531+0000","last_clean":"2024-09-15T21:55:58.797531+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:55:58.797531+0000","last_undegraded":"2024-09-15T21:55:58.797531+0000","last_fullsized":"2024-09-15T21:55:58.797531+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'6","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:26.638373+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:56:26.638373+0000","last_peered":"2024-09-15T21:56:26.638373+0000","last_clean":"2024-09-15T21:56:26.638373+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:56:26.638373+0000","last_undegraded":"2024-09-15T21:56:26.638373+0000","last_fullsized":"2024-09-15T21:56:26.638373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792879+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:55:58.792879+0000","last_peered":"2024-09-15T21:55:58.792879+0000","last_clean":"2024-09-15T21:55:58.792879+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:55:58.792879+0000","last_undegraded":"2024-09-15T21:55:58.792879+0000","last_fullsized":"2024-09-15T21:55:58.792879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796560+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:55:58.796560+0000","last_peered":"2024-09-15T21:55:58.796560+0000","last_clean":"2024-09-15T21:55:58.796560+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:55:58.796560+0000","last_undegraded":"2024-09-15T21:55:58.796560+0000","last_fullsized":"2024-09-15T21:55:58.796560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793644+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:55:58.793644+0000","last_peered":"2024-09-15T21:55:58.793644+0000","last_clean":"2024-09-15T21:55:58.793644+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:55:58.793644+0000","last_undegraded":"2024-09-15T21:55:58.793644+0000","last_fullsized":"2024-09-15T21:55:58.793644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811248+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:55:58.811248+0000","last_peered":"2024-09-15T21:55:58.811248+0000","last_clean":"2024-09-15T21:55:58.811248+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:55:58.811248+0000","last_undegraded":"2024-09-15T21:55:58.811248+0000","last_fullsized":"2024-09-15T21:55:58.811248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796645+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:55:58.796645+0000","last_peered":"2024-09-15T21:55:58.796645+0000","last_clean":"2024-09-15T21:55:58.796645+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:55:58.796645+0000","last_undegraded":"2024-09-15T21:55:58.796645+0000","last_fullsized":"2024-09-15T21:55:58.796645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792423+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:55:58.792423+0000","last_peered":"2024-09-15T21:55:58.792423+0000","last_clean":"2024-09-15T21:55:58.792423+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:55:58.792423+0000","last_undegraded":"2024-09-15T21:55:58.792423+0000","last_fullsized":"2024-09-15T21:55:58.792423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792482+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:55:58.792482+0000","last_peered":"2024-09-15T21:55:58.792482+0000","last_clean":"2024-09-15T21:55:58.792482+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:55:58.792482+0000","last_undegraded":"2024-09-15T21:55:58.792482+0000","last_fullsized":"2024-09-15T21:55:58.792482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794381+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:55:58.794381+0000","last_peered":"2024-09-15T21:55:58.794381+0000","last_clean":"2024-09-15T21:55:58.794381+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:55:58.794381+0000","last_undegraded":"2024-09-15T21:55:58.794381+0000","last_fullsized":"2024-09-15T21:55:58.794381+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802898+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:55:58.802898+0000","last_peered":"2024-09-15T21:55:58.802898+0000","last_clean":"2024-09-15T21:55:58.802898+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:55:58.802898+0000","last_undegraded":"2024-09-15T21:55:58.802898+0000","last_fullsized":"2024-09-15T21:55:58.802898+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792571+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:55:58.792571+0000","last_peered":"2024-09-15T21:55:58.792571+0000","last_clean":"2024-09-15T21:55:58.792571+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:55:58.792571+0000","last_undegraded":"2024-09-15T21:55:58.792571+0000","last_fullsized":"2024-09-15T21:55:58.792571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792927+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:55:58.792927+0000","last_peered":"2024-09-15T21:55:58.792927+0000","last_clean":"2024-09-15T21:55:58.792927+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:55:58.792927+0000","last_undegraded":"2024-09-15T21:55:58.792927+0000","last_fullsized":"2024-09-15T21:55:58.792927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.800991+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:55:58.800991+0000","last_peered":"2024-09-15T21:55:58.800991+0000","last_clean":"2024-09-15T21:55:58.800991+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:55:58.800991+0000","last_undegraded":"2024-09-15T21:55:58.800991+0000","last_fullsized":"2024-09-15T21:55:58.800991+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792342+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:55:58.792342+0000","last_peered":"2024-09-15T21:55:58.792342+0000","last_clean":"2024-09-15T21:55:58.792342+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:55:58.792342+0000","last_undegraded":"2024-09-15T21:55:58.792342+0000","last_fullsized":"2024-09-15T21:55:58.792342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796265+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:55:58.796265+0000","last_peered":"2024-09-15T21:55:58.796265+0000","last_clean":"2024-09-15T21:55:58.796265+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:55:58.796265+0000","last_undegraded":"2024-09-15T21:55:58.796265+0000","last_fullsized":"2024-09-15T21:55:58.796265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:23.634393+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:56:23.634393+0000","last_peered":"2024-09-15T21:56:23.634393+0000","last_clean":"2024-09-15T21:56:23.634393+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:56:23.634393+0000","last_undegraded":"2024-09-15T21:56:23.634393+0000","last_fullsized":"2024-09-15T21:56:23.634393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795287+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:55:58.795287+0000","last_peered":"2024-09-15T21:55:58.795287+0000","last_clean":"2024-09-15T21:55:58.795287+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:55:58.795287+0000","last_undegraded":"2024-09-15T21:55:58.795287+0000","last_fullsized":"2024-09-15T21:55:58.795287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796269+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:55:58.796269+0000","last_peered":"2024-09-15T21:55:58.796269+0000","last_clean":"2024-09-15T21:55:58.796269+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:55:58.796269+0000","last_undegraded":"2024-09-15T21:55:58.796269+0000","last_fullsized":"2024-09-15T21:55:58.796269+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793083+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:55:58.793083+0000","last_peered":"2024-09-15T21:55:58.793083+0000","last_clean":"2024-09-15T21:55:58.793083+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:55:58.793083+0000","last_undegraded":"2024-09-15T21:55:58.793083+0000","last_fullsized":"2024-09-15T21:55:58.793083+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802722+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:55:58.802722+0000","last_peered":"2024-09-15T21:55:58.802722+0000","last_clean":"2024-09-15T21:55:58.802722+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:55:58.802722+0000","last_undegraded":"2024-09-15T21:55:58.802722+0000","last_fullsized":"2024-09-15T21:55:58.802722+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791892+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:55:58.791892+0000","last_peered":"2024-09-15T21:55:58.791892+0000","last_clean":"2024-09-15T21:55:58.791892+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:55:58.791892+0000","last_undegraded":"2024-09-15T21:55:58.791892+0000","last_fullsized":"2024-09-15T21:55:58.791892+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795280+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:55:58.795280+0000","last_peered":"2024-09-15T21:55:58.795280+0000","last_clean":"2024-09-15T21:55:58.795280+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:55:58.795280+0000","last_undegraded":"2024-09-15T21:55:58.795280+0000","last_fullsized":"2024-09-15T21:55:58.795280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793377+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:55:58.793377+0000","last_peered":"2024-09-15T21:55:58.793377+0000","last_clean":"2024-09-15T21:55:58.793377+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:55:58.793377+0000","last_undegraded":"2024-09-15T21:55:58.793377+0000","last_fullsized":"2024-09-15T21:55:58.793377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802075+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:55:58.802075+0000","last_peered":"2024-09-15T21:55:58.802075+0000","last_clean":"2024-09-15T21:55:58.802075+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:55:58.802075+0000","last_undegraded":"2024-09-15T21:55:58.802075+0000","last_fullsized":"2024-09-15T21:55:58.802075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"89'34","reported_seq":179,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:28.640442+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:56:28.640442+0000","last_peered":"2024-09-15T21:56:28.640442+0000","last_clean":"2024-09-15T21:56:28.640442+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:56:28.640442+0000","last_undegraded":"2024-09-15T21:56:28.640442+0000","last_fullsized":"2024-09-15T21:56:28.640442+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62931,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":111,"num_read_kb":116,"num_write":36,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794798+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:55:58.794798+0000","last_peered":"2024-09-15T21:55:58.794798+0000","last_clean":"2024-09-15T21:55:58.794798+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:55:58.794798+0000","last_undegraded":"2024-09-15T21:55:58.794798+0000","last_fullsized":"2024-09-15T21:55:58.794798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793217+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:55:58.793217+0000","last_peered":"2024-09-15T21:55:58.793217+0000","last_clean":"2024-09-15T21:55:58.793217+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:55:58.793217+0000","last_undegraded":"2024-09-15T21:55:58.793217+0000","last_fullsized":"2024-09-15T21:55:58.793217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793579+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:55:58.793579+0000","last_peered":"2024-09-15T21:55:58.793579+0000","last_clean":"2024-09-15T21:55:58.793579+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:55:58.793579+0000","last_undegraded":"2024-09-15T21:55:58.793579+0000","last_fullsized":"2024-09-15T21:55:58.793579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797529+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:55:58.797529+0000","last_peered":"2024-09-15T21:55:58.797529+0000","last_clean":"2024-09-15T21:55:58.797529+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:55:58.797529+0000","last_undegraded":"2024-09-15T21:55:58.797529+0000","last_fullsized":"2024-09-15T21:55:58.797529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811625+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:55:58.811625+0000","last_peered":"2024-09-15T21:55:58.811625+0000","last_clean":"2024-09-15T21:55:58.811625+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:55:58.811625+0000","last_undegraded":"2024-09-15T21:55:58.811625+0000","last_fullsized":"2024-09-15T21:55:58.811625+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796574+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:55:58.796574+0000","last_peered":"2024-09-15T21:55:58.796574+0000","last_clean":"2024-09-15T21:55:58.796574+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:55:58.796574+0000","last_undegraded":"2024-09-15T21:55:58.796574+0000","last_fullsized":"2024-09-15T21:55:58.796574+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794583+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:55:58.794583+0000","last_peered":"2024-09-15T21:55:58.794583+0000","last_clean":"2024-09-15T21:55:58.794583+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:55:58.794583+0000","last_undegraded":"2024-09-15T21:55:58.794583+0000","last_fullsized":"2024-09-15T21:55:58.794583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793277+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:55:58.793277+0000","last_peered":"2024-09-15T21:55:58.793277+0000","last_clean":"2024-09-15T21:55:58.793277+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:55:58.793277+0000","last_undegraded":"2024-09-15T21:55:58.793277+0000","last_fullsized":"2024-09-15T21:55:58.793277+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:28.640444+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:56:28.640444+0000","last_peered":"2024-09-15T21:56:28.640444+0000","last_clean":"2024-09-15T21:56:28.640444+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:56:28.640444+0000","last_undegraded":"2024-09-15T21:56:28.640444+0000","last_fullsized":"2024-09-15T21:56:28.640444+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811753+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:55:58.811753+0000","last_peered":"2024-09-15T21:55:58.811753+0000","last_clean":"2024-09-15T21:55:58.811753+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:55:58.811753+0000","last_undegraded":"2024-09-15T21:55:58.811753+0000","last_fullsized":"2024-09-15T21:55:58.811753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811350+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:55:58.811350+0000","last_peered":"2024-09-15T21:55:58.811350+0000","last_clean":"2024-09-15T21:55:58.811350+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:55:58.811350+0000","last_undegraded":"2024-09-15T21:55:58.811350+0000","last_fullsized":"2024-09-15T21:55:58.811350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794691+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:55:58.794691+0000","last_peered":"2024-09-15T21:55:58.794691+0000","last_clean":"2024-09-15T21:55:58.794691+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:55:58.794691+0000","last_undegraded":"2024-09-15T21:55:58.794691+0000","last_fullsized":"2024-09-15T21:55:58.794691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793736+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:55:58.793736+0000","last_peered":"2024-09-15T21:55:58.793736+0000","last_clean":"2024-09-15T21:55:58.793736+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:55:58.793736+0000","last_undegraded":"2024-09-15T21:55:58.793736+0000","last_fullsized":"2024-09-15T21:55:58.793736+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794395+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:55:58.794395+0000","last_peered":"2024-09-15T21:55:58.794395+0000","last_clean":"2024-09-15T21:55:58.794395+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:55:58.794395+0000","last_undegraded":"2024-09-15T21:55:58.794395+0000","last_fullsized":"2024-09-15T21:55:58.794395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811845+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:55:58.811845+0000","last_peered":"2024-09-15T21:55:58.811845+0000","last_clean":"2024-09-15T21:55:58.811845+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:55:58.811845+0000","last_undegraded":"2024-09-15T21:55:58.811845+0000","last_fullsized":"2024-09-15T21:55:58.811845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791746+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:55:58.791746+0000","last_peered":"2024-09-15T21:55:58.791746+0000","last_clean":"2024-09-15T21:55:58.791746+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:55:58.791746+0000","last_undegraded":"2024-09-15T21:55:58.791746+0000","last_fullsized":"2024-09-15T21:55:58.791746+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812149+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:55:58.812149+0000","last_peered":"2024-09-15T21:55:58.812149+0000","last_clean":"2024-09-15T21:55:58.812149+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:55:58.812149+0000","last_undegraded":"2024-09-15T21:55:58.812149+0000","last_fullsized":"2024-09-15T21:55:58.812149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812666+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:55:58.812666+0000","last_peered":"2024-09-15T21:55:58.812666+0000","last_clean":"2024-09-15T21:55:58.812666+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:55:58.812666+0000","last_undegraded":"2024-09-15T21:55:58.812666+0000","last_fullsized":"2024-09-15T21:55:58.812666+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795011+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:55:58.795011+0000","last_peered":"2024-09-15T21:55:58.795011+0000","last_clean":"2024-09-15T21:55:58.795011+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:55:58.795011+0000","last_undegraded":"2024-09-15T21:55:58.795011+0000","last_fullsized":"2024-09-15T21:55:58.795011+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:24.635755+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:56:24.635755+0000","last_peered":"2024-09-15T21:56:24.635755+0000","last_clean":"2024-09-15T21:56:24.635755+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:56:24.635755+0000","last_undegraded":"2024-09-15T21:56:24.635755+0000","last_fullsized":"2024-09-15T21:56:24.635755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801307+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:55:58.801307+0000","last_peered":"2024-09-15T21:55:58.801307+0000","last_clean":"2024-09-15T21:55:58.801307+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:55:58.801307+0000","last_undegraded":"2024-09-15T21:55:58.801307+0000","last_fullsized":"2024-09-15T21:55:58.801307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795851+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:55:58.795851+0000","last_peered":"2024-09-15T21:55:58.795851+0000","last_clean":"2024-09-15T21:55:58.795851+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:55:58.795851+0000","last_undegraded":"2024-09-15T21:55:58.795851+0000","last_fullsized":"2024-09-15T21:55:58.795851+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794437+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:55:58.794437+0000","last_peered":"2024-09-15T21:55:58.794437+0000","last_clean":"2024-09-15T21:55:58.794437+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:55:58.794437+0000","last_undegraded":"2024-09-15T21:55:58.794437+0000","last_fullsized":"2024-09-15T21:55:58.794437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.799175+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:55:58.799175+0000","last_peered":"2024-09-15T21:55:58.799175+0000","last_clean":"2024-09-15T21:55:58.799175+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:55:58.799175+0000","last_undegraded":"2024-09-15T21:55:58.799175+0000","last_fullsized":"2024-09-15T21:55:58.799175+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795007+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:55:58.795007+0000","last_peered":"2024-09-15T21:55:58.795007+0000","last_clean":"2024-09-15T21:55:58.795007+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:55:58.795007+0000","last_undegraded":"2024-09-15T21:55:58.795007+0000","last_fullsized":"2024-09-15T21:55:58.795007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811344+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:55:58.811344+0000","last_peered":"2024-09-15T21:55:58.811344+0000","last_clean":"2024-09-15T21:55:58.811344+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:55:58.811344+0000","last_undegraded":"2024-09-15T21:55:58.811344+0000","last_fullsized":"2024-09-15T21:55:58.811344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792287+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:55:58.792287+0000","last_peered":"2024-09-15T21:55:58.792287+0000","last_clean":"2024-09-15T21:55:58.792287+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:55:58.792287+0000","last_undegraded":"2024-09-15T21:55:58.792287+0000","last_fullsized":"2024-09-15T21:55:58.792287+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796430+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:55:58.796430+0000","last_peered":"2024-09-15T21:55:58.796430+0000","last_clean":"2024-09-15T21:55:58.796430+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:55:58.796430+0000","last_undegraded":"2024-09-15T21:55:58.796430+0000","last_fullsized":"2024-09-15T21:55:58.796430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796451+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:55:58.796451+0000","last_peered":"2024-09-15T21:55:58.796451+0000","last_clean":"2024-09-15T21:55:58.796451+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:55:58.796451+0000","last_undegraded":"2024-09-15T21:55:58.796451+0000","last_fullsized":"2024-09-15T21:55:58.796451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812407+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:55:58.812407+0000","last_peered":"2024-09-15T21:55:58.812407+0000","last_clean":"2024-09-15T21:55:58.812407+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:55:58.812407+0000","last_undegraded":"2024-09-15T21:55:58.812407+0000","last_fullsized":"2024-09-15T21:55:58.812407+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792643+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:55:58.792643+0000","last_peered":"2024-09-15T21:55:58.792643+0000","last_clean":"2024-09-15T21:55:58.792643+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:55:58.792643+0000","last_undegraded":"2024-09-15T21:55:58.792643+0000","last_fullsized":"2024-09-15T21:55:58.792643+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811965+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:55:58.811965+0000","last_peered":"2024-09-15T21:55:58.811965+0000","last_clean":"2024-09-15T21:55:58.811965+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:55:58.811965+0000","last_undegraded":"2024-09-15T21:55:58.811965+0000","last_fullsized":"2024-09-15T21:55:58.811965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793197+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:55:58.793197+0000","last_peered":"2024-09-15T21:55:58.793197+0000","last_clean":"2024-09-15T21:55:58.793197+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:55:58.793197+0000","last_undegraded":"2024-09-15T21:55:58.793197+0000","last_fullsized":"2024-09-15T21:55:58.793197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792017+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:55:58.792017+0000","last_peered":"2024-09-15T21:55:58.792017+0000","last_clean":"2024-09-15T21:55:58.792017+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:55:58.792017+0000","last_undegraded":"2024-09-15T21:55:58.792017+0000","last_fullsized":"2024-09-15T21:55:58.792017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.804422+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:55:58.804422+0000","last_peered":"2024-09-15T21:55:58.804422+0000","last_clean":"2024-09-15T21:55:58.804422+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:55:58.804422+0000","last_undegraded":"2024-09-15T21:55:58.804422+0000","last_fullsized":"2024-09-15T21:55:58.804422+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796186+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:55:58.796186+0000","last_peered":"2024-09-15T21:55:58.796186+0000","last_clean":"2024-09-15T21:55:58.796186+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:55:58.796186+0000","last_undegraded":"2024-09-15T21:55:58.796186+0000","last_fullsized":"2024-09-15T21:55:58.796186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795141+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:55:58.795141+0000","last_peered":"2024-09-15T21:55:58.795141+0000","last_clean":"2024-09-15T21:55:58.795141+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:55:58.795141+0000","last_undegraded":"2024-09-15T21:55:58.795141+0000","last_fullsized":"2024-09-15T21:55:58.795141+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797114+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:55:58.797114+0000","last_peered":"2024-09-15T21:55:58.797114+0000","last_clean":"2024-09-15T21:55:58.797114+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:55:58.797114+0000","last_undegraded":"2024-09-15T21:55:58.797114+0000","last_fullsized":"2024-09-15T21:55:58.797114+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792689+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:55:58.792689+0000","last_peered":"2024-09-15T21:55:58.792689+0000","last_clean":"2024-09-15T21:55:58.792689+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:55:58.792689+0000","last_undegraded":"2024-09-15T21:55:58.792689+0000","last_fullsized":"2024-09-15T21:55:58.792689+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811116+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:55:58.811116+0000","last_peered":"2024-09-15T21:55:58.811116+0000","last_clean":"2024-09-15T21:55:58.811116+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:55:58.811116+0000","last_undegraded":"2024-09-15T21:55:58.811116+0000","last_fullsized":"2024-09-15T21:55:58.811116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811413+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:55:58.811413+0000","last_peered":"2024-09-15T21:55:58.811413+0000","last_clean":"2024-09-15T21:55:58.811413+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:55:58.811413+0000","last_undegraded":"2024-09-15T21:55:58.811413+0000","last_fullsized":"2024-09-15T21:55:58.811413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791995+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:55:58.791995+0000","last_peered":"2024-09-15T21:55:58.791995+0000","last_clean":"2024-09-15T21:55:58.791995+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:55:58.791995+0000","last_undegraded":"2024-09-15T21:55:58.791995+0000","last_fullsized":"2024-09-15T21:55:58.791995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.814043+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:55:58.814043+0000","last_peered":"2024-09-15T21:55:58.814043+0000","last_clean":"2024-09-15T21:55:58.814043+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:55:58.814043+0000","last_undegraded":"2024-09-15T21:55:58.814043+0000","last_fullsized":"2024-09-15T21:55:58.814043+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797394+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:55:58.797394+0000","last_peered":"2024-09-15T21:55:58.797394+0000","last_clean":"2024-09-15T21:55:58.797394+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:55:58.797394+0000","last_undegraded":"2024-09-15T21:55:58.797394+0000","last_fullsized":"2024-09-15T21:55:58.797394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812422+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:55:58.812422+0000","last_peered":"2024-09-15T21:55:58.812422+0000","last_clean":"2024-09-15T21:55:58.812422+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:55:58.812422+0000","last_undegraded":"2024-09-15T21:55:58.812422+0000","last_fullsized":"2024-09-15T21:55:58.812422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793433+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:55:58.793433+0000","last_peered":"2024-09-15T21:55:58.793433+0000","last_clean":"2024-09-15T21:55:58.793433+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:55:58.793433+0000","last_undegraded":"2024-09-15T21:55:58.793433+0000","last_fullsized":"2024-09-15T21:55:58.793433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811741+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:55:58.811741+0000","last_peered":"2024-09-15T21:55:58.811741+0000","last_clean":"2024-09-15T21:55:58.811741+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:55:58.811741+0000","last_undegraded":"2024-09-15T21:55:58.811741+0000","last_fullsized":"2024-09-15T21:55:58.811741+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795631+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:55:58.795631+0000","last_peered":"2024-09-15T21:55:58.795631+0000","last_clean":"2024-09-15T21:55:58.795631+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:55:58.795631+0000","last_undegraded":"2024-09-15T21:55:58.795631+0000","last_fullsized":"2024-09-15T21:55:58.795631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812079+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:55:58.812079+0000","last_peered":"2024-09-15T21:55:58.812079+0000","last_clean":"2024-09-15T21:55:58.812079+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:55:58.812079+0000","last_undegraded":"2024-09-15T21:55:58.812079+0000","last_fullsized":"2024-09-15T21:55:58.812079+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801845+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:55:58.801845+0000","last_peered":"2024-09-15T21:55:58.801845+0000","last_clean":"2024-09-15T21:55:58.801845+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:55:58.801845+0000","last_undegraded":"2024-09-15T21:55:58.801845+0000","last_fullsized":"2024-09-15T21:55:58.801845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.798740+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:55:58.798740+0000","last_peered":"2024-09-15T21:55:58.798740+0000","last_clean":"2024-09-15T21:55:58.798740+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:55:58.798740+0000","last_undegraded":"2024-09-15T21:55:58.798740+0000","last_fullsized":"2024-09-15T21:55:58.798740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811194+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:55:58.811194+0000","last_peered":"2024-09-15T21:55:58.811194+0000","last_clean":"2024-09-15T21:55:58.811194+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:55:58.811194+0000","last_undegraded":"2024-09-15T21:55:58.811194+0000","last_fullsized":"2024-09-15T21:55:58.811194+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811351+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:55:58.811351+0000","last_peered":"2024-09-15T21:55:58.811351+0000","last_clean":"2024-09-15T21:55:58.811351+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:55:58.811351+0000","last_undegraded":"2024-09-15T21:55:58.811351+0000","last_fullsized":"2024-09-15T21:55:58.811351+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796989+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:55:58.796989+0000","last_peered":"2024-09-15T21:55:58.796989+0000","last_clean":"2024-09-15T21:55:58.796989+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:55:58.796989+0000","last_undegraded":"2024-09-15T21:55:58.796989+0000","last_fullsized":"2024-09-15T21:55:58.796989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70139,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":535,"num_read_kb":345,"num_write":251,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":210417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514528,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1032192,"data_stored":507104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776164,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28720,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716432,"statfs":{"total":95995035648,"available":95965626368,"internally_reserved":0,"allocated":1732608,"data_stored":1164068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939690,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1048576,"data_stored":501963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168623,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1044480,"data_stored":505855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397557,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1646592,"data_stored":1102936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561080,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1028096,"data_stored":501563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790015,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1654784,"data_stored":1098518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953540,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1036288,"data_stored":505637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1040384,"data_stored":501271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1118208,"data_stored":566530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1097728,"data_stored":565524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":503412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:56:31.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T21:56:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:30 smithi203 ceph-mon[35546]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:31.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:30 smithi203 ceph-mon[35546]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:56:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:30 smithi174 ceph-mon[35244]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:31.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:30 smithi174 ceph-mon[35244]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:56:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 -- 172.21.15.28:0/1297260796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c106f10 msgr2=0x7f439c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 --2- 172.21.15.28:0/1297260796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c106f10 0x7f439c10ea90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4394009ca0 tx=0x7f439402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:56:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 -- 172.21.15.28:0/1297260796 shutdown_connections 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 --2- 172.21.15.28:0/1297260796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c106f10 0x7f439c10ea90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 --2- 172.21.15.28:0/1297260796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f439c102790 0x7f439c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 --2- 172.21.15.28:0/1297260796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f439c101de0 0x7f439c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 -- 172.21.15.28:0/1297260796 >> 172.21.15.28:0/1297260796 conn(0x7f439c0fd8c0 msgr2=0x7f439c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 -- 172.21.15.28:0/1297260796 shutdown_connections 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 -- 172.21.15.28:0/1297260796 wait complete. 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.829+0000 7f43a14be640 1 Processor -- start 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f43a14be640 1 -- start start 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f43a14be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f439c101de0 0x7f439c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f43a14be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c102790 0x7f439c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f43a14be640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f439c106f10 0x7f439c19ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f43a14be640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f439c111f70 con 0x7f439c102790 2024-09-15T21:56:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f43a14be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f439c111df0 con 0x7f439c101de0 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f43a14be640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1120f0 con 0x7f439c106f10 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c102790 0x7f439c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c102790 0x7f439c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35446/0 (socket says 172.21.15.28:35446) 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 -- 172.21.15.28:0/4245489169 learned_addr learned my addr 172.21.15.28:0/4245489169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f439affd640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f439c101de0 0x7f439c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 -- 172.21.15.28:0/4245489169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f439c106f10 msgr2=0x7f439c19ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f439c106f10 0x7f439c19ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 -- 172.21.15.28:0/4245489169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f439c101de0 msgr2=0x7f439c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f439c101de0 0x7f439c198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.830+0000 7f4393fff640 1 -- 172.21.15.28:0/4245489169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439c19d120 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.831+0000 7f4393fff640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c102790 0x7f439c1986b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f438400d950 tx=0x7f438400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.831+0000 7f4398ff9640 1 -- 172.21.15.28:0/4245489169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384017030 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.831+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439c19d410 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.831+0000 7f4398ff9640 1 -- 172.21.15.28:0/4245489169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4384004020 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.831+0000 7f4398ff9640 1 -- 172.21.15.28:0/4245489169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384005650 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.831+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f439c1a4cf0 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.832+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f439c1994d0 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.833+0000 7f4398ff9640 1 -- 172.21.15.28:0/4245489169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f43840041d0 con 0x7f439c102790 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.834+0000 7f4398ff9640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f43880a32d0 0x7f43880a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.834+0000 7f439affd640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f43880a32d0 0x7f43880a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.834+0000 7f4398ff9640 1 -- 172.21.15.28:0/4245489169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f43840cfec0 con 0x7f439c102790 2024-09-15T21:56:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.834+0000 7f439affd640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f43880a32d0 0x7f43880a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f438c00abe0 tx=0x7f438c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:31.837+0000 7f4398ff9640 1 -- 172.21.15.28:0/4245489169 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4384024330 con 0x7f439c102790 2024-09-15T21:56:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.010+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f439c0fe940 con 0x7f43880a32d0 2024-09-15T21:56:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.018+0000 7f4398ff9640 1 -- 172.21.15.28:0/4245489169 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340418 (secure 0 0 0) 0x7f439c0fe940 con 0x7f43880a32d0 2024-09-15T21:56:32.023 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:32.025 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f43880a32d0 msgr2=0x7f43880a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f43880a32d0 0x7f43880a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f438c00abe0 tx=0x7f438c00a2d0 comp rx=0 tx=0).stop 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c102790 msgr2=0x7f439c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c102790 0x7f439c1986b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f438400d950 tx=0x7f438400dd50 comp rx=0 tx=0).stop 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 shutdown_connections 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f43880a32d0 0x7f43880a5790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f439c106f10 0x7f439c19ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439c102790 0x7f439c1986b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:32.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 --2- 172.21.15.28:0/4245489169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f439c101de0 0x7f439c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:32.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 >> 172.21.15.28:0/4245489169 conn(0x7f439c0fd8c0 msgr2=0x7f439c0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:32.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 shutdown_connections 2024-09-15T21:56:32.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:32.021+0000 7f43a14be640 1 -- 172.21.15.28:0/4245489169 wait complete. 2024-09-15T21:56:32.818 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-15T21:56:31.561698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660741,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":686,"num_read_kb":469,"num_write":434,"num_write_kb":3082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":354,"ondisk_log_size":354,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363516,"kb_used_data":14192,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578308,"statfs":{"total":1151940427776,"available":1151568187392,"internally_reserved":0,"allocated":14532608,"data_stored":8024381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003279"},"pg_stats":[{"pgid":"2.1f","version":"88'13","reported_seq":100,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:28.639676+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:56:28.639676+0000","last_peered":"2024-09-15T21:56:28.639676+0000","last_clean":"2024-09-15T21:56:28.639676+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:56:28.639676+0000","last_undegraded":"2024-09-15T21:56:28.639676+0000","last_fullsized":"2024-09-15T21:56:28.639676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812304+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:55:58.812304+0000","last_peered":"2024-09-15T21:55:58.812304+0000","last_clean":"2024-09-15T21:55:58.812304+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:55:58.812304+0000","last_undegraded":"2024-09-15T21:55:58.812304+0000","last_fullsized":"2024-09-15T21:55:58.812304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795704+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:55:58.795704+0000","last_peered":"2024-09-15T21:55:58.795704+0000","last_clean":"2024-09-15T21:55:58.795704+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:55:58.795704+0000","last_undegraded":"2024-09-15T21:55:58.795704+0000","last_fullsized":"2024-09-15T21:55:58.795704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812493+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:55:58.812493+0000","last_peered":"2024-09-15T21:55:58.812493+0000","last_clean":"2024-09-15T21:55:58.812493+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:55:58.812493+0000","last_undegraded":"2024-09-15T21:55:58.812493+0000","last_fullsized":"2024-09-15T21:55:58.812493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811770+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:55:58.811770+0000","last_peered":"2024-09-15T21:55:58.811770+0000","last_clean":"2024-09-15T21:55:58.811770+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:55:58.811770+0000","last_undegraded":"2024-09-15T21:55:58.811770+0000","last_fullsized":"2024-09-15T21:55:58.811770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796959+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:55:58.796959+0000","last_peered":"2024-09-15T21:55:58.796959+0000","last_clean":"2024-09-15T21:55:58.796959+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:55:58.796959+0000","last_undegraded":"2024-09-15T21:55:58.796959+0000","last_fullsized":"2024-09-15T21:55:58.796959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.813316+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:55:58.813316+0000","last_peered":"2024-09-15T21:55:58.813316+0000","last_clean":"2024-09-15T21:55:58.813316+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:55:58.813316+0000","last_undegraded":"2024-09-15T21:55:58.813316+0000","last_fullsized":"2024-09-15T21:55:58.813316+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793109+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:55:58.793109+0000","last_peered":"2024-09-15T21:55:58.793109+0000","last_clean":"2024-09-15T21:55:58.793109+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:55:58.793109+0000","last_undegraded":"2024-09-15T21:55:58.793109+0000","last_fullsized":"2024-09-15T21:55:58.793109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.798113+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:55:58.798113+0000","last_peered":"2024-09-15T21:55:58.798113+0000","last_clean":"2024-09-15T21:55:58.798113+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:55:58.798113+0000","last_undegraded":"2024-09-15T21:55:58.798113+0000","last_fullsized":"2024-09-15T21:55:58.798113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.808786+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:55:58.808786+0000","last_peered":"2024-09-15T21:55:58.808786+0000","last_clean":"2024-09-15T21:55:58.808786+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:55:58.808786+0000","last_undegraded":"2024-09-15T21:55:58.808786+0000","last_fullsized":"2024-09-15T21:55:58.808786+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793413+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:55:58.793413+0000","last_peered":"2024-09-15T21:55:58.793413+0000","last_clean":"2024-09-15T21:55:58.793413+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:55:58.793413+0000","last_undegraded":"2024-09-15T21:55:58.793413+0000","last_fullsized":"2024-09-15T21:55:58.793413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797100+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:55:58.797100+0000","last_peered":"2024-09-15T21:55:58.797100+0000","last_clean":"2024-09-15T21:55:58.797100+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:55:58.797100+0000","last_undegraded":"2024-09-15T21:55:58.797100+0000","last_fullsized":"2024-09-15T21:55:58.797100+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796067+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:55:58.796067+0000","last_peered":"2024-09-15T21:55:58.796067+0000","last_clean":"2024-09-15T21:55:58.796067+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:55:58.796067+0000","last_undegraded":"2024-09-15T21:55:58.796067+0000","last_fullsized":"2024-09-15T21:55:58.796067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801982+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:55:58.801982+0000","last_peered":"2024-09-15T21:55:58.801982+0000","last_clean":"2024-09-15T21:55:58.801982+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:55:58.801982+0000","last_undegraded":"2024-09-15T21:55:58.801982+0000","last_fullsized":"2024-09-15T21:55:58.801982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797273+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:55:58.797273+0000","last_peered":"2024-09-15T21:55:58.797273+0000","last_clean":"2024-09-15T21:55:58.797273+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:55:58.797273+0000","last_undegraded":"2024-09-15T21:55:58.797273+0000","last_fullsized":"2024-09-15T21:55:58.797273+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812085+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:55:58.812085+0000","last_peered":"2024-09-15T21:55:58.812085+0000","last_clean":"2024-09-15T21:55:58.812085+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:55:58.812085+0000","last_undegraded":"2024-09-15T21:55:58.812085+0000","last_fullsized":"2024-09-15T21:55:58.812085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795111+0000","last_change":"2024-09-15T21:54:56.343335+0000","last_active":"2024-09-15T21:55:58.795111+0000","last_peered":"2024-09-15T21:55:58.795111+0000","last_clean":"2024-09-15T21:55:58.795111+0000","last_became_active":"2024-09-15T21:54:56.343011+0000","last_became_peered":"2024-09-15T21:54:56.343011+0000","last_unstale":"2024-09-15T21:55:58.795111+0000","last_undegraded":"2024-09-15T21:55:58.795111+0000","last_fullsized":"2024-09-15T21:55:58.795111+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:57.013184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.820660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:55:58.820660+0000","last_peered":"2024-09-15T21:55:58.820660+0000","last_clean":"2024-09-15T21:55:58.820660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:55:58.820660+0000","last_undegraded":"2024-09-15T21:55:58.820660+0000","last_fullsized":"2024-09-15T21:55:58.820660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794505+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:55:58.794505+0000","last_peered":"2024-09-15T21:55:58.794505+0000","last_clean":"2024-09-15T21:55:58.794505+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:55:58.794505+0000","last_undegraded":"2024-09-15T21:55:58.794505+0000","last_fullsized":"2024-09-15T21:55:58.794505+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812038+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:55:58.812038+0000","last_peered":"2024-09-15T21:55:58.812038+0000","last_clean":"2024-09-15T21:55:58.812038+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:55:58.812038+0000","last_undegraded":"2024-09-15T21:55:58.812038+0000","last_fullsized":"2024-09-15T21:55:58.812038+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795227+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:55:58.795227+0000","last_peered":"2024-09-15T21:55:58.795227+0000","last_clean":"2024-09-15T21:55:58.795227+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:55:58.795227+0000","last_undegraded":"2024-09-15T21:55:58.795227+0000","last_fullsized":"2024-09-15T21:55:58.795227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.800986+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:55:58.800986+0000","last_peered":"2024-09-15T21:55:58.800986+0000","last_clean":"2024-09-15T21:55:58.800986+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:55:58.800986+0000","last_undegraded":"2024-09-15T21:55:58.800986+0000","last_fullsized":"2024-09-15T21:55:58.800986+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792814+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:55:58.792814+0000","last_peered":"2024-09-15T21:55:58.792814+0000","last_clean":"2024-09-15T21:55:58.792814+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:55:58.792814+0000","last_undegraded":"2024-09-15T21:55:58.792814+0000","last_fullsized":"2024-09-15T21:55:58.792814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797185+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:55:58.797185+0000","last_peered":"2024-09-15T21:55:58.797185+0000","last_clean":"2024-09-15T21:55:58.797185+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:55:58.797185+0000","last_undegraded":"2024-09-15T21:55:58.797185+0000","last_fullsized":"2024-09-15T21:55:58.797185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811457+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:55:58.811457+0000","last_peered":"2024-09-15T21:55:58.811457+0000","last_clean":"2024-09-15T21:55:58.811457+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:55:58.811457+0000","last_undegraded":"2024-09-15T21:55:58.811457+0000","last_fullsized":"2024-09-15T21:55:58.811457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797641+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:55:58.797641+0000","last_peered":"2024-09-15T21:55:58.797641+0000","last_clean":"2024-09-15T21:55:58.797641+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:55:58.797641+0000","last_undegraded":"2024-09-15T21:55:58.797641+0000","last_fullsized":"2024-09-15T21:55:58.797641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796594+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:55:58.796594+0000","last_peered":"2024-09-15T21:55:58.796594+0000","last_clean":"2024-09-15T21:55:58.796594+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:55:58.796594+0000","last_undegraded":"2024-09-15T21:55:58.796594+0000","last_fullsized":"2024-09-15T21:55:58.796594+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812452+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:55:58.812452+0000","last_peered":"2024-09-15T21:55:58.812452+0000","last_clean":"2024-09-15T21:55:58.812452+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:55:58.812452+0000","last_undegraded":"2024-09-15T21:55:58.812452+0000","last_fullsized":"2024-09-15T21:55:58.812452+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'17","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795207+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:55:58.795207+0000","last_peered":"2024-09-15T21:55:58.795207+0000","last_clean":"2024-09-15T21:55:58.795207+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:55:58.795207+0000","last_undegraded":"2024-09-15T21:55:58.795207+0000","last_fullsized":"2024-09-15T21:55:58.795207+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811578+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:55:58.811578+0000","last_peered":"2024-09-15T21:55:58.811578+0000","last_clean":"2024-09-15T21:55:58.811578+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:55:58.811578+0000","last_undegraded":"2024-09-15T21:55:58.811578+0000","last_fullsized":"2024-09-15T21:55:58.811578+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792295+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:55:58.792295+0000","last_peered":"2024-09-15T21:55:58.792295+0000","last_clean":"2024-09-15T21:55:58.792295+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:55:58.792295+0000","last_undegraded":"2024-09-15T21:55:58.792295+0000","last_fullsized":"2024-09-15T21:55:58.792295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802418+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:55:58.802418+0000","last_peered":"2024-09-15T21:55:58.802418+0000","last_clean":"2024-09-15T21:55:58.802418+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:55:58.802418+0000","last_undegraded":"2024-09-15T21:55:58.802418+0000","last_fullsized":"2024-09-15T21:55:58.802418+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796538+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:55:58.796538+0000","last_peered":"2024-09-15T21:55:58.796538+0000","last_clean":"2024-09-15T21:55:58.796538+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:55:58.796538+0000","last_undegraded":"2024-09-15T21:55:58.796538+0000","last_fullsized":"2024-09-15T21:55:58.796538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812387+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:55:58.812387+0000","last_peered":"2024-09-15T21:55:58.812387+0000","last_clean":"2024-09-15T21:55:58.812387+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:55:58.812387+0000","last_undegraded":"2024-09-15T21:55:58.812387+0000","last_fullsized":"2024-09-15T21:55:58.812387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795409+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:55:58.795409+0000","last_peered":"2024-09-15T21:55:58.795409+0000","last_clean":"2024-09-15T21:55:58.795409+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:55:58.795409+0000","last_undegraded":"2024-09-15T21:55:58.795409+0000","last_fullsized":"2024-09-15T21:55:58.795409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801128+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:55:58.801128+0000","last_peered":"2024-09-15T21:55:58.801128+0000","last_clean":"2024-09-15T21:55:58.801128+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:55:58.801128+0000","last_undegraded":"2024-09-15T21:55:58.801128+0000","last_fullsized":"2024-09-15T21:55:58.801128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812022+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:55:58.812022+0000","last_peered":"2024-09-15T21:55:58.812022+0000","last_clean":"2024-09-15T21:55:58.812022+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:55:58.812022+0000","last_undegraded":"2024-09-15T21:55:58.812022+0000","last_fullsized":"2024-09-15T21:55:58.812022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:26.637739+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:56:26.637739+0000","last_peered":"2024-09-15T21:56:26.637739+0000","last_clean":"2024-09-15T21:56:26.637739+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:56:26.637739+0000","last_undegraded":"2024-09-15T21:56:26.637739+0000","last_fullsized":"2024-09-15T21:56:26.637739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792164+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:55:58.792164+0000","last_peered":"2024-09-15T21:55:58.792164+0000","last_clean":"2024-09-15T21:55:58.792164+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:55:58.792164+0000","last_undegraded":"2024-09-15T21:55:58.792164+0000","last_fullsized":"2024-09-15T21:55:58.792164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792705+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:55:58.792705+0000","last_peered":"2024-09-15T21:55:58.792705+0000","last_clean":"2024-09-15T21:55:58.792705+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:55:58.792705+0000","last_undegraded":"2024-09-15T21:55:58.792705+0000","last_fullsized":"2024-09-15T21:55:58.792705+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:25.636410+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:56:25.636410+0000","last_peered":"2024-09-15T21:56:25.636410+0000","last_clean":"2024-09-15T21:56:25.636410+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:56:25.636410+0000","last_undegraded":"2024-09-15T21:56:25.636410+0000","last_fullsized":"2024-09-15T21:56:25.636410+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.813389+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:55:58.813389+0000","last_peered":"2024-09-15T21:55:58.813389+0000","last_clean":"2024-09-15T21:55:58.813389+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:55:58.813389+0000","last_undegraded":"2024-09-15T21:55:58.813389+0000","last_fullsized":"2024-09-15T21:55:58.813389+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791729+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:55:58.791729+0000","last_peered":"2024-09-15T21:55:58.791729+0000","last_clean":"2024-09-15T21:55:58.791729+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:55:58.791729+0000","last_undegraded":"2024-09-15T21:55:58.791729+0000","last_fullsized":"2024-09-15T21:55:58.791729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796412+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:55:58.796412+0000","last_peered":"2024-09-15T21:55:58.796412+0000","last_clean":"2024-09-15T21:55:58.796412+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:55:58.796412+0000","last_undegraded":"2024-09-15T21:55:58.796412+0000","last_fullsized":"2024-09-15T21:55:58.796412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793036+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:55:58.793036+0000","last_peered":"2024-09-15T21:55:58.793036+0000","last_clean":"2024-09-15T21:55:58.793036+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:55:58.793036+0000","last_undegraded":"2024-09-15T21:55:58.793036+0000","last_fullsized":"2024-09-15T21:55:58.793036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812161+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:55:58.812161+0000","last_peered":"2024-09-15T21:55:58.812161+0000","last_clean":"2024-09-15T21:55:58.812161+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:55:58.812161+0000","last_undegraded":"2024-09-15T21:55:58.812161+0000","last_fullsized":"2024-09-15T21:55:58.812161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.798507+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:55:58.798507+0000","last_peered":"2024-09-15T21:55:58.798507+0000","last_clean":"2024-09-15T21:55:58.798507+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:55:58.798507+0000","last_undegraded":"2024-09-15T21:55:58.798507+0000","last_fullsized":"2024-09-15T21:55:58.798507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797065+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:55:58.797065+0000","last_peered":"2024-09-15T21:55:58.797065+0000","last_clean":"2024-09-15T21:55:58.797065+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:55:58.797065+0000","last_undegraded":"2024-09-15T21:55:58.797065+0000","last_fullsized":"2024-09-15T21:55:58.797065+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.818314+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:55:58.818314+0000","last_peered":"2024-09-15T21:55:58.818314+0000","last_clean":"2024-09-15T21:55:58.818314+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:55:58.818314+0000","last_undegraded":"2024-09-15T21:55:58.818314+0000","last_fullsized":"2024-09-15T21:55:58.818314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797531+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:55:58.797531+0000","last_peered":"2024-09-15T21:55:58.797531+0000","last_clean":"2024-09-15T21:55:58.797531+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:55:58.797531+0000","last_undegraded":"2024-09-15T21:55:58.797531+0000","last_fullsized":"2024-09-15T21:55:58.797531+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'6","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:26.638373+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:56:26.638373+0000","last_peered":"2024-09-15T21:56:26.638373+0000","last_clean":"2024-09-15T21:56:26.638373+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:56:26.638373+0000","last_undegraded":"2024-09-15T21:56:26.638373+0000","last_fullsized":"2024-09-15T21:56:26.638373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1563,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792879+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:55:58.792879+0000","last_peered":"2024-09-15T21:55:58.792879+0000","last_clean":"2024-09-15T21:55:58.792879+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:55:58.792879+0000","last_undegraded":"2024-09-15T21:55:58.792879+0000","last_fullsized":"2024-09-15T21:55:58.792879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796560+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:55:58.796560+0000","last_peered":"2024-09-15T21:55:58.796560+0000","last_clean":"2024-09-15T21:55:58.796560+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:55:58.796560+0000","last_undegraded":"2024-09-15T21:55:58.796560+0000","last_fullsized":"2024-09-15T21:55:58.796560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793644+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:55:58.793644+0000","last_peered":"2024-09-15T21:55:58.793644+0000","last_clean":"2024-09-15T21:55:58.793644+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:55:58.793644+0000","last_undegraded":"2024-09-15T21:55:58.793644+0000","last_fullsized":"2024-09-15T21:55:58.793644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811248+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:55:58.811248+0000","last_peered":"2024-09-15T21:55:58.811248+0000","last_clean":"2024-09-15T21:55:58.811248+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:55:58.811248+0000","last_undegraded":"2024-09-15T21:55:58.811248+0000","last_fullsized":"2024-09-15T21:55:58.811248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796645+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:55:58.796645+0000","last_peered":"2024-09-15T21:55:58.796645+0000","last_clean":"2024-09-15T21:55:58.796645+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:55:58.796645+0000","last_undegraded":"2024-09-15T21:55:58.796645+0000","last_fullsized":"2024-09-15T21:55:58.796645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792423+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:55:58.792423+0000","last_peered":"2024-09-15T21:55:58.792423+0000","last_clean":"2024-09-15T21:55:58.792423+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:55:58.792423+0000","last_undegraded":"2024-09-15T21:55:58.792423+0000","last_fullsized":"2024-09-15T21:55:58.792423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792482+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:55:58.792482+0000","last_peered":"2024-09-15T21:55:58.792482+0000","last_clean":"2024-09-15T21:55:58.792482+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:55:58.792482+0000","last_undegraded":"2024-09-15T21:55:58.792482+0000","last_fullsized":"2024-09-15T21:55:58.792482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794381+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:55:58.794381+0000","last_peered":"2024-09-15T21:55:58.794381+0000","last_clean":"2024-09-15T21:55:58.794381+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:55:58.794381+0000","last_undegraded":"2024-09-15T21:55:58.794381+0000","last_fullsized":"2024-09-15T21:55:58.794381+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802898+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:55:58.802898+0000","last_peered":"2024-09-15T21:55:58.802898+0000","last_clean":"2024-09-15T21:55:58.802898+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:55:58.802898+0000","last_undegraded":"2024-09-15T21:55:58.802898+0000","last_fullsized":"2024-09-15T21:55:58.802898+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792571+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:55:58.792571+0000","last_peered":"2024-09-15T21:55:58.792571+0000","last_clean":"2024-09-15T21:55:58.792571+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:55:58.792571+0000","last_undegraded":"2024-09-15T21:55:58.792571+0000","last_fullsized":"2024-09-15T21:55:58.792571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792927+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:55:58.792927+0000","last_peered":"2024-09-15T21:55:58.792927+0000","last_clean":"2024-09-15T21:55:58.792927+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:55:58.792927+0000","last_undegraded":"2024-09-15T21:55:58.792927+0000","last_fullsized":"2024-09-15T21:55:58.792927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.800991+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:55:58.800991+0000","last_peered":"2024-09-15T21:55:58.800991+0000","last_clean":"2024-09-15T21:55:58.800991+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:55:58.800991+0000","last_undegraded":"2024-09-15T21:55:58.800991+0000","last_fullsized":"2024-09-15T21:55:58.800991+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792342+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:55:58.792342+0000","last_peered":"2024-09-15T21:55:58.792342+0000","last_clean":"2024-09-15T21:55:58.792342+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:55:58.792342+0000","last_undegraded":"2024-09-15T21:55:58.792342+0000","last_fullsized":"2024-09-15T21:55:58.792342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796265+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:55:58.796265+0000","last_peered":"2024-09-15T21:55:58.796265+0000","last_clean":"2024-09-15T21:55:58.796265+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:55:58.796265+0000","last_undegraded":"2024-09-15T21:55:58.796265+0000","last_fullsized":"2024-09-15T21:55:58.796265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:28.640803+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:56:28.640803+0000","last_peered":"2024-09-15T21:56:28.640803+0000","last_clean":"2024-09-15T21:56:28.640803+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:56:28.640803+0000","last_undegraded":"2024-09-15T21:56:28.640803+0000","last_fullsized":"2024-09-15T21:56:28.640803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795287+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:55:58.795287+0000","last_peered":"2024-09-15T21:55:58.795287+0000","last_clean":"2024-09-15T21:55:58.795287+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:55:58.795287+0000","last_undegraded":"2024-09-15T21:55:58.795287+0000","last_fullsized":"2024-09-15T21:55:58.795287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796269+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:55:58.796269+0000","last_peered":"2024-09-15T21:55:58.796269+0000","last_clean":"2024-09-15T21:55:58.796269+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:55:58.796269+0000","last_undegraded":"2024-09-15T21:55:58.796269+0000","last_fullsized":"2024-09-15T21:55:58.796269+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793083+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:55:58.793083+0000","last_peered":"2024-09-15T21:55:58.793083+0000","last_clean":"2024-09-15T21:55:58.793083+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:55:58.793083+0000","last_undegraded":"2024-09-15T21:55:58.793083+0000","last_fullsized":"2024-09-15T21:55:58.793083+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802722+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:55:58.802722+0000","last_peered":"2024-09-15T21:55:58.802722+0000","last_clean":"2024-09-15T21:55:58.802722+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:55:58.802722+0000","last_undegraded":"2024-09-15T21:55:58.802722+0000","last_fullsized":"2024-09-15T21:55:58.802722+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791892+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:55:58.791892+0000","last_peered":"2024-09-15T21:55:58.791892+0000","last_clean":"2024-09-15T21:55:58.791892+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:55:58.791892+0000","last_undegraded":"2024-09-15T21:55:58.791892+0000","last_fullsized":"2024-09-15T21:55:58.791892+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795280+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:55:58.795280+0000","last_peered":"2024-09-15T21:55:58.795280+0000","last_clean":"2024-09-15T21:55:58.795280+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:55:58.795280+0000","last_undegraded":"2024-09-15T21:55:58.795280+0000","last_fullsized":"2024-09-15T21:55:58.795280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793377+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:55:58.793377+0000","last_peered":"2024-09-15T21:55:58.793377+0000","last_clean":"2024-09-15T21:55:58.793377+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:55:58.793377+0000","last_undegraded":"2024-09-15T21:55:58.793377+0000","last_fullsized":"2024-09-15T21:55:58.793377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.802075+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:55:58.802075+0000","last_peered":"2024-09-15T21:55:58.802075+0000","last_clean":"2024-09-15T21:55:58.802075+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:55:58.802075+0000","last_undegraded":"2024-09-15T21:55:58.802075+0000","last_fullsized":"2024-09-15T21:55:58.802075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"89'34","reported_seq":179,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:28.640442+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:56:28.640442+0000","last_peered":"2024-09-15T21:56:28.640442+0000","last_clean":"2024-09-15T21:56:28.640442+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:56:28.640442+0000","last_undegraded":"2024-09-15T21:56:28.640442+0000","last_fullsized":"2024-09-15T21:56:28.640442+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62931,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":111,"num_read_kb":116,"num_write":36,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794798+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:55:58.794798+0000","last_peered":"2024-09-15T21:55:58.794798+0000","last_clean":"2024-09-15T21:55:58.794798+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:55:58.794798+0000","last_undegraded":"2024-09-15T21:55:58.794798+0000","last_fullsized":"2024-09-15T21:55:58.794798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793217+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:55:58.793217+0000","last_peered":"2024-09-15T21:55:58.793217+0000","last_clean":"2024-09-15T21:55:58.793217+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:55:58.793217+0000","last_undegraded":"2024-09-15T21:55:58.793217+0000","last_fullsized":"2024-09-15T21:55:58.793217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793579+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:55:58.793579+0000","last_peered":"2024-09-15T21:55:58.793579+0000","last_clean":"2024-09-15T21:55:58.793579+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:55:58.793579+0000","last_undegraded":"2024-09-15T21:55:58.793579+0000","last_fullsized":"2024-09-15T21:55:58.793579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797529+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:55:58.797529+0000","last_peered":"2024-09-15T21:55:58.797529+0000","last_clean":"2024-09-15T21:55:58.797529+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:55:58.797529+0000","last_undegraded":"2024-09-15T21:55:58.797529+0000","last_fullsized":"2024-09-15T21:55:58.797529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811625+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:55:58.811625+0000","last_peered":"2024-09-15T21:55:58.811625+0000","last_clean":"2024-09-15T21:55:58.811625+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:55:58.811625+0000","last_undegraded":"2024-09-15T21:55:58.811625+0000","last_fullsized":"2024-09-15T21:55:58.811625+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796574+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:55:58.796574+0000","last_peered":"2024-09-15T21:55:58.796574+0000","last_clean":"2024-09-15T21:55:58.796574+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:55:58.796574+0000","last_undegraded":"2024-09-15T21:55:58.796574+0000","last_fullsized":"2024-09-15T21:55:58.796574+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794583+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:55:58.794583+0000","last_peered":"2024-09-15T21:55:58.794583+0000","last_clean":"2024-09-15T21:55:58.794583+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:55:58.794583+0000","last_undegraded":"2024-09-15T21:55:58.794583+0000","last_fullsized":"2024-09-15T21:55:58.794583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793277+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:55:58.793277+0000","last_peered":"2024-09-15T21:55:58.793277+0000","last_clean":"2024-09-15T21:55:58.793277+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:55:58.793277+0000","last_undegraded":"2024-09-15T21:55:58.793277+0000","last_fullsized":"2024-09-15T21:55:58.793277+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:28.640444+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:56:28.640444+0000","last_peered":"2024-09-15T21:56:28.640444+0000","last_clean":"2024-09-15T21:56:28.640444+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:56:28.640444+0000","last_undegraded":"2024-09-15T21:56:28.640444+0000","last_fullsized":"2024-09-15T21:56:28.640444+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811753+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:55:58.811753+0000","last_peered":"2024-09-15T21:55:58.811753+0000","last_clean":"2024-09-15T21:55:58.811753+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:55:58.811753+0000","last_undegraded":"2024-09-15T21:55:58.811753+0000","last_fullsized":"2024-09-15T21:55:58.811753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811350+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:55:58.811350+0000","last_peered":"2024-09-15T21:55:58.811350+0000","last_clean":"2024-09-15T21:55:58.811350+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:55:58.811350+0000","last_undegraded":"2024-09-15T21:55:58.811350+0000","last_fullsized":"2024-09-15T21:55:58.811350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794691+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:55:58.794691+0000","last_peered":"2024-09-15T21:55:58.794691+0000","last_clean":"2024-09-15T21:55:58.794691+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:55:58.794691+0000","last_undegraded":"2024-09-15T21:55:58.794691+0000","last_fullsized":"2024-09-15T21:55:58.794691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793736+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:55:58.793736+0000","last_peered":"2024-09-15T21:55:58.793736+0000","last_clean":"2024-09-15T21:55:58.793736+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:55:58.793736+0000","last_undegraded":"2024-09-15T21:55:58.793736+0000","last_fullsized":"2024-09-15T21:55:58.793736+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794395+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:55:58.794395+0000","last_peered":"2024-09-15T21:55:58.794395+0000","last_clean":"2024-09-15T21:55:58.794395+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:55:58.794395+0000","last_undegraded":"2024-09-15T21:55:58.794395+0000","last_fullsized":"2024-09-15T21:55:58.794395+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811845+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:55:58.811845+0000","last_peered":"2024-09-15T21:55:58.811845+0000","last_clean":"2024-09-15T21:55:58.811845+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:55:58.811845+0000","last_undegraded":"2024-09-15T21:55:58.811845+0000","last_fullsized":"2024-09-15T21:55:58.811845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791746+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:55:58.791746+0000","last_peered":"2024-09-15T21:55:58.791746+0000","last_clean":"2024-09-15T21:55:58.791746+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:55:58.791746+0000","last_undegraded":"2024-09-15T21:55:58.791746+0000","last_fullsized":"2024-09-15T21:55:58.791746+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812149+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:55:58.812149+0000","last_peered":"2024-09-15T21:55:58.812149+0000","last_clean":"2024-09-15T21:55:58.812149+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:55:58.812149+0000","last_undegraded":"2024-09-15T21:55:58.812149+0000","last_fullsized":"2024-09-15T21:55:58.812149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812666+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:55:58.812666+0000","last_peered":"2024-09-15T21:55:58.812666+0000","last_clean":"2024-09-15T21:55:58.812666+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:55:58.812666+0000","last_undegraded":"2024-09-15T21:55:58.812666+0000","last_fullsized":"2024-09-15T21:55:58.812666+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795011+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:55:58.795011+0000","last_peered":"2024-09-15T21:55:58.795011+0000","last_clean":"2024-09-15T21:55:58.795011+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:55:58.795011+0000","last_undegraded":"2024-09-15T21:55:58.795011+0000","last_fullsized":"2024-09-15T21:55:58.795011+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:56:29.642705+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:56:29.642705+0000","last_peered":"2024-09-15T21:56:29.642705+0000","last_clean":"2024-09-15T21:56:29.642705+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:56:29.642705+0000","last_undegraded":"2024-09-15T21:56:29.642705+0000","last_fullsized":"2024-09-15T21:56:29.642705+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801307+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:55:58.801307+0000","last_peered":"2024-09-15T21:55:58.801307+0000","last_clean":"2024-09-15T21:55:58.801307+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:55:58.801307+0000","last_undegraded":"2024-09-15T21:55:58.801307+0000","last_fullsized":"2024-09-15T21:55:58.801307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795851+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:55:58.795851+0000","last_peered":"2024-09-15T21:55:58.795851+0000","last_clean":"2024-09-15T21:55:58.795851+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:55:58.795851+0000","last_undegraded":"2024-09-15T21:55:58.795851+0000","last_fullsized":"2024-09-15T21:55:58.795851+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.794437+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:55:58.794437+0000","last_peered":"2024-09-15T21:55:58.794437+0000","last_clean":"2024-09-15T21:55:58.794437+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:55:58.794437+0000","last_undegraded":"2024-09-15T21:55:58.794437+0000","last_fullsized":"2024-09-15T21:55:58.794437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.799175+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:55:58.799175+0000","last_peered":"2024-09-15T21:55:58.799175+0000","last_clean":"2024-09-15T21:55:58.799175+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:55:58.799175+0000","last_undegraded":"2024-09-15T21:55:58.799175+0000","last_fullsized":"2024-09-15T21:55:58.799175+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795007+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:55:58.795007+0000","last_peered":"2024-09-15T21:55:58.795007+0000","last_clean":"2024-09-15T21:55:58.795007+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:55:58.795007+0000","last_undegraded":"2024-09-15T21:55:58.795007+0000","last_fullsized":"2024-09-15T21:55:58.795007+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811344+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:55:58.811344+0000","last_peered":"2024-09-15T21:55:58.811344+0000","last_clean":"2024-09-15T21:55:58.811344+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:55:58.811344+0000","last_undegraded":"2024-09-15T21:55:58.811344+0000","last_fullsized":"2024-09-15T21:55:58.811344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792287+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:55:58.792287+0000","last_peered":"2024-09-15T21:55:58.792287+0000","last_clean":"2024-09-15T21:55:58.792287+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:55:58.792287+0000","last_undegraded":"2024-09-15T21:55:58.792287+0000","last_fullsized":"2024-09-15T21:55:58.792287+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796430+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:55:58.796430+0000","last_peered":"2024-09-15T21:55:58.796430+0000","last_clean":"2024-09-15T21:55:58.796430+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:55:58.796430+0000","last_undegraded":"2024-09-15T21:55:58.796430+0000","last_fullsized":"2024-09-15T21:55:58.796430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796451+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:55:58.796451+0000","last_peered":"2024-09-15T21:55:58.796451+0000","last_clean":"2024-09-15T21:55:58.796451+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:55:58.796451+0000","last_undegraded":"2024-09-15T21:55:58.796451+0000","last_fullsized":"2024-09-15T21:55:58.796451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812407+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:55:58.812407+0000","last_peered":"2024-09-15T21:55:58.812407+0000","last_clean":"2024-09-15T21:55:58.812407+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:55:58.812407+0000","last_undegraded":"2024-09-15T21:55:58.812407+0000","last_fullsized":"2024-09-15T21:55:58.812407+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792643+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:55:58.792643+0000","last_peered":"2024-09-15T21:55:58.792643+0000","last_clean":"2024-09-15T21:55:58.792643+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:55:58.792643+0000","last_undegraded":"2024-09-15T21:55:58.792643+0000","last_fullsized":"2024-09-15T21:55:58.792643+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811965+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:55:58.811965+0000","last_peered":"2024-09-15T21:55:58.811965+0000","last_clean":"2024-09-15T21:55:58.811965+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:55:58.811965+0000","last_undegraded":"2024-09-15T21:55:58.811965+0000","last_fullsized":"2024-09-15T21:55:58.811965+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793197+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:55:58.793197+0000","last_peered":"2024-09-15T21:55:58.793197+0000","last_clean":"2024-09-15T21:55:58.793197+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:55:58.793197+0000","last_undegraded":"2024-09-15T21:55:58.793197+0000","last_fullsized":"2024-09-15T21:55:58.793197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792017+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:55:58.792017+0000","last_peered":"2024-09-15T21:55:58.792017+0000","last_clean":"2024-09-15T21:55:58.792017+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:55:58.792017+0000","last_undegraded":"2024-09-15T21:55:58.792017+0000","last_fullsized":"2024-09-15T21:55:58.792017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.804422+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:55:58.804422+0000","last_peered":"2024-09-15T21:55:58.804422+0000","last_clean":"2024-09-15T21:55:58.804422+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:55:58.804422+0000","last_undegraded":"2024-09-15T21:55:58.804422+0000","last_fullsized":"2024-09-15T21:55:58.804422+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796186+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:55:58.796186+0000","last_peered":"2024-09-15T21:55:58.796186+0000","last_clean":"2024-09-15T21:55:58.796186+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:55:58.796186+0000","last_undegraded":"2024-09-15T21:55:58.796186+0000","last_fullsized":"2024-09-15T21:55:58.796186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795141+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:55:58.795141+0000","last_peered":"2024-09-15T21:55:58.795141+0000","last_clean":"2024-09-15T21:55:58.795141+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:55:58.795141+0000","last_undegraded":"2024-09-15T21:55:58.795141+0000","last_fullsized":"2024-09-15T21:55:58.795141+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797114+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:55:58.797114+0000","last_peered":"2024-09-15T21:55:58.797114+0000","last_clean":"2024-09-15T21:55:58.797114+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:55:58.797114+0000","last_undegraded":"2024-09-15T21:55:58.797114+0000","last_fullsized":"2024-09-15T21:55:58.797114+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.792689+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:55:58.792689+0000","last_peered":"2024-09-15T21:55:58.792689+0000","last_clean":"2024-09-15T21:55:58.792689+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:55:58.792689+0000","last_undegraded":"2024-09-15T21:55:58.792689+0000","last_fullsized":"2024-09-15T21:55:58.792689+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811116+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:55:58.811116+0000","last_peered":"2024-09-15T21:55:58.811116+0000","last_clean":"2024-09-15T21:55:58.811116+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:55:58.811116+0000","last_undegraded":"2024-09-15T21:55:58.811116+0000","last_fullsized":"2024-09-15T21:55:58.811116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811413+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:55:58.811413+0000","last_peered":"2024-09-15T21:55:58.811413+0000","last_clean":"2024-09-15T21:55:58.811413+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:55:58.811413+0000","last_undegraded":"2024-09-15T21:55:58.811413+0000","last_fullsized":"2024-09-15T21:55:58.811413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.791995+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:55:58.791995+0000","last_peered":"2024-09-15T21:55:58.791995+0000","last_clean":"2024-09-15T21:55:58.791995+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:55:58.791995+0000","last_undegraded":"2024-09-15T21:55:58.791995+0000","last_fullsized":"2024-09-15T21:55:58.791995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.814043+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:55:58.814043+0000","last_peered":"2024-09-15T21:55:58.814043+0000","last_clean":"2024-09-15T21:55:58.814043+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:55:58.814043+0000","last_undegraded":"2024-09-15T21:55:58.814043+0000","last_fullsized":"2024-09-15T21:55:58.814043+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.797394+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:55:58.797394+0000","last_peered":"2024-09-15T21:55:58.797394+0000","last_clean":"2024-09-15T21:55:58.797394+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:55:58.797394+0000","last_undegraded":"2024-09-15T21:55:58.797394+0000","last_fullsized":"2024-09-15T21:55:58.797394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812422+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:55:58.812422+0000","last_peered":"2024-09-15T21:55:58.812422+0000","last_clean":"2024-09-15T21:55:58.812422+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:55:58.812422+0000","last_undegraded":"2024-09-15T21:55:58.812422+0000","last_fullsized":"2024-09-15T21:55:58.812422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.793433+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:55:58.793433+0000","last_peered":"2024-09-15T21:55:58.793433+0000","last_clean":"2024-09-15T21:55:58.793433+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:55:58.793433+0000","last_undegraded":"2024-09-15T21:55:58.793433+0000","last_fullsized":"2024-09-15T21:55:58.793433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811741+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:55:58.811741+0000","last_peered":"2024-09-15T21:55:58.811741+0000","last_clean":"2024-09-15T21:55:58.811741+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:55:58.811741+0000","last_undegraded":"2024-09-15T21:55:58.811741+0000","last_fullsized":"2024-09-15T21:55:58.811741+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.795631+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:55:58.795631+0000","last_peered":"2024-09-15T21:55:58.795631+0000","last_clean":"2024-09-15T21:55:58.795631+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:55:58.795631+0000","last_undegraded":"2024-09-15T21:55:58.795631+0000","last_fullsized":"2024-09-15T21:55:58.795631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.812079+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:55:58.812079+0000","last_peered":"2024-09-15T21:55:58.812079+0000","last_clean":"2024-09-15T21:55:58.812079+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:55:58.812079+0000","last_undegraded":"2024-09-15T21:55:58.812079+0000","last_fullsized":"2024-09-15T21:55:58.812079+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.801845+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:55:58.801845+0000","last_peered":"2024-09-15T21:55:58.801845+0000","last_clean":"2024-09-15T21:55:58.801845+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:55:58.801845+0000","last_undegraded":"2024-09-15T21:55:58.801845+0000","last_fullsized":"2024-09-15T21:55:58.801845+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.798740+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:55:58.798740+0000","last_peered":"2024-09-15T21:55:58.798740+0000","last_clean":"2024-09-15T21:55:58.798740+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:55:58.798740+0000","last_undegraded":"2024-09-15T21:55:58.798740+0000","last_fullsized":"2024-09-15T21:55:58.798740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811194+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:55:58.811194+0000","last_peered":"2024-09-15T21:55:58.811194+0000","last_clean":"2024-09-15T21:55:58.811194+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:55:58.811194+0000","last_undegraded":"2024-09-15T21:55:58.811194+0000","last_fullsized":"2024-09-15T21:55:58.811194+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.811351+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:55:58.811351+0000","last_peered":"2024-09-15T21:55:58.811351+0000","last_clean":"2024-09-15T21:55:58.811351+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:55:58.811351+0000","last_undegraded":"2024-09-15T21:55:58.811351+0000","last_fullsized":"2024-09-15T21:55:58.811351+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T21:55:58.796989+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:55:58.796989+0000","last_peered":"2024-09-15T21:55:58.796989+0000","last_clean":"2024-09-15T21:55:58.796989+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:55:58.796989+0000","last_undegraded":"2024-09-15T21:55:58.796989+0000","last_fullsized":"2024-09-15T21:55:58.796989+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70139,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":550,"num_read_kb":350,"num_write":251,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":210417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514528,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1032192,"data_stored":507104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776164,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28720,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716432,"statfs":{"total":95995035648,"available":95965626368,"internally_reserved":0,"allocated":1732608,"data_stored":1164068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939691,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1048576,"data_stored":501963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168623,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1044480,"data_stored":505855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397557,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1646592,"data_stored":1102936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561080,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1028096,"data_stored":501563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790016,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1654784,"data_stored":1098518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1036288,"data_stored":505637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1040384,"data_stored":501271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1118208,"data_stored":566530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1097728,"data_stored":565524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":503412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:56:32.824 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T21:56:32.825 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T21:56:32.825 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-15T21:56:32.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-15T21:56:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:32 smithi028 ceph-mon[30969]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:32 smithi028 ceph-mon[30969]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:56:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:32 smithi203 ceph-mon[35546]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:32 smithi203 ceph-mon[35546]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:56:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:32 smithi174 ceph-mon[35244]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:32 smithi174 ceph-mon[35244]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:56:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 -- 172.21.15.28:0/31200798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4106f50 msgr2=0x7f39b410eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 --2- 172.21.15.28:0/31200798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4106f50 0x7f39b410eaf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009cd0 tx=0x7f39ac02a8e0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 -- 172.21.15.28:0/31200798 shutdown_connections 2024-09-15T21:56:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 --2- 172.21.15.28:0/31200798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4106f50 0x7f39b410eaf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 --2- 172.21.15.28:0/31200798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39b4100ca0 0x7f39b4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 --2- 172.21.15.28:0/31200798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b41002f0 0x7f39b41006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 -- 172.21.15.28:0/31200798 >> 172.21.15.28:0/31200798 conn(0x7f39b4077210 msgr2=0x7f39b40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 -- 172.21.15.28:0/31200798 shutdown_connections 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 -- 172.21.15.28:0/31200798 wait complete. 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 Processor -- start 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.561+0000 7f39baea5640 1 -- start start 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39baea5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b41002f0 0x7f39b4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39baea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4100ca0 0x7f39b41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39baea5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39b4106f50 0x7f39b419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39baea5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4111fa0 con 0x7f39b4100ca0 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39baea5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4111e20 con 0x7f39b4106f50 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39baea5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4112120 con 0x7f39b41002f0 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4100ca0 0x7f39b41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4100ca0 0x7f39b41989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35464/0 (socket says 172.21.15.28:35464) 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 -- 172.21.15.28:0/3155681975 learned_addr learned my addr 172.21.15.28:0/3155681975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b8c41640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b41002f0 0x7f39b4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b9442640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39b4106f50 0x7f39b419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 -- 172.21.15.28:0/3155681975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b41002f0 msgr2=0x7f39b4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b41002f0 0x7f39b4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 -- 172.21.15.28:0/3155681975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39b4106f50 msgr2=0x7f39b419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39b4106f50 0x7f39b419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 -- 172.21.15.28:0/3155681975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b419d410 con 0x7f39b4100ca0 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b9442640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39b4106f50 0x7f39b419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.562+0000 7f39b3fff640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4100ca0 0x7f39b41989a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f399c00c970 tx=0x7f399c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.563+0000 7f39b1ffb640 1 -- 172.21.15.28:0/3155681975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c007bb0 con 0x7f39b4100ca0 2024-09-15T21:56:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.563+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b419d700 con 0x7f39b4100ca0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.563+0000 7f39b1ffb640 1 -- 172.21.15.28:0/3155681975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f399c007d50 con 0x7f39b4100ca0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.563+0000 7f39b1ffb640 1 -- 172.21.15.28:0/3155681975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c00f450 con 0x7f39b4100ca0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.563+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41a4fe0 con 0x7f39b4100ca0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.564+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b41006d0 con 0x7f39b4100ca0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.565+0000 7f39b1ffb640 1 -- 172.21.15.28:0/3155681975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f399c004f90 con 0x7f39b4100ca0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.566+0000 7f39b1ffb640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f39a00a32d0 0x7f39a00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.566+0000 7f39b8c41640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f39a00a32d0 0x7f39a00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.566+0000 7f39b1ffb640 1 -- 172.21.15.28:0/3155681975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f399c0cfb80 con 0x7f39b4100ca0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.566+0000 7f39b8c41640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f39a00a32d0 0x7f39a00a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f39a4005d00 tx=0x7f39a400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:33.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.570+0000 7f39b1ffb640 1 -- 172.21.15.28:0/3155681975 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f399c014030 con 0x7f39b4100ca0 2024-09-15T21:56:33.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.782+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f39b410a180 con 0x7f39b4100ca0 2024-09-15T21:56:33.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.783+0000 7f39b1ffb640 1 -- 172.21.15.28:0/3155681975 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f399c09c380 con 0x7f39b4100ca0 2024-09-15T21:56:33.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:33.785 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T21:56:33.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f39a00a32d0 msgr2=0x7f39a00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f39a00a32d0 0x7f39a00a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f39a4005d00 tx=0x7f39a400b410 comp rx=0 tx=0).stop 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4100ca0 msgr2=0x7f39b41989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4100ca0 0x7f39b41989a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f399c00c970 tx=0x7f399c00ce30 comp rx=0 tx=0).stop 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 shutdown_connections 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f39a00a32d0 0x7f39a00a5790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39b4106f50 0x7f39b419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4100ca0 0x7f39b41989a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 --2- 172.21.15.28:0/3155681975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39b41002f0 0x7f39b4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 >> 172.21.15.28:0/3155681975 conn(0x7f39b4077210 msgr2=0x7f39b4105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 shutdown_connections 2024-09-15T21:56:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:33.785+0000 7f39baea5640 1 -- 172.21.15.28:0/3155681975 wait complete. 2024-09-15T21:56:34.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:33 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3155681975' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T21:56:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:33 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3155681975' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T21:56:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:33 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3155681975' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T21:56:34.432 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-15T21:56:34.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:56:34.692 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:35.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:34 smithi028 ceph-mon[30969]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 -- 172.21.15.28:0/668407589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8106f10 msgr2=0x7fb2e810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 --2- 172.21.15.28:0/668407589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8106f10 0x7fb2e810ea90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009ca0 tx=0x7fb2e002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 -- 172.21.15.28:0/668407589 shutdown_connections 2024-09-15T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 --2- 172.21.15.28:0/668407589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8106f10 0x7fb2e810ea90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 --2- 172.21.15.28:0/668407589 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8102790 0x7fb2e81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 --2- 172.21.15.28:0/668407589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2e8101de0 0x7fb2e81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 -- 172.21.15.28:0/668407589 >> 172.21.15.28:0/668407589 conn(0x7fb2e80fd8c0 msgr2=0x7fb2e80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 -- 172.21.15.28:0/668407589 shutdown_connections 2024-09-15T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.162+0000 7fb2edc95640 1 -- 172.21.15.28:0/668407589 wait complete. 2024-09-15T21:56:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 Processor -- start 2024-09-15T21:56:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 -- start start 2024-09-15T21:56:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8101de0 0x7fb2e8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8102790 0x7fb2e8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2e8106f10 0x7fb2e819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e8111f70 con 0x7fb2e8102790 2024-09-15T21:56:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e8111df0 con 0x7fb2e8106f10 2024-09-15T21:56:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2edc95640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81120f0 con 0x7fb2e8101de0 2024-09-15T21:56:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.163+0000 7fb2dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8102790 0x7fb2e8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8102790 0x7fb2e8198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35476/0 (socket says 172.21.15.28:35476) 2024-09-15T21:56:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 -- 172.21.15.28:0/188757646 learned_addr learned my addr 172.21.15.28:0/188757646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2e77fe640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8101de0 0x7fb2e8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2e7fff640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2e8106f10 0x7fb2e819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 -- 172.21.15.28:0/188757646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8101de0 msgr2=0x7fb2e8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8101de0 0x7fb2e8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 -- 172.21.15.28:0/188757646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2e8106f10 msgr2=0x7fb2e819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2e8106f10 0x7fb2e819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 -- 172.21.15.28:0/188757646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e819d3c0 con 0x7fb2e8102790 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2e7fff640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2e8106f10 0x7fb2e819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2dffff640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8102790 0x7fb2e8198950 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000d770 tx=0x7fb2d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2e57fa640 1 -- 172.21.15.28:0/188757646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0017070 con 0x7fb2e8102790 2024-09-15T21:56:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2e57fa640 1 -- 172.21.15.28:0/188757646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d0005500 con 0x7fb2e8102790 2024-09-15T21:56:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e819d6b0 con 0x7fb2e8102790 2024-09-15T21:56:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2e57fa640 1 -- 172.21.15.28:0/188757646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0004e00 con 0x7fb2e8102790 2024-09-15T21:56:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.164+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e81a4f90 con 0x7fb2e8102790 2024-09-15T21:56:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.165+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e81021c0 con 0x7fb2e8102790 2024-09-15T21:56:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.167+0000 7fb2e57fa640 1 -- 172.21.15.28:0/188757646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb2d0004fa0 con 0x7fb2e8102790 2024-09-15T21:56:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.168+0000 7fb2e57fa640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2d40a32d0 0x7fb2d40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.168+0000 7fb2e57fa640 1 -- 172.21.15.28:0/188757646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fb2d00cfc00 con 0x7fb2e8102790 2024-09-15T21:56:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.168+0000 7fb2e77fe640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2d40a32d0 0x7fb2d40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.169+0000 7fb2e77fe640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2d40a32d0 0x7fb2d40a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800bbe0 tx=0x7fb2d800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.171+0000 7fb2e57fa640 1 -- 172.21.15.28:0/188757646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb2d009c460 con 0x7fb2e8102790 2024-09-15T21:56:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:34 smithi203 ceph-mon[35546]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:35.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:34 smithi174 ceph-mon[35244]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.384+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2e810a180 con 0x7fb2e8102790 2024-09-15T21:56:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.385+0000 7fb2e57fa640 1 -- 172.21.15.28:0/188757646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7fb2d009bd60 con 0x7fb2e8102790 2024-09-15T21:56:35.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:35.388 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"btime":"2024-09-15T21:55:38:559165+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:38.559150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:56:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-09-15T21:56:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2d40a32d0 msgr2=0x7fb2d40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2d40a32d0 0x7fb2d40a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800bbe0 tx=0x7fb2d800b390 comp rx=0 tx=0).stop 2024-09-15T21:56:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8102790 msgr2=0x7fb2e8198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8102790 0x7fb2e8198950 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000d770 tx=0x7fb2d000dc30 comp rx=0 tx=0).stop 2024-09-15T21:56:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 shutdown_connections 2024-09-15T21:56:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb2d40a32d0 0x7fb2d40a5790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2e8106f10 0x7fb2e819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e8102790 0x7fb2e8198950 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 --2- 172.21.15.28:0/188757646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8101de0 0x7fb2e8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.387+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 >> 172.21.15.28:0/188757646 conn(0x7fb2e80fd8c0 msgr2=0x7fb2e80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.388+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 shutdown_connections 2024-09-15T21:56:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:35.388+0000 7fb2edc95640 1 -- 172.21.15.28:0/188757646 wait complete. 2024-09-15T21:56:36.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:35 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/188757646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:36.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:56:36.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:35 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/188757646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:36.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:35 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/188757646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.804+0000 7f51318cb640 1 -- 172.21.15.28:0/3561615839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 msgr2=0x7f512c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.804+0000 7f51318cb640 1 --2- 172.21.15.28:0/3561615839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c101200 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f511c009ca0 tx=0x7f511c02a950 comp rx=0 tx=0).stop 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 -- 172.21.15.28:0/3561615839 shutdown_connections 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 --2- 172.21.15.28:0/3561615839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f512c102ef0 0x7f512c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 --2- 172.21.15.28:0/3561615839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f512c102570 0x7f512c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 --2- 172.21.15.28:0/3561615839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c101200 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 -- 172.21.15.28:0/3561615839 >> 172.21.15.28:0/3561615839 conn(0x7f512c0fc7d0 msgr2=0x7f512c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 -- 172.21.15.28:0/3561615839 shutdown_connections 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 -- 172.21.15.28:0/3561615839 wait complete. 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.805+0000 7f51318cb640 1 Processor -- start 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f51318cb640 1 -- start start 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f51318cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f51318cb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f512c102570 0x7f512c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f51318cb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f512c102ef0 0x7f512c1a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f51318cb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f512c110f80 con 0x7f512c100e20 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f51318cb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f512c110e00 con 0x7f512c102570 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f51318cb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f512c111100 con 0x7f512c102ef0 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c1992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35486/0 (socket says 172.21.15.28:35486) 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 -- 172.21.15.28:0/586888675 learned_addr learned my addr 172.21.15.28:0/586888675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512b7fe640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f512c102ef0 0x7f512c1a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512a7fc640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f512c102570 0x7f512c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 -- 172.21.15.28:0/586888675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f512c102ef0 msgr2=0x7f512c1a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f512c102ef0 0x7f512c1a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 -- 172.21.15.28:0/586888675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f512c102570 msgr2=0x7f512c1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f512c102570 0x7f512c1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512affd640 1 -- 172.21.15.28:0/586888675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c1a4870 con 0x7f512c100e20 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.806+0000 7f512a7fc640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f512c102570 0x7f512c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.807+0000 7f512affd640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c1992b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f511c005cc0 tx=0x7f511c02f650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.807+0000 7f51308c9640 1 -- 172.21.15.28:0/586888675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c034040 con 0x7f512c100e20 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.807+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512c1a4a40 con 0x7f512c100e20 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.807+0000 7f51308c9640 1 -- 172.21.15.28:0/586888675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f511c030920 con 0x7f512c100e20 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.807+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f512c1a4ef0 con 0x7f512c100e20 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.807+0000 7f51308c9640 1 -- 172.21.15.28:0/586888675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c004830 con 0x7f512c100e20 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.808+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f512c101200 con 0x7f512c100e20 2024-09-15T21:56:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.810+0000 7f51308c9640 1 -- 172.21.15.28:0/586888675 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f511c037020 con 0x7f512c100e20 2024-09-15T21:56:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.811+0000 7f51308c9640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f51140a32d0 0x7f51140a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.811+0000 7f51308c9640 1 -- 172.21.15.28:0/586888675 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f511c0f0f70 con 0x7f512c100e20 2024-09-15T21:56:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.811+0000 7f512a7fc640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f51140a32d0 0x7f51140a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.811+0000 7f512a7fc640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f51140a32d0 0x7f51140a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f512c06aac0 tx=0x7f512000a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:36.814+0000 7f51308c9640 1 -- 172.21.15.28:0/586888675 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f511c0811b0 con 0x7f512c100e20 2024-09-15T21:56:37.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.025+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f512c1a5180 con 0x7f512c100e20 2024-09-15T21:56:37.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:36 smithi028 ceph-mon[30969]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:37.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.026+0000 7f51308c9640 1 -- 172.21.15.28:0/586888675 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f511c0bd880 con 0x7f512c100e20 2024-09-15T21:56:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:37.029 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"btime":"2024-09-15T21:55:38:559165+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:38.559150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:56:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-09-15T21:56:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.028+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f51140a32d0 msgr2=0x7f51140a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.028+0000 7f51318cb640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f51140a32d0 0x7f51140a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f512c06aac0 tx=0x7f512000a5c0 comp rx=0 tx=0).stop 2024-09-15T21:56:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.028+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 msgr2=0x7f512c1992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.028+0000 7f51318cb640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c1992b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f511c005cc0 tx=0x7f511c02f650 comp rx=0 tx=0).stop 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.028+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 shutdown_connections 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.029+0000 7f51318cb640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f51140a32d0 0x7f51140a5790 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.029+0000 7f51318cb640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f512c102ef0 0x7f512c1a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.029+0000 7f51318cb640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f512c102570 0x7f512c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.029+0000 7f51318cb640 1 --2- 172.21.15.28:0/586888675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512c100e20 0x7f512c1992b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.029+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 >> 172.21.15.28:0/586888675 conn(0x7f512c0fc7d0 msgr2=0x7f512c106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.029+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 shutdown_connections 2024-09-15T21:56:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:37.029+0000 7f51318cb640 1 -- 172.21.15.28:0/586888675 wait complete. 2024-09-15T21:56:37.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:36 smithi203 ceph-mon[35546]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:37.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:36 smithi174 ceph-mon[35244]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:37.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:56:37.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:37 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/586888675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:37 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/586888675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:37 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/586888675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:38.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 -- 172.21.15.28:0/3990976715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 msgr2=0x7fd31410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 --2- 172.21.15.28:0/3990976715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31410ea90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009900 tx=0x7fd30c02a410 comp rx=0 tx=0).stop 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 -- 172.21.15.28:0/3990976715 shutdown_connections 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 --2- 172.21.15.28:0/3990976715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31410ea90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 --2- 172.21.15.28:0/3990976715 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd314102790 0x7fd3141067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 --2- 172.21.15.28:0/3990976715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd314101de0 0x7fd3141021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 -- 172.21.15.28:0/3990976715 >> 172.21.15.28:0/3990976715 conn(0x7fd3140fd8c0 msgr2=0x7fd3140ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 -- 172.21.15.28:0/3990976715 shutdown_connections 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.487+0000 7fd319d57640 1 -- 172.21.15.28:0/3990976715 wait complete. 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 Processor -- start 2024-09-15T21:56:38.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 -- start start 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd314101de0 0x7fd314198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd314102790 0x7fd314198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd314111f90 con 0x7fd314106f10 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd314111e10 con 0x7fd314101de0 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd319d57640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd314112110 con 0x7fd314102790 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31419ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35506/0 (socket says 172.21.15.28:35506) 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd313fff640 1 -- 172.21.15.28:0/2837283418 learned_addr learned my addr 172.21.15.28:0/2837283418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:38.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd3137fe640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd314101de0 0x7fd314198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd312ffd640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd314102790 0x7fd314198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd313fff640 1 -- 172.21.15.28:0/2837283418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd314102790 msgr2=0x7fd314198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd313fff640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd314102790 0x7fd314198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd313fff640 1 -- 172.21.15.28:0/2837283418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd314101de0 msgr2=0x7fd314198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.488+0000 7fd313fff640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd314101de0 0x7fd314198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.489+0000 7fd313fff640 1 -- 172.21.15.28:0/2837283418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31419d500 con 0x7fd314106f10 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.489+0000 7fd313fff640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31419ce20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd30c005ab0 tx=0x7fd30c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.489+0000 7fd310ff9640 1 -- 172.21.15.28:0/2837283418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c03f030 con 0x7fd314106f10 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.489+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31419d790 con 0x7fd314106f10 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.489+0000 7fd310ff9640 1 -- 172.21.15.28:0/2837283418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd30c003ad0 con 0x7fd314106f10 2024-09-15T21:56:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.489+0000 7fd310ff9640 1 -- 172.21.15.28:0/2837283418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c035070 con 0x7fd314106f10 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.489+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3141a50d0 con 0x7fd314106f10 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.490+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3141032c0 con 0x7fd314106f10 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.491+0000 7fd310ff9640 1 -- 172.21.15.28:0/2837283418 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd30c042030 con 0x7fd314106f10 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.492+0000 7fd310ff9640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2fc0a32d0 0x7fd2fc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.492+0000 7fd3137fe640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2fc0a32d0 0x7fd2fc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.492+0000 7fd310ff9640 1 -- 172.21.15.28:0/2837283418 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fd30c0f0a90 con 0x7fd314106f10 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.493+0000 7fd3137fe640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2fc0a32d0 0x7fd2fc0a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd304005fd0 tx=0x7fd304009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.495+0000 7fd310ff9640 1 -- 172.21.15.28:0/2837283418 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd30c0bd2f0 con 0x7fd314106f10 2024-09-15T21:56:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.673+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd31410a150 con 0x7fd314106f10 2024-09-15T21:56:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.674+0000 7fd310ff9640 1 -- 172.21.15.28:0/2837283418 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24715 (secure 0 0 0) 0x7fd30c0bcbf0 con 0x7fd314106f10 2024-09-15T21:56:38.676 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:38.676 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:55:58.788704+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:56:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.676+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2fc0a32d0 msgr2=0x7fd2fc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.676+0000 7fd319d57640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2fc0a32d0 0x7fd2fc0a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd304005fd0 tx=0x7fd304009500 comp rx=0 tx=0).stop 2024-09-15T21:56:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.676+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 msgr2=0x7fd31419ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.676+0000 7fd319d57640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31419ce20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd30c005ab0 tx=0x7fd30c004270 comp rx=0 tx=0).stop 2024-09-15T21:56:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.676+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 shutdown_connections 2024-09-15T21:56:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.676+0000 7fd319d57640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fd2fc0a32d0 0x7fd2fc0a5790 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.677+0000 7fd319d57640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106f10 0x7fd31419ce20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.677+0000 7fd319d57640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd314102790 0x7fd314198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.677+0000 7fd319d57640 1 --2- 172.21.15.28:0/2837283418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd314101de0 0x7fd314198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.677+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 >> 172.21.15.28:0/2837283418 conn(0x7fd3140fd8c0 msgr2=0x7fd3140fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.677+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 shutdown_connections 2024-09-15T21:56:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:38.677+0000 7fd319d57640 1 -- 172.21.15.28:0/2837283418 wait complete. 2024-09-15T21:56:38.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:38 smithi028 ceph-mon[30969]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:38.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:38 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2837283418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:39.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:38 smithi203 ceph-mon[35546]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:39.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:38 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2837283418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:39.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:38 smithi174 ceph-mon[35244]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:39.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:38 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2837283418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:56:39.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:56:39.665 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 -- 172.21.15.28:0/4247018017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8112b90 msgr2=0x7f42b8114f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 --2- 172.21.15.28:0/4247018017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8112b90 0x7f42b8114f80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009ca0 tx=0x7f42b002a950 comp rx=0 tx=0).stop 2024-09-15T21:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 -- 172.21.15.28:0/4247018017 shutdown_connections 2024-09-15T21:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 --2- 172.21.15.28:0/4247018017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8112b90 0x7f42b8114f80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 --2- 172.21.15.28:0/4247018017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42b8074eb0 0x7f42b8112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 --2- 172.21.15.28:0/4247018017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42b8074590 0x7f42b8074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 -- 172.21.15.28:0/4247018017 >> 172.21.15.28:0/4247018017 conn(0x7f42b80fd8c0 msgr2=0x7f42b80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 -- 172.21.15.28:0/4247018017 shutdown_connections 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 -- 172.21.15.28:0/4247018017 wait complete. 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 Processor -- start 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 -- start start 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8074590 0x7f42b81a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42b8074eb0 0x7f42b81a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.158+0000 7f42bd49d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42b8112b90 0x7f42b819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42bd49d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42b8118550 con 0x7f42b8074590 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42bd49d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81183d0 con 0x7f42b8074eb0 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42bd49d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42b81186d0 con 0x7f42b8112b90 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8074590 0x7f42b81a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8074590 0x7f42b81a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35528/0 (socket says 172.21.15.28:35528) 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 -- 172.21.15.28:0/29444039 learned_addr learned my addr 172.21.15.28:0/29444039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42affff640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42b8074eb0 0x7f42b81a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b77fe640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42b8112b90 0x7f42b819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 -- 172.21.15.28:0/29444039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42b8112b90 msgr2=0x7f42b819e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42b8112b90 0x7f42b819e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 -- 172.21.15.28:0/29444039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42b8074eb0 msgr2=0x7f42b81a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42b8074eb0 0x7f42b81a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 -- 172.21.15.28:0/29444039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b819efa0 con 0x7f42b8074590 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42affff640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42b8074eb0 0x7f42b81a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b6ffd640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8074590 0x7f42b81a46b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f42a800e9f0 tx=0x7f42a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b4ff9640 1 -- 172.21.15.28:0/29444039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a800cd80 con 0x7f42b8074590 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b819f290 con 0x7f42b8074590 2024-09-15T21:56:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b4ff9640 1 -- 172.21.15.28:0/29444039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a8010910 con 0x7f42b8074590 2024-09-15T21:56:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42b4ff9640 1 -- 172.21.15.28:0/29444039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a80188f0 con 0x7f42b8074590 2024-09-15T21:56:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.159+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b81ab580 con 0x7f42b8074590 2024-09-15T21:56:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.160+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42b8074970 con 0x7f42b8074590 2024-09-15T21:56:40.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.161+0000 7f42b4ff9640 1 -- 172.21.15.28:0/29444039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f42a8010ac0 con 0x7f42b8074590 2024-09-15T21:56:40.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.162+0000 7f42b4ff9640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42a40a32d0 0x7f42a40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:40.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.162+0000 7f42b4ff9640 1 -- 172.21.15.28:0/29444039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f42a80d02f0 con 0x7f42b8074590 2024-09-15T21:56:40.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.162+0000 7f42affff640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42a40a32d0 0x7f42a40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:40.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.163+0000 7f42affff640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42a40a32d0 0x7f42a40a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42a0005fd0 tx=0x7f42a0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:40.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.166+0000 7f42b4ff9640 1 -- 172.21.15.28:0/29444039 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f42a809cb50 con 0x7f42b8074590 2024-09-15T21:56:40.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.377+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42b819fda0 con 0x7f42b8074590 2024-09-15T21:56:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.378+0000 7f42b4ff9640 1 -- 172.21.15.28:0/29444039 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f42a809c450 con 0x7f42b8074590 2024-09-15T21:56:40.380 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:40.380 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"btime":"2024-09-15T21:55:38:559165+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:38.559150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:56:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42a40a32d0 msgr2=0x7f42a40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42a40a32d0 0x7f42a40a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42a0005fd0 tx=0x7f42a0005850 comp rx=0 tx=0).stop 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8074590 msgr2=0x7f42b81a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8074590 0x7f42b81a46b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f42a800e9f0 tx=0x7f42a800eeb0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 shutdown_connections 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f42a40a32d0 0x7f42a40a5790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42b8112b90 0x7f42b819e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42b8074eb0 0x7f42b81a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 --2- 172.21.15.28:0/29444039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42b8074590 0x7f42b81a46b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 >> 172.21.15.28:0/29444039 conn(0x7f42b80fd8c0 msgr2=0x7f42b80ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 shutdown_connections 2024-09-15T21:56:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:40.381+0000 7f42bd49d640 1 -- 172.21.15.28:0/29444039 wait complete. 2024-09-15T21:56:41.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T21:54:59.048835+0000', 'modified': '2024-09-15T21:55:38.559150+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24485, 'mds_1': 24481, 'mds_2': 24460, 'mds_3': 24458, 'mds_4': 24454}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/3176985638', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 3176985638}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 3176985638}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2641693590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2641693590}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2641693590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/2800102614', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 2800102614}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 2800102614}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6833/1988552520', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 1988552520}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 1988552520}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/280309786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 280309786}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 280309786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.203:6835/1524895518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 1524895518}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 1524895518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.174:6837/3327498469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3327498469}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3327498469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.203:6837/803844131', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 803844131}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 803844131}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.203:6839/3242294678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 3242294678}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 3242294678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.174:6839/3735159621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3735159621}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3735159621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24485, 24481, 24460, 24458, 24454]} 2024-09-15T21:56:41.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-15T21:56:41.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-15T21:56:41.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:40 smithi028 ceph-mon[30969]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:41.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:40 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/29444039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:41.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:40 smithi203 ceph-mon[35546]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:41.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:40 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/29444039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:41.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:40 smithi174 ceph-mon[35244]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:41.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:40 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/29444039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:56:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 -- 172.21.15.28:0/3540911558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260102d40 msgr2=0x7f0260103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 --2- 172.21.15.28:0/3540911558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260102d40 0x7f0260103120 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0250009900 tx=0x7f025002a480 comp rx=0 tx=0).stop 2024-09-15T21:56:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 -- 172.21.15.28:0/3540911558 shutdown_connections 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 --2- 172.21.15.28:0/3540911558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f026010c170 0x7f026010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 --2- 172.21.15.28:0/3540911558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0260103660 0x7f026010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 --2- 172.21.15.28:0/3540911558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260102d40 0x7f0260103120 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 -- 172.21.15.28:0/3540911558 >> 172.21.15.28:0/3540911558 conn(0x7f02600fc7a0 msgr2=0x7f02600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 -- 172.21.15.28:0/3540911558 shutdown_connections 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.858+0000 7f0265bd3640 1 -- 172.21.15.28:0/3540911558 wait complete. 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.859+0000 7f0265bd3640 1 Processor -- start 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.859+0000 7f0265bd3640 1 -- start start 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f0265bd3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0260102d40 0x7f02601984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f0265bd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260103660 0x7f02601989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f0265bd3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026010c170 0x7f026019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f0265bd3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0260111fb0 con 0x7f0260103660 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f0265bd3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0260111e30 con 0x7f0260102d40 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f0265bd3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0260112130 con 0x7f026010c170 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260103660 0x7f02601989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260103660 0x7f02601989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41962/0 (socket says 172.21.15.28:41962) 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 -- 172.21.15.28:0/815809964 learned_addr learned my addr 172.21.15.28:0/815809964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025f7fe640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0260102d40 0x7f02601984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025ffff640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026010c170 0x7f026019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 -- 172.21.15.28:0/815809964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026010c170 msgr2=0x7f026019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026010c170 0x7f026019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 -- 172.21.15.28:0/815809964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0260102d40 msgr2=0x7f02601984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0260102d40 0x7f02601984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 -- 172.21.15.28:0/815809964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026019d460 con 0x7f0260103660 2024-09-15T21:56:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025effd640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260103660 0x7f02601989f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f025400b3f0 tx=0x7f025400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.860+0000 7f025cff9640 1 -- 172.21.15.28:0/815809964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254015020 con 0x7f0260103660 2024-09-15T21:56:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.861+0000 7f025cff9640 1 -- 172.21.15.28:0/815809964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0254011d10 con 0x7f0260103660 2024-09-15T21:56:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.861+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026019d750 con 0x7f0260103660 2024-09-15T21:56:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.861+0000 7f025cff9640 1 -- 172.21.15.28:0/815809964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025400f2e0 con 0x7f0260103660 2024-09-15T21:56:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.861+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02601a5030 con 0x7f0260103660 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f025cff9640 1 -- 172.21.15.28:0/815809964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0254018020 con 0x7f0260103660 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 Processor -- start 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 -- start start 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f617db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6182f0 0x55671f622670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f622bb0 0x55671f624fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55671f57e090 con 0x55671f6179d0 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55671f525a70 con 0x55671f622bb0 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.863+0000 7f0267fac740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55671f57a050 con 0x55671f6182f0 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f617db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f617db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41964/0 (socket says 172.21.15.28:41964) 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025ffff640 1 -- 172.21.15.28:0/28481682 learned_addr learned my addr 172.21.15.28:0/28481682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025f7fe640 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6182f0 0x55671f622670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025cff9640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f02480a3350 0x7f02480a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025effd640 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f622bb0 0x55671f624fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025f7fe640 1 -- 172.21.15.28:0/28481682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f622bb0 msgr2=0x55671f624fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025f7fe640 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f622bb0 0x55671f624fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025f7fe640 1 -- 172.21.15.28:0/28481682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 msgr2=0x55671f617db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025f7fe640 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f617db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025f7fe640 1 -- 172.21.15.28:0/28481682 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55671f532130 con 0x55671f6182f0 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025effd640 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f622bb0 0x55671f624fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025ffff640 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f617db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:56:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025ffff640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f02480a3350 0x7f02480a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025cff9640 1 -- 172.21.15.28:0/815809964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f02540cf500 con 0x7f0260103660 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.864+0000 7f025f7fe640 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6182f0 0x55671f622670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0250005940 tx=0x7f0250031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfb8167a9eba4267 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.865+0000 7f025ffff640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f02480a3350 0x7f02480a5810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0258005c40 tx=0x7f0258004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.865+0000 7f0245ffb640 1 -- 172.21.15.28:0/28481682 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250035040 con 0x55671f6182f0 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.865+0000 7f0245ffb640 1 -- 172.21.15.28:0/28481682 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0250005020 con 0x55671f6182f0 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.865+0000 7f0245ffb640 1 -- 172.21.15.28:0/28481682 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025003fa40 con 0x55671f6182f0 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 -- 172.21.15.28:0/28481682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6182f0 msgr2=0x55671f622670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6182f0 0x55671f622670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0250005940 tx=0x7f0250031630 comp rx=0 tx=0).stop 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 -- 172.21.15.28:0/28481682 shutdown_connections 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f622bb0 0x55671f624fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6182f0 0x55671f622670 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 --2- 172.21.15.28:0/28481682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f617db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 -- 172.21.15.28:0/28481682 >> 172.21.15.28:0/28481682 conn(0x55671f613f60 msgr2=0x55671f614340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 -- 172.21.15.28:0/28481682 shutdown_connections 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 -- 172.21.15.28:0/28481682 wait complete. 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 -- start start 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 Processor -- start 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6179d0 0x55671f622450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6182f0 0x55671f61f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.866+0000 7f0267fac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55671f625780 con 0x55671f6182f0 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f0267fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55671f538f20 con 0x55671f6179d0 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f0267fac740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55671f625900 con 0x55671f622bb0 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f0267fac740 10 client.0 _ll_register_callbacks cb 0x55671f502030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6182f0 0x55671f61f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6182f0 0x55671f61f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41980/0 (socket says 172.21.15.28:41980) 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 -- 172.21.15.28:0/1511438902 learned_addr learned my addr 172.21.15.28:0/1511438902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025ffff640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025f7fe640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6179d0 0x55671f622450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 -- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 msgr2=0x55671f61fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 -- 172.21.15.28:0/1511438902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6179d0 msgr2=0x55671f622450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6179d0 0x55671f622450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 -- 172.21.15.28:0/1511438902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55671f2ef2b0 con 0x55671f6182f0 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025f7fe640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6179d0 0x55671f622450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f025effd640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6182f0 0x55671f61f7a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0254097b70 tx=0x7f025409b310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63219cd35af8fbeb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f023effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f023effd640 1 -- 172.21.15.28:0/1511438902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025409be10 con 0x55671f6182f0 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f023effd640 1 -- 172.21.15.28:0/1511438902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f0254097d80 con 0x55671f6182f0 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.867+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0250005020 con 0x55671f6182f0 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f023effd640 1 -- 172.21.15.28:0/1511438902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f025409b710 con 0x55671f6182f0 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 10 client.15078 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55671f6310a0 con 0x55671f6182f0 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f023effd640 1 -- 172.21.15.28:0/1511438902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0250005020 con 0x55671f6182f0 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f023effd640 1 -- 172.21.15.28:0/1511438902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f025409ab60 con 0x55671f6182f0 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 10 client.15078 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 10 client.15078 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 20 client.15078 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f633730 0x55671f634ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 4 client.15078 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:56:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55671f58b600 con 0x55671f633730 2024-09-15T21:56:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.868+0000 7f025f7fe640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f633730 0x55671f634ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.869+0000 7f025f7fe640 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f633730 0x55671f634ba0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37ce929a56bc2cb4 server_cookie=7f7f9f90a98d1df1 in_seq=0 out_seq=0 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.869+0000 7f023effd640 10 client.15078 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.869+0000 7f023effd640 1 -- 172.21.15.28:0/1511438902 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55671f58b600 con 0x55671f633730 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.869+0000 7f023effd640 10 client.15078 handle_command_reply: tid=0 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 client.15078 shutdown 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6182f0 msgr2=0x55671f61f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6182f0 0x55671f61f7a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0254097b70 tx=0x7f025409b310 comp rx=0 tx=0).stop 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 shutdown_connections 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f633730 0x55671f634ba0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6182f0 0x55671f61f7a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 --2- 172.21.15.28:0/1511438902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6179d0 0x55671f622450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f023effd640 0 client.15078 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:56:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 >> 172.21.15.28:0/1511438902 conn(0x55671f613f60 msgr2=0x55671f619c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 shutdown_connections 2024-09-15T21:56:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 1 -- 172.21.15.28:0/1511438902 wait complete. 2024-09-15T21:56:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.870+0000 7f0267fac740 20 client.15078 trim_cache size 0 max 16384 2024-09-15T21:56:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 Processor -- start 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 -- start start 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f61f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f61ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55671f632ee0 con 0x55671f6179d0 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55671f6335b0 con 0x55671f6182f0 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f0267fac740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55671f633c80 con 0x55671f622bb0 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f025f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f61f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.888+0000 7f025f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f61f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41986/0 (socket says 172.21.15.28:41986) 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025f7fe640 1 -- 172.21.15.28:0/2030108090 learned_addr learned my addr 172.21.15.28:0/2030108090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025ffff640 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025effd640 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f61ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025f7fe640 1 -- 172.21.15.28:0/2030108090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 msgr2=0x55671f61cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025f7fe640 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025f7fe640 1 -- 172.21.15.28:0/2030108090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 msgr2=0x55671f61ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025f7fe640 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f61ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025f7fe640 1 -- 172.21.15.28:0/2030108090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55671f6227c0 con 0x55671f6179d0 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f025f7fe640 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f61f060 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f02500096c0 tx=0x7f02500058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea19143000982823 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f02457fa640 1 -- 172.21.15.28:0/2030108090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250035040 con 0x55671f6179d0 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f02457fa640 1 -- 172.21.15.28:0/2030108090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02500055e0 con 0x55671f6179d0 2024-09-15T21:56:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.889+0000 7f02457fa640 1 -- 172.21.15.28:0/2030108090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02500035f0 con 0x55671f6179d0 2024-09-15T21:56:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 -- 172.21.15.28:0/2030108090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 msgr2=0x55671f61f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f61f060 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f02500096c0 tx=0x7f02500058d0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 -- 172.21.15.28:0/2030108090 shutdown_connections 2024-09-15T21:56:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f622bb0 0x55671f61cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f61ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 --2- 172.21.15.28:0/2030108090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f6179d0 0x55671f61f060 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 -- 172.21.15.28:0/2030108090 >> 172.21.15.28:0/2030108090 conn(0x55671f613f60 msgr2=0x55671f61f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 -- 172.21.15.28:0/2030108090 shutdown_connections 2024-09-15T21:56:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 -- 172.21.15.28:0/2030108090 wait complete. 2024-09-15T21:56:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.890+0000 7f0267fac740 1 -- start start 2024-09-15T21:56:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 1 Processor -- start 2024-09-15T21:56:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6179d0 0x55671f638310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f638850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f622bb0 0x55671f69be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55671f634a20 con 0x55671f622bb0 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55671f634350 con 0x55671f6182f0 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55671f6350f0 con 0x55671f6179d0 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f0267fac740 10 client.0 _ll_register_callbacks cb 0x7f025400b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f622bb0 0x55671f69be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f622bb0 0x55671f69be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41998/0 (socket says 172.21.15.28:41998) 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 -- 172.21.15.28:0/2364623627 learned_addr learned my addr 172.21.15.28:0/2364623627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025f7fe640 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6179d0 0x55671f638310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025effd640 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f638850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 -- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6179d0 msgr2=0x55671f638310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6179d0 0x55671f638310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 -- 172.21.15.28:0/2364623627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 msgr2=0x55671f638850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f638850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.891+0000 7f025ffff640 1 -- 172.21.15.28:0/2364623627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55671f69c370 con 0x55671f622bb0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f025ffff640 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f622bb0 0x55671f69be30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0258010eb0 tx=0x7f0258004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a01c6c08d38a388 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f023effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f023effd640 1 -- 172.21.15.28:0/2364623627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0258012cd0 con 0x55671f622bb0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02500055e0 con 0x55671f622bb0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f023effd640 1 -- 172.21.15.28:0/2364623627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f025802a420 con 0x55671f622bb0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f023effd640 1 -- 172.21.15.28:0/2364623627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f025802a620 con 0x55671f622bb0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f0267fac740 10 client.15084 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55671f69d720 con 0x55671f622bb0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f023effd640 1 -- 172.21.15.28:0/2364623627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02500055e0 con 0x55671f622bb0 2024-09-15T21:56:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f023effd640 1 -- 172.21.15.28:0/2364623627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f025800f040 con 0x55671f622bb0 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.892+0000 7f0267fac740 10 client.15084 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f0267fac740 10 client.15084 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f0267fac740 20 client.15084 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f0267fac740 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f6a22e0 0x55671f6a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f0267fac740 4 client.15084 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "status"}) -- 0x55671f585b10 con 0x55671f6a22e0 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f025f7fe640 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f6a22e0 0x55671f6a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f023effd640 10 client.15084 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f025f7fe640 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f6a22e0 0x55671f6a0ff0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85505e812999a894 server_cookie=952c8a53e9ab5cd7 in_seq=0 out_seq=0 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f023effd640 1 -- 172.21.15.28:0/2364623627 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55671f585b10 con 0x55671f6a22e0 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f023effd640 10 client.15084 handle_command_reply: tid=0 2024-09-15T21:56:41.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.893+0000 7f0267fac740 1 client.15084 shutdown 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f622bb0 msgr2=0x55671f69be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f622bb0 0x55671f69be30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0258010eb0 tx=0x7f0258004ab0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 shutdown_connections 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55671f6a22e0 0x55671f6a0ff0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55671f622bb0 0x55671f69be30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55671f6182f0 0x55671f638850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 --2- 172.21.15.28:0/2364623627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55671f6179d0 0x55671f638310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f023effd640 0 client.15084 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 >> 172.21.15.28:0/2364623627 conn(0x55671f613f60 msgr2=0x55671f61e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 shutdown_connections 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 1 -- 172.21.15.28:0/2364623627 wait complete. 2024-09-15T21:56:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0267fac740 20 client.15084 trim_cache size 0 max 16384 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24454, 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 64.342105584999999, 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:56:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-09-15T21:56:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:56:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 130.70109669000001 2024-09-15T21:56:41.903 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:56:41.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f02480a3350 msgr2=0x7f02480a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0265bd3640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f02480a3350 0x7f02480a5810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0258005c40 tx=0x7f0258004a40 comp rx=0 tx=0).stop 2024-09-15T21:56:41.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260103660 msgr2=0x7f02601989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:41.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.894+0000 7f0265bd3640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260103660 0x7f02601989f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f025400b3f0 tx=0x7f025400b8b0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 shutdown_connections 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f02480a3350 0x7f02480a5810 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026010c170 0x7f026019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0260103660 0x7f02601989f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 --2- 172.21.15.28:0/815809964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0260102d40 0x7f02601984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 >> 172.21.15.28:0/815809964 conn(0x7f02600fc7a0 msgr2=0x7f02600fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 shutdown_connections 2024-09-15T21:56:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:41.895+0000 7f0265bd3640 1 -- 172.21.15.28:0/815809964 wait complete. 2024-09-15T21:56:42.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T21:56:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:43.163 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:42 smithi028 ceph-mon[30969]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:43.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:42 smithi203 ceph-mon[35546]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:43.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:42 smithi174 ceph-mon[35244]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:43.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 -- 172.21.15.28:0/2600529487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220100e20 msgr2=0x7f5220101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 --2- 172.21.15.28:0/2600529487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220100e20 0x7f5220101200 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5210009900 tx=0x7f521002a480 comp rx=0 tx=0).stop 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 -- 172.21.15.28:0/2600529487 shutdown_connections 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 --2- 172.21.15.28:0/2600529487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220102ef0 0x7f522010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 --2- 172.21.15.28:0/2600529487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5220102570 0x7f52201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 --2- 172.21.15.28:0/2600529487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220100e20 0x7f5220101200 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 -- 172.21.15.28:0/2600529487 >> 172.21.15.28:0/2600529487 conn(0x7f52200fc7d0 msgr2=0x7f52200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 -- 172.21.15.28:0/2600529487 shutdown_connections 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 -- 172.21.15.28:0/2600529487 wait complete. 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.554+0000 7f5225d53640 1 Processor -- start 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f5225d53640 1 -- start start 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f5225d53640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5220100e20 0x7f5220198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f5225d53640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220102570 0x7f5220198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f5225d53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220102ef0 0x7f522019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f5225d53640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5220112060 con 0x7f5220102ef0 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f5225d53640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5220111ee0 con 0x7f5220100e20 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f5225d53640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52201121e0 con 0x7f5220102570 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220102ef0 0x7f522019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220102ef0 0x7f522019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42024/0 (socket says 172.21.15.28:42024) 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 -- 172.21.15.28:0/3367768994 learned_addr learned my addr 172.21.15.28:0/3367768994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521f7fe640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5220100e20 0x7f5220198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521effd640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220102570 0x7f5220198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 -- 172.21.15.28:0/3367768994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220102570 msgr2=0x7f5220198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220102570 0x7f5220198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 -- 172.21.15.28:0/3367768994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5220100e20 msgr2=0x7f5220198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5220100e20 0x7f5220198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.555+0000 7f521ffff640 1 -- 172.21.15.28:0/3367768994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522019d530 con 0x7f5220102ef0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.556+0000 7f521ffff640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220102ef0 0x7f522019ce50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f521800eb10 tx=0x7f521800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.556+0000 7f521cff9640 1 -- 172.21.15.28:0/3367768994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218004b20 con 0x7f5220102ef0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.556+0000 7f521cff9640 1 -- 172.21.15.28:0/3367768994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5218004cc0 con 0x7f5220102ef0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.556+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522019d820 con 0x7f5220102ef0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.556+0000 7f521cff9640 1 -- 172.21.15.28:0/3367768994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218010760 con 0x7f5220102ef0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.556+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52201a5100 con 0x7f5220102ef0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.558+0000 7f521cff9640 1 -- 172.21.15.28:0/3367768994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5218005020 con 0x7f5220102ef0 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 Processor -- start 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 -- start start 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e767db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e772670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e774fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d99e6ce090 con 0x55d99e772bb0 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d99e675a70 con 0x55d99e7679d0 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f522812c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d99e6ca050 con 0x55d99e7682f0 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e774fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e774fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42028/0 (socket says 172.21.15.28:42028) 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 -- 172.21.15.28:0/815117068 learned_addr learned my addr 172.21.15.28:0/815117068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521f7fe640 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e772670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e767db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521cff9640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f52080a3350 0x7f52080a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 -- 172.21.15.28:0/815117068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 msgr2=0x55d99e772670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e772670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 -- 172.21.15.28:0/815117068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 msgr2=0x55d99e767db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.559+0000 7f521effd640 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e767db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.560+0000 7f521ffff640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f52080a3350 0x7f52080a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.560+0000 7f521effd640 1 -- 172.21.15.28:0/815117068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d99e682130 con 0x55d99e772bb0 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.560+0000 7f521cff9640 1 -- 172.21.15.28:0/3367768994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f52180d0340 con 0x7f5220102ef0 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.560+0000 7f521effd640 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e774fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f52140149c0 tx=0x7f521400c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653a1a15fe267c70 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.560+0000 7f5205ffb640 1 -- 172.21.15.28:0/815117068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52140142d0 con 0x55d99e772bb0 2024-09-15T21:56:43.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.560+0000 7f5205ffb640 1 -- 172.21.15.28:0/815117068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521400fc40 con 0x55d99e772bb0 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.560+0000 7f5205ffb640 1 -- 172.21.15.28:0/815117068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214012af0 con 0x55d99e772bb0 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 -- 172.21.15.28:0/815117068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 msgr2=0x55d99e774fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f521ffff640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f52080a3350 0x7f52080a5810 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5218098900 tx=0x7f521800efd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e774fa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f52140149c0 tx=0x7f521400c4d0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 -- 172.21.15.28:0/815117068 shutdown_connections 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e774fa0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e772670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 --2- 172.21.15.28:0/815117068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e767db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 -- 172.21.15.28:0/815117068 >> 172.21.15.28:0/815117068 conn(0x55d99e763f60 msgr2=0x55d99e764340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 -- 172.21.15.28:0/815117068 shutdown_connections 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 -- 172.21.15.28:0/815117068 wait complete. 2024-09-15T21:56:43.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 -- start start 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.561+0000 7f522812c740 1 Processor -- start 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f522812c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e772450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f522812c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7682f0 0x55d99e76f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f522812c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e772bb0 0x55d99e76fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f522812c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d99e775780 con 0x55d99e7682f0 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f522812c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d99e688f20 con 0x55d99e7679d0 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f522812c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d99e775900 con 0x55d99e772bb0 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f522812c740 10 client.0 _ll_register_callbacks cb 0x55d99e652030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7682f0 0x55d99e76f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7682f0 0x55d99e76f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42032/0 (socket says 172.21.15.28:42032) 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 -- 172.21.15.28:0/556582406 learned_addr learned my addr 172.21.15.28:0/556582406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521effd640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e772450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521effd640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e772bb0 0x55d99e76fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 -- 172.21.15.28:0/556582406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e772bb0 msgr2=0x55d99e76fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e772bb0 0x55d99e76fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 -- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 msgr2=0x55d99e772450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e772450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521f7fe640 1 -- 172.21.15.28:0/556582406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d99e43f2b0 con 0x55d99e7682f0 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.562+0000 7f521effd640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e772450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f521f7fe640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7682f0 0x55d99e76f7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5210000c00 tx=0x7f52100043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4b953c7b39d6e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f51feffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f51feffd640 1 -- 172.21.15.28:0/556582406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210035040 con 0x55d99e7682f0 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f51feffd640 1 -- 172.21.15.28:0/556582406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f5210003a00 con 0x55d99e7682f0 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f521400fc40 con 0x55d99e7682f0 2024-09-15T21:56:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f51feffd640 1 -- 172.21.15.28:0/556582406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f521004e050 con 0x55d99e7682f0 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f522812c740 10 client.15096 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d99e7810a0 con 0x55d99e7682f0 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f51feffd640 1 -- 172.21.15.28:0/556582406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521400fc40 con 0x55d99e7682f0 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f51feffd640 1 -- 172.21.15.28:0/556582406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f521003c440 con 0x55d99e7682f0 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f522812c740 10 client.15096 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f522812c740 10 client.15096 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.563+0000 7f522812c740 20 client.15096 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.564+0000 7f522812c740 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e783710 0x55d99e784b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.564+0000 7f522812c740 4 client.15096 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.564+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d99e6db600 con 0x55d99e783710 2024-09-15T21:56:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.564+0000 7f521effd640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e783710 0x55d99e784b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.564+0000 7f521effd640 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e783710 0x55d99e784b80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6e042f1cf1532e7a server_cookie=ac66522fb1a4ce42 in_seq=0 out_seq=0 2024-09-15T21:56:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.564+0000 7f51feffd640 10 client.15096 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f51feffd640 1 -- 172.21.15.28:0/556582406 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d99e6db600 con 0x55d99e783710 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f51feffd640 10 client.15096 handle_command_reply: tid=0 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 client.15096 shutdown 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7682f0 msgr2=0x55d99e76f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7682f0 0x55d99e76f7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5210000c00 tx=0x7f52100043b0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 shutdown_connections 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e772bb0 0x55d99e76fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e783710 0x55d99e784b80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7682f0 0x55d99e76f7a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 --2- 172.21.15.28:0/556582406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7679d0 0x55d99e772450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f51feffd640 0 client.15096 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 >> 172.21.15.28:0/556582406 conn(0x55d99e763f60 msgr2=0x55d99e769c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:43.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 shutdown_connections 2024-09-15T21:56:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.565+0000 7f522812c740 1 -- 172.21.15.28:0/556582406 wait complete. 2024-09-15T21:56:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.566+0000 7f522812c740 20 client.15096 trim_cache size 0 max 16384 2024-09-15T21:56:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.583+0000 7f522812c740 1 Processor -- start 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.583+0000 7f522812c740 1 -- start start 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f522812c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7679d0 0x55d99e784b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f522812c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7682f0 0x55d99e782560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f522812c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e78ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f522812c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d99e782ec0 con 0x55d99e772bb0 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f522812c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d99e783590 con 0x55d99e7682f0 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f522812c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d99e783c60 con 0x55d99e7679d0 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e78ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e78ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42046/0 (socket says 172.21.15.28:42046) 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 -- 172.21.15.28:0/1412321782 learned_addr learned my addr 172.21.15.28:0/1412321782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7679d0 0x55d99e784b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521f7fe640 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7682f0 0x55d99e782560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 -- 172.21.15.28:0/1412321782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7679d0 msgr2=0x55d99e784b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7679d0 0x55d99e784b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 -- 172.21.15.28:0/1412321782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7682f0 msgr2=0x55d99e782560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7682f0 0x55d99e782560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 -- 172.21.15.28:0/1412321782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d99e768120 con 0x55d99e772bb0 2024-09-15T21:56:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f521effd640 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e78ce60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f52140149c0 tx=0x7f5214012730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cdcb499a8391ce7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.584+0000 7f52057fa640 1 -- 172.21.15.28:0/1412321782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52140140b0 con 0x55d99e772bb0 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f52057fa640 1 -- 172.21.15.28:0/1412321782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5214035020 con 0x55d99e772bb0 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f52057fa640 1 -- 172.21.15.28:0/1412321782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214012af0 con 0x55d99e772bb0 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 -- 172.21.15.28:0/1412321782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 msgr2=0x55d99e78ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e78ce60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f52140149c0 tx=0x7f5214012730 comp rx=0 tx=0).stop 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 -- 172.21.15.28:0/1412321782 shutdown_connections 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e772bb0 0x55d99e78ce60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e7682f0 0x55d99e782560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 --2- 172.21.15.28:0/1412321782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7679d0 0x55d99e784b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 -- 172.21.15.28:0/1412321782 >> 172.21.15.28:0/1412321782 conn(0x55d99e763f60 msgr2=0x55d99e763000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 -- 172.21.15.28:0/1412321782 shutdown_connections 2024-09-15T21:56:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.585+0000 7f522812c740 1 -- 172.21.15.28:0/1412321782 wait complete. 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 -- start start 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 Processor -- start 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7679d0 0x55d99e788510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e788a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e772bb0 0x55d99e7e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d99e784a00 con 0x55d99e7679d0 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d99e784330 con 0x55d99e772bb0 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d99e7850d0 con 0x55d99e7682f0 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f522812c740 10 client.0 _ll_register_callbacks cb 0x7f5210009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f521effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7679d0 0x55d99e788510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f521effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7679d0 0x55d99e788510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42060/0 (socket says 172.21.15.28:42060) 2024-09-15T21:56:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f521effd640 1 -- 172.21.15.28:0/3334395646 learned_addr learned my addr 172.21.15.28:0/3334395646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f521f7fe640 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e788a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.586+0000 7f521effd640 1 -- 172.21.15.28:0/3334395646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 msgr2=0x55d99e788a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f521effd640 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e788a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f521effd640 1 -- 172.21.15.28:0/3334395646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e772bb0 msgr2=0x55d99e7e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f521effd640 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e772bb0 0x55d99e7e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f521effd640 1 -- 172.21.15.28:0/3334395646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d99e7e6490 con 0x55d99e7679d0 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f521f7fe640 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e788a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f521effd640 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7679d0 0x55d99e788510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f52140124b0 tx=0x7f5214005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47276a0dbcdc5e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f51feffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f51feffd640 1 -- 172.21.15.28:0/3334395646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52140140e0 con 0x55d99e7679d0 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5214035020 con 0x55d99e7679d0 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f51feffd640 1 -- 172.21.15.28:0/3334395646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f5214014280 con 0x55d99e7679d0 2024-09-15T21:56:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f51feffd640 1 -- 172.21.15.28:0/3334395646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5214043050 con 0x55d99e7679d0 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f522812c740 10 client.15102 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d99e7ed820 con 0x55d99e7679d0 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f51feffd640 1 -- 172.21.15.28:0/3334395646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5214035020 con 0x55d99e7679d0 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.587+0000 7f51feffd640 1 -- 172.21.15.28:0/3334395646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f5214012830 con 0x55d99e7679d0 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f522812c740 10 client.15102 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f522812c740 10 client.15102 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f522812c740 20 client.15102 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f522812c740 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e7f1ae0 0x55d99e7f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f522812c740 4 client.15102 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "status"}) -- 0x55d99e6d5b10 con 0x55d99e7f1ae0 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f521f7fe640 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e7f1ae0 0x55d99e7f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f51feffd640 10 client.15102 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:56:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f521f7fe640 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e7f1ae0 0x55d99e7f07f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1536797d81eaae30 server_cookie=370ae864fe1397f4 in_seq=0 out_seq=0 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.588+0000 7f51feffd640 1 -- 172.21.15.28:0/3334395646 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d99e6d5b10 con 0x55d99e7f1ae0 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f51feffd640 10 client.15102 handle_command_reply: tid=0 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 client.15102 shutdown 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7679d0 msgr2=0x55d99e788510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7679d0 0x55d99e788510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f52140124b0 tx=0x7f5214005c50 comp rx=0 tx=0).stop 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 shutdown_connections 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d99e772bb0 0x55d99e7e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d99e7682f0 0x55d99e788a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55d99e7f1ae0 0x55d99e7f07f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 --2- 172.21.15.28:0/3334395646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d99e7679d0 0x55d99e788510 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 >> 172.21.15.28:0/3334395646 conn(0x55d99e763f60 msgr2=0x55d99e76df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f51feffd640 0 client.15102 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 shutdown_connections 2024-09-15T21:56:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 1 -- 172.21.15.28:0/3334395646 wait complete. 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f522812c740 20 client.15102 trim_cache size 0 max 16384 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24458, 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 67.042423240999995, 2024-09-15T21:56:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:56:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-09-15T21:56:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:56:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 126.894638859 2024-09-15T21:56:43.597 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:56:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f52080a3350 msgr2=0x7f52080a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f5225d53640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f52080a3350 0x7f52080a5810 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5218098900 tx=0x7f521800efd0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220102ef0 msgr2=0x7f522019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f5225d53640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220102ef0 0x7f522019ce50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f521800eb10 tx=0x7f521800c2d0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 shutdown_connections 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f5225d53640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f52080a3350 0x7f52080a5810 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.589+0000 7f5225d53640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5220102ef0 0x7f522019ce50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.590+0000 7f5225d53640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5220102570 0x7f5220198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.590+0000 7f5225d53640 1 --2- 172.21.15.28:0/3367768994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5220100e20 0x7f5220198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.590+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 >> 172.21.15.28:0/3367768994 conn(0x7f52200fc7d0 msgr2=0x7f52200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.590+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 shutdown_connections 2024-09-15T21:56:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:43.590+0000 7f5225d53640 1 -- 172.21.15.28:0/3367768994 wait complete. 2024-09-15T21:56:44.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-15T21:56:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:45.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:44 smithi028 ceph-mon[30969]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:45.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:44 smithi203 ceph-mon[35546]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 -- 172.21.15.28:0/4199874053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4101420 msgr2=0x7fe5c410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 --2- 172.21.15.28:0/4199874053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4101420 0x7fe5c410d950 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009cd0 tx=0x7fe5bc02a8e0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 -- 172.21.15.28:0/4199874053 shutdown_connections 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 --2- 172.21.15.28:0/4199874053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4101420 0x7fe5c410d950 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 --2- 172.21.15.28:0/4199874053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5c4100a80 0x7fe5c4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 --2- 172.21.15.28:0/4199874053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5c4106a80 0x7fe5c4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 -- 172.21.15.28:0/4199874053 >> 172.21.15.28:0/4199874053 conn(0x7fe5c40fc7a0 msgr2=0x7fe5c40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 -- 172.21.15.28:0/4199874053 shutdown_connections 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.229+0000 7fe5ca957640 1 -- 172.21.15.28:0/4199874053 wait complete. 2024-09-15T21:56:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 Processor -- start 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 -- start start 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4100a80 0x7fe5c4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5c4101420 0x7fe5c41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5c4106a80 0x7fe5c419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4110ec0 con 0x7fe5c4100a80 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4110d40 con 0x7fe5c4106a80 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5ca957640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4111040 con 0x7fe5c4101420 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.230+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4100a80 0x7fe5c4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4100a80 0x7fe5c4198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42078/0 (socket says 172.21.15.28:42078) 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 -- 172.21.15.28:0/2185683866 learned_addr learned my addr 172.21.15.28:0/2185683866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5c4101420 0x7fe5c41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5c4106a80 0x7fe5c419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 -- 172.21.15.28:0/2185683866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5c4101420 msgr2=0x7fe5c41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5c4101420 0x7fe5c41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 -- 172.21.15.28:0/2185683866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5c4106a80 msgr2=0x7fe5c419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5c4106a80 0x7fe5c419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 -- 172.21.15.28:0/2185683866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c419d410 con 0x7fe5c4100a80 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4100a80 0x7fe5c4198460 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400b4a0 tx=0x7fe5b400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c17fa640 1 -- 172.21.15.28:0/2185683866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4014070 con 0x7fe5c4100a80 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c419d700 con 0x7fe5c4100a80 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c17fa640 1 -- 172.21.15.28:0/2185683866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5b4010af0 con 0x7fe5c4100a80 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5c17fa640 1 -- 172.21.15.28:0/2185683866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4017d10 con 0x7fe5c4100a80 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.231+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c41a4fe0 con 0x7fe5c4100a80 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.234+0000 7fe5c17fa640 1 -- 172.21.15.28:0/2185683866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe5b4017070 con 0x7fe5c4100a80 2024-09-15T21:56:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.234+0000 7fe5ccd30740 1 Processor -- start 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.234+0000 7fe5ccd30740 1 -- start start 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2bb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559a7a210090 con 0x559a7a2a98d0 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559a7a1b7a70 con 0x559a7a2b8fc0 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559a7a20c050 con 0x559a7a2aa1f0 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2a9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42086/0 (socket says 172.21.15.28:42086) 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 -- 172.21.15.28:0/3945950892 learned_addr learned my addr 172.21.15.28:0/3945950892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2bb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c17fa640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe5ac0a32d0 0x7fe5ac0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 -- 172.21.15.28:0/3945950892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 msgr2=0x559a7a2b8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 -- 172.21.15.28:0/3945950892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 msgr2=0x559a7a2bb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2bb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 -- 172.21.15.28:0/3945950892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a7a1c4130 con 0x559a7a2a98d0 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe5ac0a32d0 0x7fe5ac0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.235+0000 7fe5c17fa640 1 -- 172.21.15.28:0/2185683866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fe5b40cf0c0 con 0x7fe5c4100a80 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.236+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2a9cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc005cc0 tx=0x7fe5bc031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6a5d7e5a9a339f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.236+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe5ac0a32d0 0x7fe5ac0a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc031860 tx=0x7fe5bc03f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.236+0000 7fe5aa7fc640 1 -- 172.21.15.28:0/3945950892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc0318c0 con 0x559a7a2a98d0 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.236+0000 7fe5aa7fc640 1 -- 172.21.15.28:0/3945950892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5bc036040 con 0x559a7a2a98d0 2024-09-15T21:56:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.236+0000 7fe5aa7fc640 1 -- 172.21.15.28:0/3945950892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc030e60 con 0x559a7a2a98d0 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3945950892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 msgr2=0x559a7a2a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2a9cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc005cc0 tx=0x7fe5bc031630 comp rx=0 tx=0).stop 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3945950892 shutdown_connections 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2bb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3945950892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2a9cb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3945950892 >> 172.21.15.28:0/3945950892 conn(0x559a7a2a5f60 msgr2=0x559a7a2a6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3945950892 shutdown_connections 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3945950892 wait complete. 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.237+0000 7fe5ccd30740 1 -- start start 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 1 Processor -- start 2024-09-15T21:56:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2bbb90 con 0x559a7a2a98d0 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559a7a1caf20 con 0x559a7a2b8fc0 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2bbd10 con 0x559a7a2aa1f0 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5ccd30740 10 client.0 _ll_register_callbacks cb 0x559a7a194030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42090/0 (socket says 172.21.15.28:42090) 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 -- 172.21.15.28:0/3921737876 learned_addr learned my addr 172.21.15.28:0/3921737876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 -- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 msgr2=0x559a7a2b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 -- 172.21.15.28:0/3921737876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 msgr2=0x559a7a2b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2b61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 -- 172.21.15.28:0/3921737876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a79f812b0 con 0x559a7a2a98d0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b8920 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4009420 tx=0x7fe5b4009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889e57c084b1441 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.238+0000 7fe5a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5a37fe640 1 -- 172.21.15.28:0/3921737876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b401e070 con 0x559a7a2a98d0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5a37fe640 1 -- 172.21.15.28:0/3921737876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fe5b4010ca0 con 0x559a7a2a98d0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5bc036040 con 0x559a7a2a98d0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5a37fe640 1 -- 172.21.15.28:0/3921737876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe5b409ac30 con 0x559a7a2a98d0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 10 client.15114 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559a7a2c75a0 con 0x559a7a2a98d0 2024-09-15T21:56:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5a37fe640 1 -- 172.21.15.28:0/3921737876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5bc036040 con 0x559a7a2a98d0 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5a37fe640 1 -- 172.21.15.28:0/3921737876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe5b4097940 con 0x559a7a2a98d0 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 10 client.15114 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 10 client.15114 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 20 client.15114 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a2c9c00 0x559a7a2cb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 4 client.15114 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.239+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a7a21d600 con 0x559a7a2c9c00 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.240+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a2c9c00 0x559a7a2cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.240+0000 7fe5a37fe640 10 client.15114 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.240+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a2c9c00 0x559a7a2cb070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a7f09b00aae0c919 server_cookie=ca6f0a66566ed544 in_seq=0 out_seq=0 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5a37fe640 1 -- 172.21.15.28:0/3921737876 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559a7a21d600 con 0x559a7a2c9c00 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5a37fe640 10 client.15114 handle_command_reply: tid=0 2024-09-15T21:56:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 client.15114 shutdown 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 msgr2=0x559a7a2b8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b8920 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4009420 tx=0x7fe5b4009380 comp rx=0 tx=0).stop 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 shutdown_connections 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a2c9c00 0x559a7a2cb070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2b8fc0 0x559a7a2b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5a37fe640 0 client.15114 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/3921737876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b8920 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 >> 172.21.15.28:0/3921737876 conn(0x559a7a2a5f60 msgr2=0x559a7a2abcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 shutdown_connections 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 1 -- 172.21.15.28:0/3921737876 wait complete. 2024-09-15T21:56:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.241+0000 7fe5ccd30740 20 client.15114 trim_cache size 0 max 16384 2024-09-15T21:56:45.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:44 smithi174 ceph-mon[35244]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.259+0000 7fe5ccd30740 1 Processor -- start 2024-09-15T21:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.259+0000 7fe5ccd30740 1 -- start start 2024-09-15T21:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.259+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2a98d0 0x559a7a2cb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.259+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2aa1f0 0x559a7a2ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.259+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 0x559a7a2c9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2c93b0 con 0x559a7a2aa1f0 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2c9a80 con 0x559a7a2a98d0 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2ca150 con 0x559a7a2b8fc0 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2aa1f0 0x559a7a2ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2aa1f0 0x559a7a2ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42104/0 (socket says 172.21.15.28:42104) 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 -- 172.21.15.28:0/150770568 learned_addr learned my addr 172.21.15.28:0/150770568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2a98d0 0x559a7a2cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 -- 172.21.15.28:0/150770568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 msgr2=0x559a7a2c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 0x559a7a2c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 -- 172.21.15.28:0/150770568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2a98d0 msgr2=0x559a7a2cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2a98d0 0x559a7a2cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 -- 172.21.15.28:0/150770568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a7a2b8c90 con 0x559a7a2aa1f0 2024-09-15T21:56:45.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2aa1f0 0x559a7a2ca9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4105190 tx=0x7fe5b800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7635a0ad1cbaba81 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:45.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5a9ffb640 1 -- 172.21.15.28:0/150770568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8003de0 con 0x559a7a2aa1f0 2024-09-15T21:56:45.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5a9ffb640 1 -- 172.21.15.28:0/150770568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5b8011920 con 0x559a7a2aa1f0 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.260+0000 7fe5a9ffb640 1 -- 172.21.15.28:0/150770568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b801ca50 con 0x559a7a2aa1f0 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 -- 172.21.15.28:0/150770568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2aa1f0 msgr2=0x559a7a2ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2aa1f0 0x559a7a2ca9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4105190 tx=0x7fe5b800bf50 comp rx=0 tx=0).stop 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 -- 172.21.15.28:0/150770568 shutdown_connections 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 0x559a7a2c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2aa1f0 0x559a7a2ca9a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/150770568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2a98d0 0x559a7a2cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 -- 172.21.15.28:0/150770568 >> 172.21.15.28:0/150770568 conn(0x559a7a2a5f60 msgr2=0x559a7a2a6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 -- 172.21.15.28:0/150770568 shutdown_connections 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 -- 172.21.15.28:0/150770568 wait complete. 2024-09-15T21:56:45.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.261+0000 7fe5ccd30740 1 -- start start 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 1 Processor -- start 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 0x559a7a2d22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2caef0 con 0x559a7a2a98d0 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2ca820 con 0x559a7a2aa1f0 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559a7a2cb5c0 con 0x559a7a2b8fc0 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5ccd30740 10 client.0 _ll_register_callbacks cb 0x7fe5b400b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42112/0 (socket says 172.21.15.28:42112) 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 -- 172.21.15.28:0/4072531104 learned_addr learned my addr 172.21.15.28:0/4072531104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c8ef4640 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 0x559a7a2d22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 -- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 msgr2=0x559a7a2d22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 0x559a7a2d22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 -- 172.21.15.28:0/4072531104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2aa1f0 msgr2=0x559a7a2b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 -- 172.21.15.28:0/4072531104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a7a2aa020 con 0x559a7a2a98d0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5c3fff640 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b22a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400be60 tx=0x7fe5b4017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f0ed7f49305566 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.262+0000 7fe5a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5a37fe640 1 -- 172.21.15.28:0/4072531104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b401e070 con 0x559a7a2a98d0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5a37fe640 1 -- 172.21.15.28:0/4072531104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fe5b40cc900 con 0x559a7a2a98d0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5b8011920 con 0x559a7a2a98d0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5a37fe640 1 -- 172.21.15.28:0/4072531104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe5b409b9d0 con 0x559a7a2a98d0 2024-09-15T21:56:45.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 10 client.15120 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559a7a2d37b0 con 0x559a7a2a98d0 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5a37fe640 1 -- 172.21.15.28:0/4072531104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5b8011920 con 0x559a7a2a98d0 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5a37fe640 1 -- 172.21.15.28:0/4072531104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe5b40d2280 con 0x559a7a2a98d0 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 10 client.15120 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 10 client.15120 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 20 client.15120 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a338510 0x559a7a337220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 4 client.15120 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "status"}) -- 0x559a7a217b10 con 0x559a7a338510 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.263+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a338510 0x559a7a337220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5c37fe640 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a338510 0x559a7a337220 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2317793f27d4f09 server_cookie=eee530dc5647d04 in_seq=0 out_seq=0 2024-09-15T21:56:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5a37fe640 10 client.15120 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5a37fe640 1 -- 172.21.15.28:0/4072531104 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559a7a217b10 con 0x559a7a338510 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5a37fe640 10 client.15120 handle_command_reply: tid=0 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 client.15120 shutdown 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 msgr2=0x559a7a2b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b22a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400be60 tx=0x7fe5b4017730 comp rx=0 tx=0).stop 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 shutdown_connections 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x559a7a338510 0x559a7a337220 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559a7a2b8fc0 0x559a7a2d22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a7a2aa1f0 0x559a7a2b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 --2- 172.21.15.28:0/4072531104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7a2a98d0 0x559a7a2b22a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5a37fe640 0 client.15120 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:56:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.264+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 >> 172.21.15.28:0/4072531104 conn(0x559a7a2a5f60 msgr2=0x559a7a2b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 shutdown_connections 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ccd30740 1 -- 172.21.15.28:0/4072531104 wait complete. 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ccd30740 20 client.15120 trim_cache size 0 max 16384 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24460, 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:56:45.272 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 69.730733932999996, 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 131.44083714499999 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:56:45.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe5ac0a32d0 msgr2=0x7fe5ac0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe5ac0a32d0 0x7fe5ac0a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc031860 tx=0x7fe5bc03f000 comp rx=0 tx=0).stop 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4100a80 msgr2=0x7fe5c4198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4100a80 0x7fe5c4198460 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400b4a0 tx=0x7fe5b400b960 comp rx=0 tx=0).stop 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 shutdown_connections 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe5ac0a32d0 0x7fe5ac0a5790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5c4106a80 0x7fe5c419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5c4101420 0x7fe5c41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 --2- 172.21.15.28:0/2185683866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c4100a80 0x7fe5c4198460 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 >> 172.21.15.28:0/2185683866 conn(0x7fe5c40fc7a0 msgr2=0x7fe5c406aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 shutdown_connections 2024-09-15T21:56:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:45.265+0000 7fe5ca957640 1 -- 172.21.15.28:0/2185683866 wait complete. 2024-09-15T21:56:45.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T21:56:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 -- 172.21.15.28:0/4091108235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 msgr2=0x7fbb041021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/4091108235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb041021c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009900 tx=0x7fbaf402a410 comp rx=0 tx=0).stop 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 -- 172.21.15.28:0/4091108235 shutdown_connections 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/4091108235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb04106f10 0x7fbb0410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/4091108235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb04102790 0x7fbb041067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/4091108235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb041021c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 -- 172.21.15.28:0/4091108235 >> 172.21.15.28:0/4091108235 conn(0x7fbb040fd8c0 msgr2=0x7fbb040ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 -- 172.21.15.28:0/4091108235 shutdown_connections 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.739+0000 7fbb0a64b640 1 -- 172.21.15.28:0/4091108235 wait complete. 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 Processor -- start 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 -- start start 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb04198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb04102790 0x7fbb04198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb04106f10 0x7fbb0419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb04111fc0 con 0x7fbb04101de0 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb04111e40 con 0x7fbb04106f10 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb0a64b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb04112140 con 0x7fbb04102790 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb04198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb04198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42140/0 (socket says 172.21.15.28:42140) 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb03fff640 1 -- 172.21.15.28:0/2189375533 learned_addr learned my addr 172.21.15.28:0/2189375533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb08be8640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb04106f10 0x7fbb0419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.740+0000 7fbb037fe640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb04102790 0x7fbb04198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb03fff640 1 -- 172.21.15.28:0/2189375533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb04102790 msgr2=0x7fbb04198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb04102790 0x7fbb04198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb03fff640 1 -- 172.21.15.28:0/2189375533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb04106f10 msgr2=0x7fbb0419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb04106f10 0x7fbb0419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb03fff640 1 -- 172.21.15.28:0/2189375533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0419d3f0 con 0x7fbb04101de0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb04198440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4004770 tx=0x7fbaf4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb017fa640 1 -- 172.21.15.28:0/2189375533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf403f070 con 0x7fbb04101de0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb017fa640 1 -- 172.21.15.28:0/2189375533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbaf4031ca0 con 0x7fbb04101de0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0419d680 con 0x7fbb04101de0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb017fa640 1 -- 172.21.15.28:0/2189375533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4035030 con 0x7fbb04101de0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.741+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb041a4fc0 con 0x7fbb04101de0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.744+0000 7fbb017fa640 1 -- 172.21.15.28:0/2189375533 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbaf4042070 con 0x7fbb04101de0 2024-09-15T21:56:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.744+0000 7fbb0ca24740 1 Processor -- start 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.744+0000 7fbb0ca24740 1 -- start start 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328b8b0 0x55f32328bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f32329aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f32329d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3231f2090 con 0x55f32329afc0 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f323199a70 con 0x55f32328c1d0 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f3231ee050 con 0x55f32328b8b0 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f32329d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f32329d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42152/0 (socket says 172.21.15.28:42152) 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 -- 172.21.15.28:0/989520835 learned_addr learned my addr 172.21.15.28:0/989520835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb03fff640 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f32329aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb08be8640 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328b8b0 0x55f32328bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb017fa640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbaec0a32d0 0x7fbaec0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 -- 172.21.15.28:0/989520835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328b8b0 msgr2=0x55f32328bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328b8b0 0x55f32328bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 -- 172.21.15.28:0/989520835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 msgr2=0x55f32329aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f32329aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 -- 172.21.15.28:0/989520835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3231a6130 con 0x55f32329afc0 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb03fff640 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f32329aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb08be8640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbaec0a32d0 0x7fbaec0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb017fa640 1 -- 172.21.15.28:0/2189375533 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fbaf40f0580 con 0x7fbb04101de0 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.745+0000 7fbb037fe640 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f32329d3b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800bcb0 tx=0x7fbaf80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a0a0f290cb5afb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbaea7fc640 1 -- 172.21.15.28:0/989520835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8004030 con 0x55f32329afc0 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbaea7fc640 1 -- 172.21.15.28:0/989520835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbaf80041d0 con 0x55f32329afc0 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbaea7fc640 1 -- 172.21.15.28:0/989520835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8011800 con 0x55f32329afc0 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 -- 172.21.15.28:0/989520835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 msgr2=0x55f32329d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb08be8640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbaec0a32d0 0x7fbaec0a5790 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbafc007ec0 tx=0x7fbafc004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f32329d3b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800bcb0 tx=0x7fbaf80107a0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 -- 172.21.15.28:0/989520835 shutdown_connections 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f32329d3b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f32329aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/989520835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328b8b0 0x55f32328bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 -- 172.21.15.28:0/989520835 >> 172.21.15.28:0/989520835 conn(0x55f323287f60 msgr2=0x55f323288340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 -- 172.21.15.28:0/989520835 shutdown_connections 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.746+0000 7fbb0ca24740 1 -- 172.21.15.28:0/989520835 wait complete. 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 -- start start 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 Processor -- start 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32328b8b0 0x55f32329a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f323297bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32329afc0 0x55f3232980f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f32329db90 con 0x55f32328b8b0 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f3231acf20 con 0x55f32328c1d0 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f32329dd10 con 0x55f32329afc0 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb0ca24740 10 client.0 _ll_register_callbacks cb 0x55f323176030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32328b8b0 0x55f32329a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32328b8b0 0x55f32329a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42160/0 (socket says 172.21.15.28:42160) 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.747+0000 7fbb03fff640 1 -- 172.21.15.28:0/2503947909 learned_addr learned my addr 172.21.15.28:0/2503947909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb037fe640 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f323297bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb03fff640 1 -- 172.21.15.28:0/2503947909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32329afc0 msgr2=0x55f3232980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32329afc0 0x55f3232980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb03fff640 1 -- 172.21.15.28:0/2503947909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 msgr2=0x55f323297bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f323297bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb03fff640 1 -- 172.21.15.28:0/2503947909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f322f632b0 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32328b8b0 0x55f32329a860 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4031550 tx=0x7fbaf402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66644c6cec1b8fc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbae37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbae37fe640 1 -- 172.21.15.28:0/2503947909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4049050 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbae37fe640 1 -- 172.21.15.28:0/2503947909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fbaf40eddc0 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaf80041d0 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbae37fe640 1 -- 172.21.15.28:0/2503947909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbaf40bc270 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb0ca24740 10 client.15132 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f3232a94f0 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.748+0000 7fbae37fe640 1 -- 172.21.15.28:0/2503947909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbaf80041d0 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbae37fe640 1 -- 172.21.15.28:0/2503947909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fbaf403b5f0 con 0x55f32328b8b0 2024-09-15T21:56:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbb0ca24740 10 client.15132 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbb0ca24740 10 client.15132 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbb0ca24740 20 client.15132 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f3232abb30 0x55f3232acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbb0ca24740 4 client.15132 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3231ff600 con 0x55f3232abb30 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.749+0000 7fbb037fe640 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f3232abb30 0x55f3232acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.750+0000 7fbb037fe640 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f3232abb30 0x55f3232acfa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c401c94d928d04a server_cookie=7067def9a521dc2a in_seq=0 out_seq=0 2024-09-15T21:56:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.750+0000 7fbae37fe640 10 client.15132 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:56:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.750+0000 7fbae37fe640 1 -- 172.21.15.28:0/2503947909 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f3231ff600 con 0x55f3232abb30 2024-09-15T21:56:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.750+0000 7fbae37fe640 10 client.15132 handle_command_reply: tid=0 2024-09-15T21:56:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.750+0000 7fbb0ca24740 1 client.15132 shutdown 2024-09-15T21:56:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32328b8b0 msgr2=0x55f32329a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32328b8b0 0x55f32329a860 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4031550 tx=0x7fbaf402a410 comp rx=0 tx=0).stop 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 shutdown_connections 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f3232abb30 0x55f3232acfa0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32329afc0 0x55f3232980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328c1d0 0x55f323297bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2503947909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32328b8b0 0x55f32329a860 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 >> 172.21.15.28:0/2503947909 conn(0x55f323287f60 msgr2=0x55f32328dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbae37fe640 0 client.15132 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 shutdown_connections 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2503947909 wait complete. 2024-09-15T21:56:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.751+0000 7fbb0ca24740 20 client.15132 trim_cache size 0 max 16384 2024-09-15T21:56:46.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.768+0000 7fbb0ca24740 1 Processor -- start 2024-09-15T21:56:46.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.768+0000 7fbb0ca24740 1 -- start start 2024-09-15T21:56:46.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f323288340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3232ab2e0 con 0x55f32329afc0 2024-09-15T21:56:46.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f3232ab9b0 con 0x55f32328b8b0 2024-09-15T21:56:46.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f3232ac080 con 0x55f32328c1d0 2024-09-15T21:56:46.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f323288340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f323288340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42162/0 (socket says 172.21.15.28:42162) 2024-09-15T21:56:46.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 -- 172.21.15.28:0/4259472651 learned_addr learned my addr 172.21.15.28:0/4259472651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb037fe640 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb03fff640 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 -- 172.21.15.28:0/4259472651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 msgr2=0x55f3232ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 -- 172.21.15.28:0/4259472651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 msgr2=0x55f3232ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 -- 172.21.15.28:0/4259472651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f323288880 con 0x55f32329afc0 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbb08be8640 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f323288340 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbafc012610 tx=0x7fbafc014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d215380a95d2fa84 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.769+0000 7fbae9ffb640 1 -- 172.21.15.28:0/4259472651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc01e920 con 0x55f32329afc0 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbae9ffb640 1 -- 172.21.15.28:0/4259472651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbafc022030 con 0x55f32329afc0 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbae9ffb640 1 -- 172.21.15.28:0/4259472651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc01b6a0 con 0x55f32329afc0 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 -- 172.21.15.28:0/4259472651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 msgr2=0x55f323288340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f323288340 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fbafc012610 tx=0x7fbafc014070 comp rx=0 tx=0).stop 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 -- 172.21.15.28:0/4259472651 shutdown_connections 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f323288340 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/4259472651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 -- 172.21.15.28:0/4259472651 >> 172.21.15.28:0/4259472651 conn(0x55f323287f60 msgr2=0x55f323282dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 -- 172.21.15.28:0/4259472651 shutdown_connections 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.770+0000 7fbb0ca24740 1 -- 172.21.15.28:0/4259472651 wait complete. 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 -- start start 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 Processor -- start 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f3232aec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f3232ace20 con 0x55f32329afc0 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f3232ac750 con 0x55f32328b8b0 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f3232ad4f0 con 0x55f32328c1d0 2024-09-15T21:56:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb0ca24740 10 client.0 _ll_register_callbacks cb 0x7fbaf40b8310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb08be8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f3232aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb08be8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f3232aec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42166/0 (socket says 172.21.15.28:42166) 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb08be8640 1 -- 172.21.15.28:0/2707881 learned_addr learned my addr 172.21.15.28:0/2707881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.771+0000 7fbb037fe640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb08be8640 1 -- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 msgr2=0x55f3232ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb08be8640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb08be8640 1 -- 172.21.15.28:0/2707881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 msgr2=0x55f3232ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb08be8640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb08be8640 1 -- 172.21.15.28:0/2707881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f323314230 con 0x55f32329afc0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb08be8640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f3232aec20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbafc01dc40 tx=0x7fbafc0075b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ade36d81a77a36 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbae37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbae37fe640 1 -- 172.21.15.28:0/2707881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc01e720 con 0x55f32329afc0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbae37fe640 1 -- 172.21.15.28:0/2707881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fbafc01e8c0 con 0x55f32329afc0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbafc022030 con 0x55f32329afc0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbae37fe640 1 -- 172.21.15.28:0/2707881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbafc03b050 con 0x55f32329afc0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb0ca24740 10 client.15138 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f323315530 con 0x55f32329afc0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.772+0000 7fbae37fe640 1 -- 172.21.15.28:0/2707881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbafc022030 con 0x55f32329afc0 2024-09-15T21:56:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbae37fe640 1 -- 172.21.15.28:0/2707881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fbafc01bce0 con 0x55f32329afc0 2024-09-15T21:56:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb0ca24740 10 client.15138 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb0ca24740 10 client.15138 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T21:56:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb0ca24740 20 client.15138 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f323319d20 0x55f323318a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb0ca24740 4 client.15138 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:56:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "status"}) -- 0x55f3231f9b10 con 0x55f323319d20 2024-09-15T21:56:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f323319d20 0x55f323318a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbae37fe640 10 client.15138 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:56:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.773+0000 7fbb03fff640 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f323319d20 0x55f323318a30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa7b1683641d9f94 server_cookie=4cf08721d2aecb59 in_seq=0 out_seq=0 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbae37fe640 1 -- 172.21.15.28:0/2707881 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f3231f9b10 con 0x55f323319d20 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbae37fe640 10 client.15138 handle_command_reply: tid=0 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 client.15138 shutdown 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 msgr2=0x55f3232aec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f3232aec20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbafc01dc40 tx=0x7fbafc0075b0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 shutdown_connections 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f323319d20 0x55f323318a30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f32329afc0 0x55f3232aec20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f32328c1d0 0x55f3232ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 --2- 172.21.15.28:0/2707881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f32328b8b0 0x55f3232ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbae37fe640 0 client.15138 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:56:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 >> 172.21.15.28:0/2707881 conn(0x55f323287f60 msgr2=0x55f323292a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 shutdown_connections 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 1 -- 172.21.15.28:0/2707881 wait complete. 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.774+0000 7fbb0ca24740 20 client.15138 trim_cache size 0 max 16384 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24481, 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:56:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:56:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:56:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 72.245757396000002, 2024-09-15T21:56:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:56:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-09-15T21:56:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:56:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 127.478710929 2024-09-15T21:56:46.783 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbaec0a32d0 msgr2=0x7fbaec0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbaec0a32d0 0x7fbaec0a5790 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbafc007ec0 tx=0x7fbafc004a40 comp rx=0 tx=0).stop 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 msgr2=0x7fbb04198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb04198440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4004770 tx=0x7fbaf4003c10 comp rx=0 tx=0).stop 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 shutdown_connections 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fbaec0a32d0 0x7fbaec0a5790 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb04106f10 0x7fbb0419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb04102790 0x7fbb04198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 --2- 172.21.15.28:0/2189375533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb04101de0 0x7fbb04198440 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 >> 172.21.15.28:0/2189375533 conn(0x7fbb040fd8c0 msgr2=0x7fbb040feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 shutdown_connections 2024-09-15T21:56:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:46.775+0000 7fbb0a64b640 1 -- 172.21.15.28:0/2189375533 wait complete. 2024-09-15T21:56:46.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:46 smithi028 ceph-mon[30969]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:47.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:46 smithi203 ceph-mon[35546]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:47.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:46 smithi174 ceph-mon[35244]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:47.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T21:56:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 -- 172.21.15.28:0/1929207960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 msgr2=0x7fc66410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 --2- 172.21.15.28:0/1929207960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc66410df40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009900 tx=0x7fc64c02a410 comp rx=0 tx=0).stop 2024-09-15T21:56:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 -- 172.21.15.28:0/1929207960 shutdown_connections 2024-09-15T21:56:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 --2- 172.21.15.28:0/1929207960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc66410e480 0x7fc664110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 --2- 172.21.15.28:0/1929207960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc66410df40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 --2- 172.21.15.28:0/1929207960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6641008d0 0x7fc664100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 -- 172.21.15.28:0/1929207960 >> 172.21.15.28:0/1929207960 conn(0x7fc6640fc7a0 msgr2=0x7fc6640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 -- 172.21.15.28:0/1929207960 shutdown_connections 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.315+0000 7fc6699cd640 1 -- 172.21.15.28:0/1929207960 wait complete. 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 Processor -- start 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 -- start start 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6641008d0 0x7fc6641a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc6641a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc66410e480 0x7fc66419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc664114250 con 0x7fc664101280 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641140d0 con 0x7fc66410e480 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc6699cd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641143d0 con 0x7fc6641008d0 2024-09-15T21:56:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc6641a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc6641a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42188/0 (socket says 172.21.15.28:42188) 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.316+0000 7fc65bfff640 1 -- 172.21.15.28:0/3523955283 learned_addr learned my addr 172.21.15.28:0/3523955283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6641008d0 0x7fc6641a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc66410e480 0x7fc66419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc65bfff640 1 -- 172.21.15.28:0/3523955283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6641008d0 msgr2=0x7fc6641a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc65bfff640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6641008d0 0x7fc6641a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc65bfff640 1 -- 172.21.15.28:0/3523955283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc66410e480 msgr2=0x7fc66419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc65bfff640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc66410e480 0x7fc66419a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc65bfff640 1 -- 172.21.15.28:0/3523955283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66419acf0 con 0x7fc664101280 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6641008d0 0x7fc6641a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc66410e480 0x7fc66419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc65bfff640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc6641a0940 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc64c004770 tx=0x7fc64c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc6689cb640 1 -- 172.21.15.28:0/3523955283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c03f070 con 0x7fc664101280 2024-09-15T21:56:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc6689cb640 1 -- 172.21.15.28:0/3523955283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc64c031ca0 con 0x7fc664101280 2024-09-15T21:56:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66419af80 con 0x7fc664101280 2024-09-15T21:56:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc6689cb640 1 -- 172.21.15.28:0/3523955283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c035030 con 0x7fc664101280 2024-09-15T21:56:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.317+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6641a72d0 con 0x7fc664101280 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.319+0000 7fc6689cb640 1 -- 172.21.15.28:0/3523955283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc64c042070 con 0x7fc664101280 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 Processor -- start 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 -- start start 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf3edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf49670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf4bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ebdea5090 con 0x557ebdf3e9d0 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ebde4ca70 con 0x557ebdf3f2f0 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc66bda6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557ebdea1050 con 0x557ebdf49bb0 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc6637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf3edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc6637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf3edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42194/0 (socket says 172.21.15.28:42194) 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc6637fe640 1 -- 172.21.15.28:0/3947024419 learned_addr learned my addr 172.21.15.28:0/3947024419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc65bfff640 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf4bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.320+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf49670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6637fe640 1 -- 172.21.15.28:0/3947024419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 msgr2=0x557ebdf4bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf4bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6637fe640 1 -- 172.21.15.28:0/3947024419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 msgr2=0x557ebdf49670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf49670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6637fe640 1 -- 172.21.15.28:0/3947024419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebde59130 con 0x557ebdf3e9d0 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6689cb640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc6500a3350 0x7fc6500a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf3edb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009f50 tx=0x7fc65c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33b792d3e371b49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc6500a3350 0x7fc6500a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc6689cb640 1 -- 172.21.15.28:0/3523955283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fc64c0f0620 con 0x7fc664101280 2024-09-15T21:56:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc659ffb640 1 -- 172.21.15.28:0/3947024419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c016070 con 0x557ebdf3e9d0 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc659ffb640 1 -- 172.21.15.28:0/3947024419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc65c012930 con 0x557ebdf3e9d0 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.321+0000 7fc659ffb640 1 -- 172.21.15.28:0/3947024419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c0119f0 con 0x557ebdf3e9d0 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 -- 172.21.15.28:0/3947024419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 msgr2=0x557ebdf3edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc6500a3350 0x7fc6500a5810 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x557ebdf3bdc0 tx=0x7fc654008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf3edb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009f50 tx=0x7fc65c010800 comp rx=0 tx=0).stop 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 -- 172.21.15.28:0/3947024419 shutdown_connections 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf4bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf49670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3947024419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf3edb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 -- 172.21.15.28:0/3947024419 >> 172.21.15.28:0/3947024419 conn(0x557ebdf3af60 msgr2=0x557ebdf3b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 -- 172.21.15.28:0/3947024419 shutdown_connections 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 -- 172.21.15.28:0/3947024419 wait complete. 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.322+0000 7fc66bda6740 1 -- start start 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 1 Processor -- start 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf467f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf46d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf4c780 con 0x557ebdf3f2f0 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ebde5ff20 con 0x557ebdf3e9d0 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf4c900 con 0x557ebdf49bb0 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc66bda6740 10 client.0 _ll_register_callbacks cb 0x557ebde29030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42196/0 (socket says 172.21.15.28:42196) 2024-09-15T21:56:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 -- 172.21.15.28:0/1161620246 learned_addr learned my addr 172.21.15.28:0/1161620246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc662ffd640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc6637fe640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf46d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 -- 172.21.15.28:0/1161620246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 msgr2=0x557ebdf46d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf46d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 -- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 msgr2=0x557ebdf494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc65bfff640 1 -- 172.21.15.28:0/1161620246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebdc162b0 con 0x557ebdf3f2f0 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.323+0000 7fc6637fe640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf46d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc65bfff640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf467f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc64c000c00 tx=0x7fc64c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab58cbd9f7fe37f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc642ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc642ffd640 1 -- 172.21.15.28:0/1161620246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c049070 con 0x557ebdf3f2f0 2024-09-15T21:56:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc642ffd640 1 -- 172.21.15.28:0/1161620246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fc64c0b8df0 con 0x557ebdf3f2f0 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc65c012930 con 0x557ebdf3f2f0 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc642ffd640 1 -- 172.21.15.28:0/1161620246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc64c031e50 con 0x557ebdf3f2f0 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc66bda6740 10 client.15150 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557ebdf580f0 con 0x557ebdf3f2f0 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc642ffd640 1 -- 172.21.15.28:0/1161620246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc65c012930 con 0x557ebdf3f2f0 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc642ffd640 1 -- 172.21.15.28:0/1161620246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc64c0b8040 con 0x557ebdf3f2f0 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc66bda6740 10 client.15150 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc66bda6740 10 client.15150 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.324+0000 7fc66bda6740 20 client.15150 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.325+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdf5a740 0x557ebdf5bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.325+0000 7fc66bda6740 4 client.15150 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.325+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ebdeb2600 con 0x557ebdf5a740 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.325+0000 7fc662ffd640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdf5a740 0x557ebdf5bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.325+0000 7fc642ffd640 10 client.15150 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:56:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.325+0000 7fc662ffd640 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdf5a740 0x557ebdf5bbb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c3defd65ba33b2 server_cookie=34bcdbef6323db20 in_seq=0 out_seq=0 2024-09-15T21:56:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc642ffd640 1 -- 172.21.15.28:0/1161620246 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557ebdeb2600 con 0x557ebdf5a740 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc642ffd640 10 client.15150 handle_command_reply: tid=0 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 client.15150 shutdown 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 msgr2=0x557ebdf467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf467f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc64c000c00 tx=0x7fc64c009260 comp rx=0 tx=0).stop 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 shutdown_connections 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdf5a740 0x557ebdf5bbb0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf46d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf467f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1161620246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc642ffd640 0 client.15150 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:56:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 >> 172.21.15.28:0/1161620246 conn(0x557ebdf3af60 msgr2=0x557ebdf40c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 shutdown_connections 2024-09-15T21:56:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 1 -- 172.21.15.28:0/1161620246 wait complete. 2024-09-15T21:56:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.326+0000 7fc66bda6740 20 client.15150 trim_cache size 0 max 16384 2024-09-15T21:56:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 Processor -- start 2024-09-15T21:56:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 -- start start 2024-09-15T21:56:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf5bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf35ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf45100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf59ef0 con 0x557ebdf3e9d0 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf5a5c0 con 0x557ebdf3f2f0 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc66bda6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf5ac90 con 0x557ebdf49bb0 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf5bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf5bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42204/0 (socket says 172.21.15.28:42204) 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 -- 172.21.15.28:0/1435870375 learned_addr learned my addr 172.21.15.28:0/1435870375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc6637fe640 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf45100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc65bfff640 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf35ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 -- 172.21.15.28:0/1435870375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 msgr2=0x557ebdf45100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf45100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 -- 172.21.15.28:0/1435870375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 msgr2=0x557ebdf35ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf35ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc662ffd640 1 -- 172.21.15.28:0/1435870375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebdf3bbf0 con 0x557ebdf3e9d0 2024-09-15T21:56:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.345+0000 7fc6637fe640 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf45100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc662ffd640 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf5bbb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc65400fbd0 tx=0x7fc654012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40db93b51c0f9a72 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc6597fa640 1 -- 172.21.15.28:0/1435870375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65401b040 con 0x557ebdf3e9d0 2024-09-15T21:56:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc6597fa640 1 -- 172.21.15.28:0/1435870375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc65401c040 con 0x557ebdf3e9d0 2024-09-15T21:56:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc6597fa640 1 -- 172.21.15.28:0/1435870375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654015670 con 0x557ebdf3e9d0 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc66bda6740 1 -- 172.21.15.28:0/1435870375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 msgr2=0x557ebdf5bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf5bbb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc65400fbd0 tx=0x7fc654012000 comp rx=0 tx=0).stop 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc66bda6740 1 -- 172.21.15.28:0/1435870375 shutdown_connections 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf45100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3f2f0 0x557ebdf35ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc66bda6740 1 --2- 172.21.15.28:0/1435870375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3e9d0 0x557ebdf5bbb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.346+0000 7fc66bda6740 1 -- 172.21.15.28:0/1435870375 >> 172.21.15.28:0/1435870375 conn(0x557ebdf3af60 msgr2=0x557ebdf467b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 -- 172.21.15.28:0/1435870375 shutdown_connections 2024-09-15T21:56:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 -- 172.21.15.28:0/1435870375 wait complete. 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 -- start start 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 Processor -- start 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf642d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf5d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf5db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf5ba30 con 0x557ebdf3f2f0 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf5b360 con 0x557ebdf3e9d0 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.347+0000 7fc66bda6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557ebdf5c100 con 0x557ebdf49bb0 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf5d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf5d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42220/0 (socket says 172.21.15.28:42220) 2024-09-15T21:56:48.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 -- 172.21.15.28:0/3492185733 learned_addr learned my addr 172.21.15.28:0/3492185733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc66bda6740 10 client.0 _ll_register_callbacks cb 0x7fc64c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf642d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc6637fe640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf5db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 -- 172.21.15.28:0/3492185733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 msgr2=0x557ebdf5db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf5db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 -- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 msgr2=0x557ebdf642d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf642d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 -- 172.21.15.28:0/3492185733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebdf5e0d0 con 0x557ebdf3f2f0 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc65bfff640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf5d650 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0b9150 tx=0x7fc64c03b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb2eb0dd801d6de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc642ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc642ffd640 1 -- 172.21.15.28:0/3492185733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0bc2c0 con 0x557ebdf3f2f0 2024-09-15T21:56:48.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc642ffd640 1 -- 172.21.15.28:0/3492185733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fc64c0c4f60 con 0x557ebdf3f2f0 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc65401c040 con 0x557ebdf3f2f0 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc642ffd640 1 -- 172.21.15.28:0/3492185733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc64c0bce80 con 0x557ebdf3f2f0 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc66bda6740 10 client.15156 fetch_fsmap learned FSMap version 28 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.348+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557ebdf62b60 con 0x557ebdf3f2f0 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc642ffd640 1 -- 172.21.15.28:0/3492185733 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc65401c040 con 0x557ebdf3f2f0 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc642ffd640 1 -- 172.21.15.28:0/3492185733 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc64c0b8040 con 0x557ebdf3f2f0 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc66bda6740 10 client.15156 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc66bda6740 10 client.15156 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc66bda6740 20 client.15156 populate_metadata read hostname 'smithi028' 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdfc94e0 0x557ebdfc81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc66bda6740 4 client.15156 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "status"}) -- 0x557ebdeacb10 con 0x557ebdfc94e0 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.349+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdfc94e0 0x557ebdfc81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc642ffd640 10 client.15156 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc662ffd640 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdfc94e0 0x557ebdfc81f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a31ea69bd626d93 server_cookie=a8a0d7058c88b240 in_seq=0 out_seq=0 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc642ffd640 1 -- 172.21.15.28:0/3492185733 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557ebdeacb10 con 0x557ebdfc94e0 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc642ffd640 10 client.15156 handle_command_reply: tid=0 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 client.15156 shutdown 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 msgr2=0x557ebdf5d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf5d650 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc64c0b9150 tx=0x7fc64c03b8f0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 shutdown_connections 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebdfc94e0 0x557ebdfc81f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebdf49bb0 0x557ebdf5db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebdf3f2f0 0x557ebdf5d650 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 --2- 172.21.15.28:0/3492185733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebdf3e9d0 0x557ebdf642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 >> 172.21.15.28:0/3492185733 conn(0x557ebdf3af60 msgr2=0x557ebdf45080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc642ffd640 0 client.15156 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 shutdown_connections 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 1 -- 172.21.15.28:0/3492185733 wait complete. 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.350+0000 7fc66bda6740 20 client.15156 trim_cache size 0 max 16384 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-15T21:56:48.358 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24485, 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 108.925751346, 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 120.07449142500001 2024-09-15T21:56:48.359 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc6500a3350 msgr2=0x7fc6500a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc6500a3350 0x7fc6500a5810 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x557ebdf3bdc0 tx=0x7fc654008040 comp rx=0 tx=0).stop 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 msgr2=0x7fc6641a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc6641a0940 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc64c004770 tx=0x7fc64c003c10 comp rx=0 tx=0).stop 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 shutdown_connections 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc6500a3350 0x7fc6500a5810 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc66410e480 0x7fc66419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc664101280 0x7fc6641a0940 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 --2- 172.21.15.28:0/3523955283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6641008d0 0x7fc6641a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 >> 172.21.15.28:0/3523955283 conn(0x7fc6640fc7a0 msgr2=0x7fc6640feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:48.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 shutdown_connections 2024-09-15T21:56:48.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:48.351+0000 7fc6699cd640 1 -- 172.21.15.28:0/3523955283 wait complete. 2024-09-15T21:56:49.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:48 smithi028 ceph-mon[30969]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:49.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:48 smithi203 ceph-mon[35546]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:49.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:48 smithi174 ceph-mon[35244]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:49.393 INFO:tasks.cephadm:Setup complete, yielding 2024-09-15T21:56:49.393 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-15T21:56:49.404 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:56:49.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-15T21:56:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 -- 172.21.15.28:0/4276221216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 msgr2=0x7fa694108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 --2- 172.21.15.28:0/4276221216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa694108f20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009ca0 tx=0x7fa68c02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 -- 172.21.15.28:0/4276221216 shutdown_connections 2024-09-15T21:56:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 --2- 172.21.15.28:0/4276221216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa694108f20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 --2- 172.21.15.28:0/4276221216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa694101cd0 0x7fa694102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 --2- 172.21.15.28:0/4276221216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa694100ad0 0x7fa694100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 -- 172.21.15.28:0/4276221216 >> 172.21.15.28:0/4276221216 conn(0x7fa6940fc280 msgr2=0x7fa6940fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 -- 172.21.15.28:0/4276221216 shutdown_connections 2024-09-15T21:56:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.201+0000 7fa69b648640 1 -- 172.21.15.28:0/4276221216 wait complete. 2024-09-15T21:56:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 Processor -- start 2024-09-15T21:56:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 -- start start 2024-09-15T21:56:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa694100ad0 0x7fa69419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa694101cd0 0x7fa69419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa6941a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa69410c730 con 0x7fa694102690 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa69410c5b0 con 0x7fa694101cd0 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa69b648640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa69410c8b0 con 0x7fa694100ad0 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa699be5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa6941a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa699be5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa6941a15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42236/0 (socket says 172.21.15.28:42236) 2024-09-15T21:56:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa699be5640 1 -- 172.21.15.28:0/915044521 learned_addr learned my addr 172.21.15.28:0/915044521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa698be3640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa694101cd0 0x7fa69419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.202+0000 7fa6993e4640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa694100ad0 0x7fa69419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa699be5640 1 -- 172.21.15.28:0/915044521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa694100ad0 msgr2=0x7fa69419a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa699be5640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa694100ad0 0x7fa69419a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa699be5640 1 -- 172.21.15.28:0/915044521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa694101cd0 msgr2=0x7fa69419a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa699be5640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa694101cd0 0x7fa69419a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa699be5640 1 -- 172.21.15.28:0/915044521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6941a1c50 con 0x7fa694102690 2024-09-15T21:56:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa699be5640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa6941a15e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa68c005e30 tx=0x7fa68c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa6867fc640 1 -- 172.21.15.28:0/915044521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c003b70 con 0x7fa694102690 2024-09-15T21:56:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa6867fc640 1 -- 172.21.15.28:0/915044521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa68c003d10 con 0x7fa694102690 2024-09-15T21:56:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6941a1ee0 con 0x7fa694102690 2024-09-15T21:56:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa6867fc640 1 -- 172.21.15.28:0/915044521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c02fd30 con 0x7fa694102690 2024-09-15T21:56:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.203+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6941a23e0 con 0x7fa694102690 2024-09-15T21:56:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.204+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6941a2640 con 0x7fa694102690 2024-09-15T21:56:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.205+0000 7fa6867fc640 1 -- 172.21.15.28:0/915044521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa68c004050 con 0x7fa694102690 2024-09-15T21:56:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.206+0000 7fa6867fc640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa6800a3380 0x7fa6800a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.206+0000 7fa6867fc640 1 -- 172.21.15.28:0/915044521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fa68c0f10b0 con 0x7fa694102690 2024-09-15T21:56:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.206+0000 7fa6993e4640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa6800a3380 0x7fa6800a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.207+0000 7fa6993e4640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa6800a3380 0x7fa6800a5840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa694101b30 tx=0x7fa68800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.209+0000 7fa6867fc640 1 -- 172.21.15.28:0/915044521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa68c0bdae0 con 0x7fa694102690 2024-09-15T21:56:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.401+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fa6941a2810 con 0x7fa6800a3380 2024-09-15T21:56:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.403+0000 7fa6867fc640 1 -- 172.21.15.28:0/915044521 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fa6941a2810 con 0x7fa6800a3380 2024-09-15T21:56:50.405 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-09-15T21:56:50.405 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-09-15T21:56:50.405 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-09-15T21:56:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa6800a3380 msgr2=0x7fa6800a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa6800a3380 0x7fa6800a5840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa694101b30 tx=0x7fa68800b390 comp rx=0 tx=0).stop 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 msgr2=0x7fa6941a15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa6941a15e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa68c005e30 tx=0x7fa68c031ee0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 shutdown_connections 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fa6800a3380 0x7fa6800a5840 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa694102690 0x7fa6941a15e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa694101cd0 0x7fa69419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 --2- 172.21.15.28:0/915044521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa694100ad0 0x7fa69419a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 >> 172.21.15.28:0/915044521 conn(0x7fa6940fc280 msgr2=0x7fa6940fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 shutdown_connections 2024-09-15T21:56:50.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:50.405+0000 7fa69b648640 1 -- 172.21.15.28:0/915044521 wait complete. 2024-09-15T21:56:51.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:50 smithi028 ceph-mon[30969]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:51.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:50 smithi028 ceph-mon[30969]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:51.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-15T21:56:51.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:50 smithi203 ceph-mon[35546]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:51.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:50 smithi203 ceph-mon[35546]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:51.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:50 smithi174 ceph-mon[35244]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-09-15T21:56:51.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:50 smithi174 ceph-mon[35244]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:51.343 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 -- 172.21.15.28:0/1136528382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658104fd0 msgr2=0x7f66581073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 --2- 172.21.15.28:0/1136528382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658104fd0 0x7f66581073c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6648009900 tx=0x7f664802a410 comp rx=0 tx=0).stop 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 -- 172.21.15.28:0/1136528382 shutdown_connections 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 --2- 172.21.15.28:0/1136528382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6658107900 0x7f6658109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 --2- 172.21.15.28:0/1136528382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658104fd0 0x7f66581073c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 --2- 172.21.15.28:0/1136528382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66581026a0 0x7f6658104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 -- 172.21.15.28:0/1136528382 >> 172.21.15.28:0/1136528382 conn(0x7f66580fc2a0 msgr2=0x7f66580fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 -- 172.21.15.28:0/1136528382 shutdown_connections 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 -- 172.21.15.28:0/1136528382 wait complete. 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 Processor -- start 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.836+0000 7f665fed4640 1 -- start start 2024-09-15T21:56:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665fed4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66581026a0 0x7f665819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665fed4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6658104fd0 0x7f665819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665fed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658107900 0x7f66581a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665fed4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f665810d970 con 0x7f6658107900 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665fed4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f665810d7f0 con 0x7f6658104fd0 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665fed4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f665810daf0 con 0x7f66581026a0 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658107900 0x7f66581a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658107900 0x7f66581a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46392/0 (socket says 172.21.15.28:46392) 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 -- 172.21.15.28:0/3717890915 learned_addr learned my addr 172.21.15.28:0/3717890915 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665d46f640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6658104fd0 0x7f665819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665dc70640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66581026a0 0x7f665819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 -- 172.21.15.28:0/3717890915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66581026a0 msgr2=0x7f665819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66581026a0 0x7f665819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 -- 172.21.15.28:0/3717890915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6658104fd0 msgr2=0x7f665819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6658104fd0 0x7f665819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 -- 172.21.15.28:0/3717890915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66581a1ed0 con 0x7f6658107900 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.837+0000 7f665e471640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658107900 0x7f66581a1860 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f665000b970 tx=0x7f665000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.838+0000 7f664effd640 1 -- 172.21.15.28:0/3717890915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650019070 con 0x7f6658107900 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.838+0000 7f664effd640 1 -- 172.21.15.28:0/3717890915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6650014440 con 0x7f6658107900 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.838+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66581a21c0 con 0x7f6658107900 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.838+0000 7f664effd640 1 -- 172.21.15.28:0/3717890915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66500075d0 con 0x7f6658107900 2024-09-15T21:56:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.838+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66581a2750 con 0x7f6658107900 2024-09-15T21:56:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.839+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66581093f0 con 0x7f6658107900 2024-09-15T21:56:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.840+0000 7f664effd640 1 -- 172.21.15.28:0/3717890915 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f665001c070 con 0x7f6658107900 2024-09-15T21:56:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.841+0000 7f664effd640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f66440a3300 0x7f66440a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.841+0000 7f665dc70640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f66440a3300 0x7f66440a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.841+0000 7f664effd640 1 -- 172.21.15.28:0/3717890915 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f66500cf680 con 0x7f6658107900 2024-09-15T21:56:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.841+0000 7f665dc70640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f66440a3300 0x7f66440a57c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6654009990 tx=0x7f6654009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:51.844+0000 7f664effd640 1 -- 172.21.15.28:0/3717890915 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f665009b9a0 con 0x7f6658107900 2024-09-15T21:56:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.031+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f6658062630 con 0x7f66440a3300 2024-09-15T21:56:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.043+0000 7f664effd640 1 -- 172.21.15.28:0/3717890915 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4061 (secure 0 0 0) 0x7f6658062630 con 0x7f66440a3300 2024-09-15T21:56:52.046 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-15T21:56:52.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (2m) 56s ago 2m 15.4M - 19.1.1-326-g8921c8ef f1455bd0219c ea2f62eab5bd 2024-09-15T21:56:52.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi174 running (2m) 57s ago 2m 28.8M - 19.1.1-326-g8921c8ef f1455bd0219c 52e7bc84e2d0 2024-09-15T21:56:52.046 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi203 running (2m) 57s ago 2m 17.6M - 19.1.1-326-g8921c8ef f1455bd0219c 5e9426b12798 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 56s ago 2m 15.9M - 19.1.1-326-g8921c8ef f1455bd0219c 8a8c18c74584 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi174 running (2m) 57s ago 2m 15.9M - 19.1.1-326-g8921c8ef f1455bd0219c 5a4236373597 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi203 running (2m) 57s ago 2m 18.3M - 19.1.1-326-g8921c8ef f1455bd0219c ca2aca01a685 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (2m) 56s ago 2m 15.1M - 19.1.1-326-g8921c8ef f1455bd0219c 41ea9524f670 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi174 running (2m) 57s ago 2m 17.2M - 19.1.1-326-g8921c8ef f1455bd0219c 3fe66e522486 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi203 running (2m) 57s ago 2m 19.5M - 19.1.1-326-g8921c8ef f1455bd0219c 5fa2b0e44dc8 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 56s ago 2m 13.6M - 19.1.1-326-g8921c8ef f1455bd0219c 1d6c8bef2d45 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi174 running (2m) 57s ago 2m 14.7M - 19.1.1-326-g8921c8ef f1455bd0219c 6eb2bd6a32e1 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi203 running (2m) 57s ago 2m 15.8M - 19.1.1-326-g8921c8ef f1455bd0219c 7678aedb8bc2 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (9m) 56s ago 9m 509M - 19.1.1-326-g8921c8ef f1455bd0219c c43e87b526bf 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi174 *:8443,8765 running (7m) 57s ago 7m 443M - 19.1.1-326-g8921c8ef f1455bd0219c 539fe850ed94 2024-09-15T21:56:52.047 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi203 *:8443,8765 running (7m) 57s ago 7m 443M - 19.1.1-326-g8921c8ef f1455bd0219c 26da5c3b0f95 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (9m) 56s ago 9m 50.6M 2048M 19.1.1-326-g8921c8ef f1455bd0219c ac21064d7e14 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi174 running (8m) 57s ago 8m 40.3M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 210357fd4b1e 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi203 running (8m) 57s ago 8m 41.0M 2048M 19.1.1-326-g8921c8ef f1455bd0219c af6e405c8f2f 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 56s ago 7m 49.3M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 72ea0fd64976 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi174 running (7m) 57s ago 7m 46.6M 4272M 19.1.1-326-g8921c8ef f1455bd0219c b9c849c2afe6 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi203 running (6m) 57s ago 6m 47.4M 4272M 19.1.1-326-g8921c8ef f1455bd0219c c6ee242e6dbd 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 56s ago 6m 44.8M 4273M 19.1.1-326-g8921c8ef f1455bd0219c e13beb93320f 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi174 running (5m) 57s ago 5m 42.6M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 5197eab51244 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi203 running (5m) 57s ago 5m 43.7M 4272M 19.1.1-326-g8921c8ef f1455bd0219c bc631adc11eb 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (5m) 56s ago 5m 42.1M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 25b3e578e2e4 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi174 running (4m) 57s ago 4m 52.6M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 6a9447c58340 2024-09-15T21:56:52.048 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi203 running (4m) 57s ago 4m 51.6M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 31fed15665c4 2024-09-15T21:56:52.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 56s ago 3m 50.2M 4273M 19.1.1-326-g8921c8ef f1455bd0219c 331bff8d956a 2024-09-15T21:56:52.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi174 running (3m) 57s ago 3m 44.4M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 925ed1606397 2024-09-15T21:56:52.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi203 running (2m) 57s ago 2m 42.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 9ca081ec77be 2024-09-15T21:56:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f66440a3300 msgr2=0x7f66440a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f66440a3300 0x7f66440a57c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6654009990 tx=0x7f6654009110 comp rx=0 tx=0).stop 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658107900 msgr2=0x7f66581a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658107900 0x7f66581a1860 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f665000b970 tx=0x7f665000bd70 comp rx=0 tx=0).stop 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 shutdown_connections 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f66440a3300 0x7f66440a57c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6658107900 0x7f66581a1860 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6658104fd0 0x7f665819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 --2- 172.21.15.28:0/3717890915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66581026a0 0x7f665819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 >> 172.21.15.28:0/3717890915 conn(0x7f66580fc2a0 msgr2=0x7f6658101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 shutdown_connections 2024-09-15T21:56:52.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:52.046+0000 7f665fed4640 1 -- 172.21.15.28:0/3717890915 wait complete. 2024-09-15T21:56:52.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-15T21:56:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:53.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:52 smithi028 ceph-mon[30969]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:53.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:52 smithi028 ceph-mon[30969]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:53.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:52 smithi203 ceph-mon[35546]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:53.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:52 smithi203 ceph-mon[35546]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:53.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:52 smithi174 ceph-mon[35244]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:53.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:52 smithi174 ceph-mon[35244]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:53.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.561+0000 7f0351b01640 1 -- 172.21.15.28:0/789547820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 msgr2=0x7f034c0ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:53.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.561+0000 7f0351b01640 1 --2- 172.21.15.28:0/789547820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c0ffc90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0334009900 tx=0x7f033402a410 comp rx=0 tx=0).stop 2024-09-15T21:56:53.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 -- 172.21.15.28:0/789547820 shutdown_connections 2024-09-15T21:56:53.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 --2- 172.21.15.28:0/789547820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034c100350 0x7f034c10a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 --2- 172.21.15.28:0/789547820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c0ffc90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 --2- 172.21.15.28:0/789547820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f034c1031f0 0x7f034c1035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 -- 172.21.15.28:0/789547820 >> 172.21.15.28:0/789547820 conn(0x7f034c0fba40 msgr2=0x7f034c0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 -- 172.21.15.28:0/789547820 shutdown_connections 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 -- 172.21.15.28:0/789547820 wait complete. 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 Processor -- start 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 -- start start 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.562+0000 7f0351b01640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034c100350 0x7f034c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0351b01640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f034c1031f0 0x7f034c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0351b01640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0351b01640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f034c10d8a0 con 0x7f034c103bc0 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0351b01640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f034c10d720 con 0x7f034c1031f0 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0351b01640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f034c10da20 con 0x7f034c100350 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c19f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46400/0 (socket says 172.21.15.28:46400) 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 -- 172.21.15.28:0/2354966890 learned_addr learned my addr 172.21.15.28:0/2354966890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034b7fe640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034c100350 0x7f034c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034affd640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f034c1031f0 0x7f034c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 -- 172.21.15.28:0/2354966890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034c100350 msgr2=0x7f034c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034c100350 0x7f034c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 -- 172.21.15.28:0/2354966890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f034c1031f0 msgr2=0x7f034c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f034c1031f0 0x7f034c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 -- 172.21.15.28:0/2354966890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c19fbf0 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034affd640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f034c1031f0 0x7f034c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f034bfff640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c19f580 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f034400cad0 tx=0x7f034400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0348ff9640 1 -- 172.21.15.28:0/2354966890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034400ae20 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0348ff9640 1 -- 172.21.15.28:0/2354966890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0344005020 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c19fee0 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0348ff9640 1 -- 172.21.15.28:0/2354966890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034400f6f0 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.563+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c1a03e0 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.564+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c1091e0 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.566+0000 7f0348ff9640 1 -- 172.21.15.28:0/2354966890 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f03440051d0 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.567+0000 7f0348ff9640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f03380a3380 0x7f03380a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.567+0000 7f034b7fe640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f03380a3380 0x7f03380a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.567+0000 7f0348ff9640 1 -- 172.21.15.28:0/2354966890 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7f03440cfe20 con 0x7f034c103bc0 2024-09-15T21:56:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.567+0000 7f034b7fe640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f03380a3380 0x7f03380a5840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f033c005fd0 tx=0x7f033c00b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.571+0000 7f0348ff9640 1 -- 172.21.15.28:0/2354966890 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0344014030 con 0x7f034c103bc0 2024-09-15T21:56:53.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.759+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f034c062630 con 0x7f03380a3380 2024-09-15T21:56:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.765+0000 7f0348ff9640 1 -- 172.21.15.28:0/2354966890 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f034c062630 con 0x7f03380a3380 2024-09-15T21:56:53.767 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-15T21:56:53.768 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 59s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi174=b;smithi174=e;smithi174=h;smithi174=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-09-15T21:56:53.768 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 59s ago 8m smithi028=x;smithi174=y;smithi203=z;count:3 2024-09-15T21:56:53.768 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 59s ago 8m smithi028:172.21.15.28=a;smithi174:172.21.15.174=b;smithi203:172.21.15.203=c;count:3 2024-09-15T21:56:53.768 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 59s ago - 2024-09-15T21:56:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f03380a3380 msgr2=0x7f03380a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f03380a3380 0x7f03380a5840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f033c005fd0 tx=0x7f033c00b5b0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 msgr2=0x7f034c19f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c19f580 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f034400cad0 tx=0x7f034400cf90 comp rx=0 tx=0).stop 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 shutdown_connections 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f03380a3380 0x7f03380a5840 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c103bc0 0x7f034c19f580 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f034c1031f0 0x7f034c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 --2- 172.21.15.28:0/2354966890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f034c100350 0x7f034c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 >> 172.21.15.28:0/2354966890 conn(0x7f034c0fba40 msgr2=0x7f034c1063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 shutdown_connections 2024-09-15T21:56:53.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:53.767+0000 7f0351b01640 1 -- 172.21.15.28:0/2354966890 wait complete. 2024-09-15T21:56:54.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-15T21:56:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:55.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:54 smithi028 ceph-mon[30969]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:55.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:54 smithi028 ceph-mon[30969]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:55.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:54 smithi203 ceph-mon[35546]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:55.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:54 smithi203 ceph-mon[35546]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:55.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:54 smithi174 ceph-mon[35244]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:55.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:54 smithi174 ceph-mon[35244]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:55.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 -- 172.21.15.28:0/1612664175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64101e50 msgr2=0x7fcf641086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 --2- 172.21.15.28:0/1612664175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64101e50 0x7fcf641086e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009ca0 tx=0x7fcf5c02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 -- 172.21.15.28:0/1612664175 shutdown_connections 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 --2- 172.21.15.28:0/1612664175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64101e50 0x7fcf641086e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 --2- 172.21.15.28:0/1612664175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf64101490 0x7fcf64101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 --2- 172.21.15.28:0/1612664175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf64100290 0x7fcf64100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 -- 172.21.15.28:0/1612664175 >> 172.21.15.28:0/1612664175 conn(0x7fcf640fba40 msgr2=0x7fcf640fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 -- 172.21.15.28:0/1612664175 shutdown_connections 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 -- 172.21.15.28:0/1612664175 wait complete. 2024-09-15T21:56:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 Processor -- start 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 -- start start 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64100290 0x7fcf64195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.291+0000 7fcf6a672640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf64101490 0x7fcf64196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf6a672640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf64101e50 0x7fcf6419d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf6a672640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410b6b0 con 0x7fcf64100290 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf6a672640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410b530 con 0x7fcf64101e50 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf6a672640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410b830 con 0x7fcf64101490 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64100290 0x7fcf64195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64100290 0x7fcf64195d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46418/0 (socket says 172.21.15.28:46418) 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 -- 172.21.15.28:0/4219349981 learned_addr learned my addr 172.21.15.28:0/4219349981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf68c0f640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf64101e50 0x7fcf6419d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf5bfff640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf64101490 0x7fcf64196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:55.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 -- 172.21.15.28:0/4219349981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf64101490 msgr2=0x7fcf64196240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf64101490 0x7fcf64196240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 -- 172.21.15.28:0/4219349981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf64101e50 msgr2=0x7fcf6419d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf64101e50 0x7fcf6419d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 -- 172.21.15.28:0/4219349981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6419d920 con 0x7fcf64100290 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf63fff640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64100290 0x7fcf64195d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400d770 tx=0x7fcf5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf61ffb640 1 -- 172.21.15.28:0/4219349981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54017070 con 0x7fcf64100290 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf61ffb640 1 -- 172.21.15.28:0/4219349981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf54009c30 con 0x7fcf64100290 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6419dc10 con 0x7fcf64100290 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.292+0000 7fcf61ffb640 1 -- 172.21.15.28:0/4219349981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54003dd0 con 0x7fcf64100290 2024-09-15T21:56:55.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.293+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6419e1a0 con 0x7fcf64100290 2024-09-15T21:56:55.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.294+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf64107640 con 0x7fcf64100290 2024-09-15T21:56:55.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.295+0000 7fcf61ffb640 1 -- 172.21.15.28:0/4219349981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fcf54003fb0 con 0x7fcf64100290 2024-09-15T21:56:55.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.295+0000 7fcf61ffb640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcf500a3350 0x7fcf500a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:55.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.296+0000 7fcf5bfff640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcf500a3350 0x7fcf500a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:55.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.296+0000 7fcf61ffb640 1 -- 172.21.15.28:0/4219349981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fcf540cfb20 con 0x7fcf64100290 2024-09-15T21:56:55.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.296+0000 7fcf5bfff640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcf500a3350 0x7fcf500a5810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c005fd0 tx=0x7fcf4c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:55.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.298+0000 7fcf61ffb640 1 -- 172.21.15.28:0/4219349981 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcf54009de0 con 0x7fcf64100290 2024-09-15T21:56:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.480+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fcf64062630 con 0x7fcf500a3350 2024-09-15T21:56:55.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.482+0000 7fcf61ffb640 1 -- 172.21.15.28:0/4219349981 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fcf64062630 con 0x7fcf500a3350 2024-09-15T21:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-09-15T21:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-09-15T21:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout:smithi174 172.21.15.174 2024-09-15T21:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout:smithi203 172.21.15.203 2024-09-15T21:56:55.485 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-09-15T21:56:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcf500a3350 msgr2=0x7fcf500a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcf500a3350 0x7fcf500a5810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c005fd0 tx=0x7fcf4c005850 comp rx=0 tx=0).stop 2024-09-15T21:56:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64100290 msgr2=0x7fcf64195d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64100290 0x7fcf64195d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400d770 tx=0x7fcf5400dc30 comp rx=0 tx=0).stop 2024-09-15T21:56:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 shutdown_connections 2024-09-15T21:56:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcf500a3350 0x7fcf500a5810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf64101e50 0x7fcf6419d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf64101490 0x7fcf64196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 --2- 172.21.15.28:0/4219349981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf64100290 0x7fcf64195d00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 >> 172.21.15.28:0/4219349981 conn(0x7fcf640fba40 msgr2=0x7fcf64104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.484+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 shutdown_connections 2024-09-15T21:56:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:55.485+0000 7fcf6a672640 1 -- 172.21.15.28:0/4219349981 wait complete. 2024-09-15T21:56:56.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:55 smithi028 ceph-mon[30969]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:56.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:55 smithi203 ceph-mon[35546]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:56.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:55 smithi174 ceph-mon[35244]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:56.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-15T21:56:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:56 smithi028 ceph-mon[30969]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:56.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:56 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:56:57.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 -- 172.21.15.28:0/1035920770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec1037d0 msgr2=0x7fcfec10a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:57.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 --2- 172.21.15.28:0/1035920770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec1037d0 0x7fcfec10a060 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009ca0 tx=0x7fcfe402a8c0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 -- 172.21.15.28:0/1035920770 shutdown_connections 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 --2- 172.21.15.28:0/1035920770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec1037d0 0x7fcfec10a060 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 --2- 172.21.15.28:0/1035920770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfec102e10 0x7fcfec103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 --2- 172.21.15.28:0/1035920770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfec101c10 0x7fcfec102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 -- 172.21.15.28:0/1035920770 >> 172.21.15.28:0/1035920770 conn(0x7fcfec0fd3c0 msgr2=0x7fcfec0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 -- 172.21.15.28:0/1035920770 shutdown_connections 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.059+0000 7fcff265c640 1 -- 172.21.15.28:0/1035920770 wait complete. 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.060+0000 7fcff265c640 1 Processor -- start 2024-09-15T21:56:57.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.060+0000 7fcff265c640 1 -- start start 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.060+0000 7fcff265c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfec101c10 0x7fcfec075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.060+0000 7fcff265c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec102e10 0x7fcfec0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.060+0000 7fcff265c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfec1037d0 0x7fcfec074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.060+0000 7fcff265c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10d890 con 0x7fcfec102e10 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfec101c10 0x7fcfec075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec102e10 0x7fcfec0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec102e10 0x7fcfec0745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46428/0 (socket says 172.21.15.28:46428) 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 -- 172.21.15.28:0/3068263634 learned_addr learned my addr 172.21.15.28:0/3068263634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfec101c10 0x7fcfec075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:39022/0 (socket says 172.21.15.28:39022) 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10d710 con 0x7fcfec1037d0 2024-09-15T21:56:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10da10 con 0x7fcfec101c10 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcff0bf9640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfec1037d0 0x7fcfec074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 -- 172.21.15.28:0/3068263634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfec101c10 msgr2=0x7fcfec075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfec101c10 0x7fcfec075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 -- 172.21.15.28:0/3068263634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfec1037d0 msgr2=0x7fcfec074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfec1037d0 0x7fcfec074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 -- 172.21.15.28:0/3068263634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfec0753e0 con 0x7fcfec102e10 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcff0bf9640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfec1037d0 0x7fcfec074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfeb7fe640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec102e10 0x7fcfec0745e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000d950 tx=0x7fcfe000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.061+0000 7fcfe97fa640 1 -- 172.21.15.28:0/3068263634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0017030 con 0x7fcfec102e10 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.062+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfec1a1fe0 con 0x7fcfec102e10 2024-09-15T21:56:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.062+0000 7fcfe97fa640 1 -- 172.21.15.28:0/3068263634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe0004020 con 0x7fcfec102e10 2024-09-15T21:56:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.063+0000 7fcfe97fa640 1 -- 172.21.15.28:0/3068263634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe00047a0 con 0x7fcfec102e10 2024-09-15T21:56:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.063+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfec1a2490 con 0x7fcfec102e10 2024-09-15T21:56:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.064+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfec102010 con 0x7fcfec102e10 2024-09-15T21:56:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.064+0000 7fcfe97fa640 1 -- 172.21.15.28:0/3068263634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fcfe00041d0 con 0x7fcfec102e10 2024-09-15T21:56:57.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.065+0000 7fcfe97fa640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcfd40a31b0 0x7fcfd40a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:57.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.065+0000 7fcfebfff640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcfd40a31b0 0x7fcfd40a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:57.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.065+0000 7fcfe97fa640 1 -- 172.21.15.28:0/3068263634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fcfe00cf0a0 con 0x7fcfec102e10 2024-09-15T21:56:57.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.066+0000 7fcfebfff640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcfd40a31b0 0x7fcfd40a5670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005fd0 tx=0x7fcfdc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:57.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.070+0000 7fcfe97fa640 1 -- 172.21.15.28:0/3068263634 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcfe009b900 con 0x7fcfec102e10 2024-09-15T21:56:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:56 smithi203 ceph-mon[35546]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:56 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:56:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:56 smithi174 ceph-mon[35244]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-15T21:56:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:56 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:56:57.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.307+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fcfec062630 con 0x7fcfd40a31b0 2024-09-15T21:56:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.310+0000 7fcfe97fa640 1 -- 172.21.15.28:0/3068263634 <== mgr.14150 v2:172.21.15.28:6800/1670367809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fcfec062630 con 0x7fcfd40a31b0 2024-09-15T21:56:57.312 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-15T21:56:57.312 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-09-15T21:56:57.312 INFO:teuthology.orchestra.run.smithi028.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-15T21:56:57.312 INFO:teuthology.orchestra.run.smithi028.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-15T21:56:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcfd40a31b0 msgr2=0x7fcfd40a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcfd40a31b0 0x7fcfd40a5670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005fd0 tx=0x7fcfdc00b390 comp rx=0 tx=0).stop 2024-09-15T21:56:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec102e10 msgr2=0x7fcfec0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec102e10 0x7fcfec0745e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000d950 tx=0x7fcfe000de10 comp rx=0 tx=0).stop 2024-09-15T21:56:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 shutdown_connections 2024-09-15T21:56:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcfd40a31b0 0x7fcfd40a5670 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfec1037d0 0x7fcfec074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec102e10 0x7fcfec0745e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 --2- 172.21.15.28:0/3068263634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfec101c10 0x7fcfec075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 >> 172.21.15.28:0/3068263634 conn(0x7fcfec0fd3c0 msgr2=0x7fcfec1061b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 shutdown_connections 2024-09-15T21:56:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:57.312+0000 7fcff265c640 1 -- 172.21.15.28:0/3068263634 wait complete. 2024-09-15T21:56:58.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:57 smithi174 ceph-mon[35244]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:58.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:57 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-09-15T21:56:58.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:57 smithi028 ceph-mon[30969]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:58.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:57 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-09-15T21:56:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:57 smithi203 ceph-mon[35546]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:56:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:57 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-09-15T21:56:58.564 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-15T21:56:58.573 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:56:58.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-15T21:56:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:58 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-09-15T21:56:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T21:56:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T21:56:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:58 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:58 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 -- 172.21.15.28:0/2573803671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8104fd0 msgr2=0x7fdea81073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 --2- 172.21.15.28:0/2573803671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8104fd0 0x7fdea81073c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009900 tx=0x7fde9c02a410 comp rx=0 tx=0).stop 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 -- 172.21.15.28:0/2573803671 shutdown_connections 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 --2- 172.21.15.28:0/2573803671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea8107900 0x7fdea8109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 --2- 172.21.15.28:0/2573803671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8104fd0 0x7fdea81073c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 --2- 172.21.15.28:0/2573803671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdea81026a0 0x7fdea8104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 -- 172.21.15.28:0/2573803671 >> 172.21.15.28:0/2573803671 conn(0x7fdea80fc2a0 msgr2=0x7fdea80fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 -- 172.21.15.28:0/2573803671 shutdown_connections 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 -- 172.21.15.28:0/2573803671 wait complete. 2024-09-15T21:56:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 Processor -- start 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.339+0000 7fdeb0772640 1 -- start start 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeb0772640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea81026a0 0x7fdea8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeb0772640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdea8104fd0 0x7fdea8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeb0772640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea8107900 0x7fdea819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeb0772640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdea810d8f0 con 0x7fdea81026a0 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeb0772640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdea810d770 con 0x7fdea8107900 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeb0772640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdea810da70 con 0x7fdea8104fd0 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea81026a0 0x7fdea8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea81026a0 0x7fdea8197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46444/0 (socket says 172.21.15.28:46444) 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 -- 172.21.15.28:0/102803750 learned_addr learned my addr 172.21.15.28:0/102803750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:56:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeaed0f640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea8107900 0x7fdea819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeadd0d640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdea8104fd0 0x7fdea8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 -- 172.21.15.28:0/102803750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdea8104fd0 msgr2=0x7fdea8198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdea8104fd0 0x7fdea8198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 -- 172.21.15.28:0/102803750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea8107900 msgr2=0x7fdea819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea8107900 0x7fdea819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 -- 172.21.15.28:0/102803750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea819fb80 con 0x7fdea81026a0 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeae50e640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea81026a0 0x7fdea8197f50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fde9800eb90 tx=0x7fde9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdea77fe640 1 -- 172.21.15.28:0/102803750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde98004d20 con 0x7fdea81026a0 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdea77fe640 1 -- 172.21.15.28:0/102803750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9800ee90 con 0x7fdea81026a0 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea819fe70 con 0x7fdea81026a0 2024-09-15T21:56:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.340+0000 7fdea77fe640 1 -- 172.21.15.28:0/102803750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde98010830 con 0x7fdea81026a0 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.341+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea81a02c0 con 0x7fdea81026a0 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.342+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea80fdd90 con 0x7fdea81026a0 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.343+0000 7fdea77fe640 1 -- 172.21.15.28:0/102803750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fde98005020 con 0x7fdea81026a0 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.344+0000 7fdea77fe640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fde900a32f0 0x7fde900a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.344+0000 7fdeadd0d640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fde900a32f0 0x7fde900a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.344+0000 7fdea77fe640 1 -- 172.21.15.28:0/102803750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fde980d02a0 con 0x7fdea81026a0 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.344+0000 7fdeadd0d640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fde900a32f0 0x7fde900a57b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fde9c02aa00 tx=0x7fde9c031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:56:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.348+0000 7fdea77fe640 1 -- 172.21.15.28:0/102803750 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fde9809cc40 con 0x7fdea81026a0 2024-09-15T21:56:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.588+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fdea802d7c0 con 0x7fdea81026a0 2024-09-15T21:56:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.589+0000 7fdea77fe640 1 -- 172.21.15.28:0/102803750 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3138 (secure 0 0 0) 0x7fde9809c540 con 0x7fdea81026a0 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:e28 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:btime 2024-09-15T21:55:38:559165+0000 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 28 2024-09-15T21:56:59.591 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-15T21:54:59.048835+0000 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-09-15T21:55:38.559150+0000 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T21:56:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 5 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2,3,4 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24485,1=24481,2=24460,3=24458,4=24454} 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3,4] 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24485 members: 24485,24481,24460,24458,24454 2024-09-15T21:56:59.593 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{0:24485} state up:active seq 4 addr [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{0:24443} state up:standby-replay seq 1 addr [v2:172.21.15.174:6834/280309786,v1:172.21.15.174:6835/280309786] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{1:24481} state up:active seq 15 addr [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{1:24425} state up:standby-replay seq 1 addr [v2:172.21.15.174:6832/1988552520,v1:172.21.15.174:6833/1988552520] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{2:24460} state up:active seq 17 addr [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{2:24424} state up:standby-replay seq 1 addr [v2:172.21.15.203:6832/2800102614,v1:172.21.15.203:6833/2800102614] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{3:24458} state up:active seq 16 addr [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{3:14628} state up:standby-replay seq 1 addr [v2:172.21.15.28:6840/2641693590,v1:172.21.15.28:6841/2641693590] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{4:24454} state up:active seq 18 addr [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{4:14622} state up:standby-replay seq 1 addr [v2:172.21.15.28:6838/3176985638,v1:172.21.15.28:6839/3176985638] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:59.594 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-09-15T21:56:59.595 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:56:59.595 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.28:6834/2109233987,v1:172.21.15.28:6835/2109233987] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.595 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.28:6836/2070322956,v1:172.21.15.28:6837/2070322956] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T21:56:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-09-15T21:56:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fde900a32f0 msgr2=0x7fde900a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fde900a32f0 0x7fde900a57b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fde9c02aa00 tx=0x7fde9c031400 comp rx=0 tx=0).stop 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea81026a0 msgr2=0x7fdea8197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea81026a0 0x7fdea8197f50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fde9800eb90 tx=0x7fde9800c6a0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 shutdown_connections 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fde900a32f0 0x7fde900a57b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdea8107900 0x7fdea819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdea8104fd0 0x7fdea8198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 --2- 172.21.15.28:0/102803750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea81026a0 0x7fdea8197f50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 >> 172.21.15.28:0/102803750 conn(0x7fdea80fc2a0 msgr2=0x7fdea8103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 shutdown_connections 2024-09-15T21:56:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:56:59.592+0000 7fdeb0772640 1 -- 172.21.15.28:0/102803750 wait complete. 2024-09-15T21:57:00.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:59 smithi028 ceph-mon[30969]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T21:57:00.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:00.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:59 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:00.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:56:59 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/102803750' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T21:57:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:59 smithi203 ceph-mon[35546]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T21:57:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:59 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:56:59 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/102803750' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T21:57:00.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-15T21:57:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:59 smithi174 ceph-mon[35244]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T21:57:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:59 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:56:59 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/102803750' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T21:57:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:00.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 -- 172.21.15.28:0/740131090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca8103bc0 msgr2=0x7fcca80ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 --2- 172.21.15.28:0/740131090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca8103bc0 0x7fcca80ffc90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009900 tx=0x7fcc9c02a410 comp rx=0 tx=0).stop 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 -- 172.21.15.28:0/740131090 shutdown_connections 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 --2- 172.21.15.28:0/740131090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcca8100350 0x7fcca810a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 --2- 172.21.15.28:0/740131090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca8103bc0 0x7fcca80ffc90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 --2- 172.21.15.28:0/740131090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcca81031f0 0x7fcca81035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 -- 172.21.15.28:0/740131090 >> 172.21.15.28:0/740131090 conn(0x7fcca80fba40 msgr2=0x7fcca80fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 -- 172.21.15.28:0/740131090 shutdown_connections 2024-09-15T21:57:00.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.968+0000 7fccadc42640 1 -- 172.21.15.28:0/740131090 wait complete. 2024-09-15T21:57:00.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 Processor -- start 2024-09-15T21:57:00.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 -- start start 2024-09-15T21:57:00.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcca8100350 0x7fcca8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:00.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca81031f0 0x7fcca8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:00.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcca8103bc0 0x7fcca819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:00.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcca810d8a0 con 0x7fcca81031f0 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcca810d720 con 0x7fcca8103bc0 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fccadc42640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcca810da20 con 0x7fcca8100350 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fcca6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca81031f0 0x7fcca8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fcca6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca81031f0 0x7fcca8198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39972/0 (socket says 172.21.15.28:39972) 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fcca6ffd640 1 -- 172.21.15.28:0/937964766 learned_addr learned my addr 172.21.15.28:0/937964766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fcca7fff640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcca8103bc0 0x7fcca819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fcca77fe640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcca8100350 0x7fcca8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.969+0000 7fcca6ffd640 1 -- 172.21.15.28:0/937964766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcca8100350 msgr2=0x7fcca8197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca6ffd640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcca8100350 0x7fcca8197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca6ffd640 1 -- 172.21.15.28:0/937964766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcca8103bc0 msgr2=0x7fcca819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca6ffd640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcca8103bc0 0x7fcca819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca6ffd640 1 -- 172.21.15.28:0/937964766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca819fbf0 con 0x7fcca81031f0 2024-09-15T21:57:00.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca7fff640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcca8103bc0 0x7fcca819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:00.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca6ffd640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca81031f0 0x7fcca8198500 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005ab0 tx=0x7fcc9c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:00.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca4ff9640 1 -- 172.21.15.28:0/937964766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c03f030 con 0x7fcca81031f0 2024-09-15T21:57:00.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca819fe80 con 0x7fcca81031f0 2024-09-15T21:57:00.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca4ff9640 1 -- 172.21.15.28:0/937964766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc9c003ce0 con 0x7fcca81031f0 2024-09-15T21:57:00.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fcca4ff9640 1 -- 172.21.15.28:0/937964766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c035070 con 0x7fcca81031f0 2024-09-15T21:57:00.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.970+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca81a02e0 con 0x7fcca81031f0 2024-09-15T21:57:00.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.971+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca81091e0 con 0x7fcca81031f0 2024-09-15T21:57:00.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.972+0000 7fcca4ff9640 1 -- 172.21.15.28:0/937964766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fcc9c042030 con 0x7fcca81031f0 2024-09-15T21:57:00.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.973+0000 7fcca4ff9640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcc900a32f0 0x7fcc900a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:00.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.973+0000 7fcca77fe640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcc900a32f0 0x7fcc900a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:00.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.973+0000 7fcca4ff9640 1 -- 172.21.15.28:0/937964766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fcc9c0effc0 con 0x7fcca81031f0 2024-09-15T21:57:00.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.974+0000 7fcca77fe640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcc900a32f0 0x7fcc900a57b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcc98005db0 tx=0x7fcc98005d40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:00.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:00.977+0000 7fcca4ff9640 1 -- 172.21.15.28:0/937964766 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcc9c0bc900 con 0x7fcca81031f0 2024-09-15T21:57:01.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.152+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fcca8062e90 con 0x7fcca81031f0 2024-09-15T21:57:01.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:00 smithi028 ceph-mon[30969]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-15T21:57:01.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:01.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:01.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:00 smithi028 ceph-mon[30969]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:01.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.153+0000 7fcca4ff9640 1 -- 172.21.15.28:0/937964766 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v92) ==== 56+0+5616 (secure 0 0 0) 0x7fcc9c0bc200 con 0x7fcca81031f0 2024-09-15T21:57:01.154 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 92 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:fsid e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-15T21:47:10.448749+0000 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-09-15T21:56:58.911816+0000 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 27 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-09-15T21:57:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/896252218,v1:172.21.15.28:6803/896252218] [v2:172.21.15.28:6804/896252218,v1:172.21.15.28:6805/896252218] exists,up c5066bae-2a3f-4b91-9da9-39d341d3257e 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/4110616785,v1:172.21.15.174:6801/4110616785] [v2:172.21.15.174:6802/4110616785,v1:172.21.15.174:6803/4110616785] exists,up b85e5570-ddd1-4e02-858e-948a4517801e 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/2523083594,v1:172.21.15.203:6801/2523083594] [v2:172.21.15.203:6802/2523083594,v1:172.21.15.203:6803/2523083594] exists,up 41e4d0e4-7b72-4ad8-9c2e-a260d638777f 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/1752867147,v1:172.21.15.28:6811/1752867147] [v2:172.21.15.28:6812/1752867147,v1:172.21.15.28:6813/1752867147] exists,up bd998a1a-b9b4-4950-8925-5b3d2da0e73c 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 32 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/3085836897,v1:172.21.15.174:6809/3085836897] [v2:172.21.15.174:6810/3085836897,v1:172.21.15.174:6811/3085836897] exists,up 7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 37 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/3676469409,v1:172.21.15.203:6809/3676469409] [v2:172.21.15.203:6810/3676469409,v1:172.21.15.203:6811/3676469409] exists,up 40516595-c52a-4503-b534-2b5c855d431e 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 44 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/908626673,v1:172.21.15.28:6819/908626673] [v2:172.21.15.28:6820/908626673,v1:172.21.15.28:6821/908626673] exists,up 4db3191e-2eb9-4ce3-b007-11d4305d0c91 2024-09-15T21:57:01.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 49 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/1679850033,v1:172.21.15.174:6817/1679850033] [v2:172.21.15.174:6818/1679850033,v1:172.21.15.174:6819/1679850033] exists,up c6259a57-5604-481c-b739-f187239955d0 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 56 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/3629574,v1:172.21.15.203:6817/3629574] [v2:172.21.15.203:6818/3629574,v1:172.21.15.203:6819/3629574] exists,up e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 63 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/306652678,v1:172.21.15.28:6827/306652678] [v2:172.21.15.28:6828/306652678,v1:172.21.15.28:6829/306652678] exists,up fbe08e2a-856e-4836-bfa3-fa269e447a9f 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 68 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6824/2504623164,v1:172.21.15.174:6825/2504623164] [v2:172.21.15.174:6826/2504623164,v1:172.21.15.174:6827/2504623164] exists,up ff5afcc0-0e93-459e-87a8-1a34428aef91 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 76 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/2779316485,v1:172.21.15.203:6825/2779316485] [v2:172.21.15.203:6826/2779316485,v1:172.21.15.203:6827/2779316485] exists,up 3393ee20-fde1-48f1-b9f8-407c434f1386 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.a [2,11] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1c [8,11] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1d [2,4] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1b [4,8] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.21 [5,9] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.2c [4,9] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.34 [4,9] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.5 [9,10] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.7 [5,8] 2024-09-15T21:57:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.c [9,10] 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.14 [9,8] 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/671259623 expires 2024-09-16T21:47:57.453457+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/711457183 expires 2024-09-16T21:47:57.453457+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1395107553 expires 2024-09-16T21:47:57.453457+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2625692755 expires 2024-09-16T21:47:57.453457+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2625692755 expires 2024-09-16T21:47:57.453457+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/123978199 expires 2024-09-16T21:47:32.835958+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2709495038 expires 2024-09-16T21:47:32.835958+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2627662465 expires 2024-09-16T21:47:32.835958+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/3052473616 expires 2024-09-16T21:47:32.835958+0000 2024-09-15T21:57:01.158 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3052473616 expires 2024-09-16T21:47:32.835958+0000 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcc900a32f0 msgr2=0x7fcc900a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcc900a32f0 0x7fcc900a57b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcc98005db0 tx=0x7fcc98005d40 comp rx=0 tx=0).stop 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca81031f0 msgr2=0x7fcca8198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca81031f0 0x7fcca8198500 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c005ab0 tx=0x7fcc9c004360 comp rx=0 tx=0).stop 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 shutdown_connections 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fcc900a32f0 0x7fcc900a57b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcca8103bc0 0x7fcca819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca81031f0 0x7fcca8198500 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 --2- 172.21.15.28:0/937964766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcca8100350 0x7fcca8197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 >> 172.21.15.28:0/937964766 conn(0x7fcca80fba40 msgr2=0x7fcca81063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 shutdown_connections 2024-09-15T21:57:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:01.155+0000 7fccadc42640 1 -- 172.21.15.28:0/937964766 wait complete. 2024-09-15T21:57:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:00 smithi203 ceph-mon[35546]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-15T21:57:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:00 smithi203 ceph-mon[35546]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:01.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:00 smithi174 ceph-mon[35244]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-15T21:57:01.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:01.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:01.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:00 smithi174 ceph-mon[35244]: from='mgr.14150 172.21.15.28:0/3070225139' entity='mgr.x' 2024-09-15T21:57:01.858 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-15T21:57:01.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:57:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:02.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:01 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/937964766' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T21:57:02.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:01 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/937964766' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T21:57:02.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:01 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/937964766' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.597+0000 7f908cf37640 1 -- 172.21.15.28:0/595348630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088106a80 msgr2=0x7f9088106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.597+0000 7f908cf37640 1 --2- 172.21.15.28:0/595348630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088106a80 0x7f9088106e60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9078009930 tx=0x7f907802a420 comp rx=0 tx=0).stop 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.597+0000 7f908cf37640 1 -- 172.21.15.28:0/595348630 shutdown_connections 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.597+0000 7f908cf37640 1 --2- 172.21.15.28:0/595348630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088101420 0x7f908810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.597+0000 7f908cf37640 1 --2- 172.21.15.28:0/595348630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9088100a80 0x7f9088100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.597+0000 7f908cf37640 1 --2- 172.21.15.28:0/595348630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088106a80 0x7f9088106e60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.597+0000 7f908cf37640 1 -- 172.21.15.28:0/595348630 >> 172.21.15.28:0/595348630 conn(0x7f90880fc7a0 msgr2=0x7f90880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.598+0000 7f908cf37640 1 -- 172.21.15.28:0/595348630 shutdown_connections 2024-09-15T21:57:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.598+0000 7f908cf37640 1 -- 172.21.15.28:0/595348630 wait complete. 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.598+0000 7f908cf37640 1 Processor -- start 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.598+0000 7f908cf37640 1 -- start start 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f908cf37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088100a80 0x7f9088198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f908cf37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088101420 0x7f90881989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f908cf37640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9088106a80 0x7f908819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f908cf37640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9088110e90 con 0x7f9088101420 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f908cf37640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9088110d10 con 0x7f9088106a80 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f908cf37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9088111010 con 0x7f9088100a80 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088101420 0x7f90881989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088101420 0x7f90881989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39990/0 (socket says 172.21.15.28:39990) 2024-09-15T21:57:02.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 -- 172.21.15.28:0/3985492766 learned_addr learned my addr 172.21.15.28:0/3985492766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f908659c640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088100a80 0x7f9088198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9086d9d640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9088106a80 0x7f908819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 -- 172.21.15.28:0/3985492766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088100a80 msgr2=0x7f9088198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088100a80 0x7f9088198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 -- 172.21.15.28:0/3985492766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9088106a80 msgr2=0x7f908819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9088106a80 0x7f908819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 -- 172.21.15.28:0/3985492766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f908819d410 con 0x7f9088101420 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.599+0000 7f9085d9b640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088101420 0x7f90881989a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f907c00d950 tx=0x7f907c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.600+0000 7f90777fe640 1 -- 172.21.15.28:0/3985492766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c017070 con 0x7f9088101420 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.600+0000 7f90777fe640 1 -- 172.21.15.28:0/3985492766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f907c009c30 con 0x7f9088101420 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.600+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f908819d700 con 0x7f9088101420 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.600+0000 7f90777fe640 1 -- 172.21.15.28:0/3985492766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c00f6b0 con 0x7f9088101420 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.600+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90881a4fe0 con 0x7f9088101420 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.601+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9088100ee0 con 0x7f9088101420 2024-09-15T21:57:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.602+0000 7f90777fe640 1 -- 172.21.15.28:0/3985492766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f907c0109d0 con 0x7f9088101420 2024-09-15T21:57:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.603+0000 7f90777fe640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90700a3240 0x7f90700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.603+0000 7f90777fe640 1 -- 172.21.15.28:0/3985492766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f907c0cf920 con 0x7f9088101420 2024-09-15T21:57:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.603+0000 7f908659c640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90700a3240 0x7f90700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.603+0000 7f908659c640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90700a3240 0x7f90700a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9078006010 tx=0x7f9078000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.607+0000 7f90777fe640 1 -- 172.21.15.28:0/3985492766 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f907c009de0 con 0x7f9088101420 2024-09-15T21:57:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.818+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f908810a060 con 0x7f9088101420 2024-09-15T21:57:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.819+0000 7f90777fe640 1 -- 172.21.15.28:0/3985492766 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f907c09c2c0 con 0x7f9088101420 2024-09-15T21:57:02.821 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:02.821 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"btime":"2024-09-15T21:55:38:559165+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:38.559150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:57:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-09-15T21:57:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90700a3240 msgr2=0x7f90700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90700a3240 0x7f90700a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9078006010 tx=0x7f9078000bc0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088101420 msgr2=0x7f90881989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088101420 0x7f90881989a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f907c00d950 tx=0x7f907c00de10 comp rx=0 tx=0).stop 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 shutdown_connections 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90700a3240 0x7f90700a5700 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9088106a80 0x7f908819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9088101420 0x7f90881989a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 --2- 172.21.15.28:0/3985492766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9088100a80 0x7f9088198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 >> 172.21.15.28:0/3985492766 conn(0x7f90880fc7a0 msgr2=0x7f908806aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 shutdown_connections 2024-09-15T21:57:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:02.822+0000 7f908cf37640 1 -- 172.21.15.28:0/3985492766 wait complete. 2024-09-15T21:57:03.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:02 smithi028 ceph-mon[30969]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s 2024-09-15T21:57:03.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:02 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3985492766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:03.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:02 smithi203 ceph-mon[35546]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s 2024-09-15T21:57:03.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:02 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3985492766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:03.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:02 smithi174 ceph-mon[35244]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s 2024-09-15T21:57:03.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:02 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3985492766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:03.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:57:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 -- 172.21.15.28:0/4175799185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c102ef0 msgr2=0x7f910c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 --2- 172.21.15.28:0/4175799185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c102ef0 0x7f910c10da40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9104009ca0 tx=0x7f910402a950 comp rx=0 tx=0).stop 2024-09-15T21:57:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 -- 172.21.15.28:0/4175799185 shutdown_connections 2024-09-15T21:57:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 --2- 172.21.15.28:0/4175799185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c102ef0 0x7f910c10da40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 --2- 172.21.15.28:0/4175799185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f910c102570 0x7f910c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 --2- 172.21.15.28:0/4175799185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f910c100e20 0x7f910c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 -- 172.21.15.28:0/4175799185 >> 172.21.15.28:0/4175799185 conn(0x7f910c0fc7d0 msgr2=0x7f910c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 -- 172.21.15.28:0/4175799185 shutdown_connections 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 -- 172.21.15.28:0/4175799185 wait complete. 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.228+0000 7f911184c640 1 Processor -- start 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f911184c640 1 -- start start 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f911184c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c100e20 0x7f910c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f911184c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f910c102570 0x7f910c198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f911184c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f910c102ef0 0x7f910c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f911184c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f910c110f80 con 0x7f910c100e20 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f911184c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f910c110e00 con 0x7f910c102570 2024-09-15T21:57:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f911184c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f910c111100 con 0x7f910c102ef0 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c100e20 0x7f910c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c100e20 0x7f910c1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40004/0 (socket says 172.21.15.28:40004) 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 -- 172.21.15.28:0/675265306 learned_addr learned my addr 172.21.15.28:0/675265306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910a7fc640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f910c102570 0x7f910c198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910b7fe640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f910c102ef0 0x7f910c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 -- 172.21.15.28:0/675265306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f910c102ef0 msgr2=0x7f910c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f910c102ef0 0x7f910c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 -- 172.21.15.28:0/675265306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f910c102570 msgr2=0x7f910c198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f910c102570 0x7f910c198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910affd640 1 -- 172.21.15.28:0/675265306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910c19d490 con 0x7f910c100e20 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.229+0000 7f910b7fe640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f910c102ef0 0x7f910c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.230+0000 7f910affd640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c100e20 0x7f910c1984e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00d770 tx=0x7f90fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.230+0000 7f911084a640 1 -- 172.21.15.28:0/675265306 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc017070 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.230+0000 7f911084a640 1 -- 172.21.15.28:0/675265306 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90fc005500 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.230+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910c19d780 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.230+0000 7f911084a640 1 -- 172.21.15.28:0/675265306 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc004e00 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.230+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f910c1a5060 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.231+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910c101200 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.232+0000 7f911084a640 1 -- 172.21.15.28:0/675265306 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f90fc004fa0 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.233+0000 7f911084a640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90f40a32c0 0x7f90f40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.233+0000 7f910a7fc640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90f40a32c0 0x7f90f40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.233+0000 7f911084a640 1 -- 172.21.15.28:0/675265306 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f90fc0cfad0 con 0x7f910c100e20 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.233+0000 7f910a7fc640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90f40a32c0 0x7f90f40a5780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f910c1052b0 tx=0x7f910000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.236+0000 7f911084a640 1 -- 172.21.15.28:0/675265306 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f90fc09c4f0 con 0x7f910c100e20 2024-09-15T21:57:04.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.449+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f910c199840 con 0x7f910c100e20 2024-09-15T21:57:04.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.449+0000 7f911084a640 1 -- 172.21.15.28:0/675265306 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f90fc09bdf0 con 0x7f910c100e20 2024-09-15T21:57:04.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:04.452 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"btime":"2024-09-15T21:55:38:559165+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:55:38.559150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:57:04.452 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90f40a32c0 msgr2=0x7f90f40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90f40a32c0 0x7f90f40a5780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f910c1052b0 tx=0x7f910000b410 comp rx=0 tx=0).stop 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c100e20 msgr2=0x7f910c1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c100e20 0x7f910c1984e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00d770 tx=0x7f90fc00dc30 comp rx=0 tx=0).stop 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 shutdown_connections 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f90f40a32c0 0x7f90f40a5780 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f910c102ef0 0x7f910c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f910c102570 0x7f910c198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 --2- 172.21.15.28:0/675265306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f910c100e20 0x7f910c1984e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 >> 172.21.15.28:0/675265306 conn(0x7f910c0fc7d0 msgr2=0x7f910c06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 shutdown_connections 2024-09-15T21:57:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:04.452+0000 7f911184c640 1 -- 172.21.15.28:0/675265306 wait complete. 2024-09-15T21:57:05.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:04 smithi028 ceph-mon[30969]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:05.118 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:04 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/675265306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:05.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:57:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:04 smithi203 ceph-mon[35546]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:04 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/675265306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:04 smithi174 ceph-mon[35244]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:04 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/675265306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 -- 172.21.15.28:0/1505160873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118102ef0 msgr2=0x7f211810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 --2- 172.21.15.28:0/1505160873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118102ef0 0x7f211810da40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2108009ca0 tx=0x7f210802a950 comp rx=0 tx=0).stop 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 -- 172.21.15.28:0/1505160873 shutdown_connections 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 --2- 172.21.15.28:0/1505160873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118102ef0 0x7f211810da40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 --2- 172.21.15.28:0/1505160873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2118102570 0x7f21181029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 --2- 172.21.15.28:0/1505160873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2118100e20 0x7f2118101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 -- 172.21.15.28:0/1505160873 >> 172.21.15.28:0/1505160873 conn(0x7f21180fc7d0 msgr2=0x7f21180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 -- 172.21.15.28:0/1505160873 shutdown_connections 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.900+0000 7f211fc63640 1 -- 172.21.15.28:0/1505160873 wait complete. 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 Processor -- start 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 -- start start 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118100e20 0x7f21181985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2118102570 0x7f2118198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2118102ef0 0x7f211819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2118112120 con 0x7f2118100e20 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2118111fa0 con 0x7f2118102ef0 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211fc63640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21181122a0 con 0x7f2118102570 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211d9ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118100e20 0x7f21181985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211d9ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118100e20 0x7f21181985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40034/0 (socket says 172.21.15.28:40034) 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.901+0000 7f211d9ff640 1 -- 172.21.15.28:0/186538905 learned_addr learned my addr 172.21.15.28:0/186538905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d1fe640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2118102570 0x7f2118198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d9ff640 1 -- 172.21.15.28:0/186538905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2118102570 msgr2=0x7f2118198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211e200640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2118102ef0 0x7f211819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d9ff640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2118102570 0x7f2118198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d9ff640 1 -- 172.21.15.28:0/186538905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2118102ef0 msgr2=0x7f211819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d9ff640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2118102ef0 0x7f211819ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d9ff640 1 -- 172.21.15.28:0/186538905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211819d590 con 0x7f2118100e20 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d1fe640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2118102570 0x7f2118198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211d9ff640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118100e20 0x7f21181985e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f211000b4a0 tx=0x7f211000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f2116ffd640 1 -- 172.21.15.28:0/186538905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110014070 con 0x7f2118100e20 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f211819d880 con 0x7f2118100e20 2024-09-15T21:57:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f2116ffd640 1 -- 172.21.15.28:0/186538905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2110010af0 con 0x7f2118100e20 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f2116ffd640 1 -- 172.21.15.28:0/186538905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110017d10 con 0x7f2118100e20 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.902+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21181a5160 con 0x7f2118100e20 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.903+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2118101280 con 0x7f2118100e20 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.904+0000 7f2116ffd640 1 -- 172.21.15.28:0/186538905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2110017070 con 0x7f2118100e20 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.905+0000 7f2116ffd640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f21000a3240 0x7f21000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.905+0000 7f211d1fe640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f21000a3240 0x7f21000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.905+0000 7f2116ffd640 1 -- 172.21.15.28:0/186538905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f21100cf170 con 0x7f2118100e20 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.906+0000 7f211d1fe640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f21000a3240 0x7f21000a5700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f21180ffe50 tx=0x7f210c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:05.909+0000 7f2116ffd640 1 -- 172.21.15.28:0/186538905 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f211009bab0 con 0x7f2118100e20 2024-09-15T21:57:06.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.090+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2118109090 con 0x7f2118100e20 2024-09-15T21:57:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.092+0000 7f2116ffd640 1 -- 172.21.15.28:0/186538905 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24762 (secure 0 0 0) 0x7f211009b3b0 con 0x7f2118100e20 2024-09-15T21:57:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:06.094 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:56:58.911816+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:57:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f21000a3240 msgr2=0x7f21000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f21000a3240 0x7f21000a5700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f21180ffe50 tx=0x7f210c008040 comp rx=0 tx=0).stop 2024-09-15T21:57:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118100e20 msgr2=0x7f21181985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118100e20 0x7f21181985e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f211000b4a0 tx=0x7f211000b960 comp rx=0 tx=0).stop 2024-09-15T21:57:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 shutdown_connections 2024-09-15T21:57:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f21000a3240 0x7f21000a5700 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2118102ef0 0x7f211819ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2118102570 0x7f2118198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 --2- 172.21.15.28:0/186538905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2118100e20 0x7f21181985e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 >> 172.21.15.28:0/186538905 conn(0x7f21180fc7d0 msgr2=0x7f21180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 shutdown_connections 2024-09-15T21:57:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:06.094+0000 7f211fc63640 1 -- 172.21.15.28:0/186538905 wait complete. 2024-09-15T21:57:06.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T21:54:59.048835+0000', 'modified': '2024-09-15T21:55:38.559150+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24485, 'mds_1': 24481, 'mds_2': 24460, 'mds_3': 24458, 'mds_4': 24454}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/3176985638', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 3176985638}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 3176985638}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2641693590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2641693590}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2641693590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/2800102614', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 2800102614}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 2800102614}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6833/1988552520', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 1988552520}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 1988552520}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/280309786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 280309786}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 280309786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.203:6835/1524895518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 1524895518}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 1524895518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.174:6837/3327498469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3327498469}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3327498469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.203:6837/803844131', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 803844131}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 803844131}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.203:6839/3242294678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 3242294678}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 3242294678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.174:6839/3735159621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3735159621}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3735159621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24485, 24481, 24460, 24458, 24454]} 2024-09-15T21:57:06.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-15T21:57:06.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-15T21:57:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:07.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:06 smithi028 ceph-mon[30969]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:07.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:06 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/186538905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:07.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:06 smithi203 ceph-mon[35546]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:07.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:06 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/186538905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:07.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:06 smithi174 ceph-mon[35244]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:07.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:06 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/186538905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.475+0000 7f64f6a14640 1 -- 172.21.15.28:0/2825249041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 msgr2=0x7f64f01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.475+0000 7f64f6a14640 1 --2- 172.21.15.28:0/2825249041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01067e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f64e4009900 tx=0x7f64e402a410 comp rx=0 tx=0).stop 2024-09-15T21:57:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.475+0000 7f64f6a14640 1 -- 172.21.15.28:0/2825249041 shutdown_connections 2024-09-15T21:57:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.475+0000 7f64f6a14640 1 --2- 172.21.15.28:0/2825249041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64f0106f10 0x7f64f010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.475+0000 7f64f6a14640 1 --2- 172.21.15.28:0/2825249041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01067e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.475+0000 7f64f6a14640 1 --2- 172.21.15.28:0/2825249041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64f0101de0 0x7f64f01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.475+0000 7f64f6a14640 1 -- 172.21.15.28:0/2825249041 >> 172.21.15.28:0/2825249041 conn(0x7f64f00fd8c0 msgr2=0x7f64f00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.476+0000 7f64f6a14640 1 -- 172.21.15.28:0/2825249041 shutdown_connections 2024-09-15T21:57:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.476+0000 7f64f6a14640 1 -- 172.21.15.28:0/2825249041 wait complete. 2024-09-15T21:57:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.476+0000 7f64f6a14640 1 Processor -- start 2024-09-15T21:57:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.476+0000 7f64f6a14640 1 -- start start 2024-09-15T21:57:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.476+0000 7f64f6a14640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64f0101de0 0x7f64f01983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.476+0000 7f64f6a14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64f6a14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64f0106f10 0x7f64f019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64f6a14640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64f0111f30 con 0x7f64f0102790 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64f6a14640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f64f0111db0 con 0x7f64f0101de0 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64f6a14640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64f01120b0 con 0x7f64f0106f10 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40048/0 (socket says 172.21.15.28:40048) 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 -- 172.21.15.28:0/799683632 learned_addr learned my addr 172.21.15.28:0/799683632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64f0106f10 0x7f64f019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64effff640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64f0101de0 0x7f64f01983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 -- 172.21.15.28:0/799683632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64f0106f10 msgr2=0x7f64f019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64f0106f10 0x7f64f019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 -- 172.21.15.28:0/799683632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64f0101de0 msgr2=0x7f64f01983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64f0101de0 0x7f64f01983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 -- 172.21.15.28:0/799683632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f019d350 con 0x7f64f0102790 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64f0106f10 0x7f64f019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64effff640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64f0101de0 0x7f64f01983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01988e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f64e4006010 tx=0x7f64e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ed7fa640 1 -- 172.21.15.28:0/799683632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e403f030 con 0x7f64f0102790 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f019d5e0 con 0x7f64f0102790 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ed7fa640 1 -- 172.21.15.28:0/799683632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e4003ce0 con 0x7f64f0102790 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.477+0000 7f64ed7fa640 1 -- 172.21.15.28:0/799683632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4035070 con 0x7f64f0102790 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.478+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f01a4f20 con 0x7f64f0102790 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64ed7fa640 1 -- 172.21.15.28:0/799683632 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f64e4042030 con 0x7f64f0102790 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 Processor -- start 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 -- start start 2024-09-15T21:57:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a51b0 0x55723f7b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55723f70b090 con 0x55723f7a51b0 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55723f6b2a70 con 0x55723f7a4890 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64f8ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55723f707050 con 0x55723f7b3fc0 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a51b0 0x55723f7b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64ed7fa640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f64d80a3240 0x7f64d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a51b0 0x55723f7b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40054/0 (socket says 172.21.15.28:40054) 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.480+0000 7f64effff640 1 -- 172.21.15.28:0/4267320460 learned_addr learned my addr 172.21.15.28:0/4267320460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f64d80a3240 0x7f64d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64ed7fa640 1 -- 172.21.15.28:0/799683632 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f64e40f0360 con 0x7f64f0102790 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64effff640 1 -- 172.21.15.28:0/4267320460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 msgr2=0x55723f7b63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64effff640 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64effff640 1 -- 172.21.15.28:0/4267320460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 msgr2=0x55723f7a4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64effff640 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7a4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64effff640 1 -- 172.21.15.28:0/4267320460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55723f6bf130 con 0x55723f7a51b0 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64effff640 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a51b0 0x55723f7b3a80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f64e0008c60 tx=0x7f64e000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b2036b51360e11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f64d80a3240 0x7f64d80a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f64e8005c40 tx=0x7f64e8009280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64d67fc640 1 -- 172.21.15.28:0/4267320460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0014070 con 0x55723f7a51b0 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64d67fc640 1 -- 172.21.15.28:0/4267320460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e00047a0 con 0x55723f7a51b0 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.481+0000 7f64d67fc640 1 -- 172.21.15.28:0/4267320460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0010740 con 0x55723f7a51b0 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 -- 172.21.15.28:0/4267320460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a51b0 msgr2=0x55723f7b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a51b0 0x55723f7b3a80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f64e0008c60 tx=0x7f64e000e720 comp rx=0 tx=0).stop 2024-09-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 -- 172.21.15.28:0/4267320460 shutdown_connections 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a51b0 0x55723f7b3a80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4267320460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7a4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 -- 172.21.15.28:0/4267320460 >> 172.21.15.28:0/4267320460 conn(0x55723f7a0f60 msgr2=0x55723f7a1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 -- 172.21.15.28:0/4267320460 shutdown_connections 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 -- 172.21.15.28:0/4267320460 wait complete. 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.482+0000 7f64f8ded740 1 -- start start 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 1 Processor -- start 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a51b0 0x55723f7b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55723f7b6b90 con 0x55723f7a4890 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55723f6c5f20 con 0x55723f7a51b0 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55723f7b6d10 con 0x55723f7b3fc0 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f8ded740 10 client.0 _ll_register_callbacks cb 0x55723f68f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40062/0 (socket says 172.21.15.28:40062) 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 -- 172.21.15.28:0/2191493188 learned_addr learned my addr 172.21.15.28:0/2191493188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a51b0 0x55723f7b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 -- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 msgr2=0x55723f7b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 -- 172.21.15.28:0/2191493188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a51b0 msgr2=0x55723f7b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a51b0 0x55723f7b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64effff640 1 -- 172.21.15.28:0/2191493188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55723f47c2b0 con 0x55723f7a4890 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a51b0 0x55723f7b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.483+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64effff640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7b3860 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f64e0000c00 tx=0x7f64e0004ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3e46c4143a8311 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64cf7fe640 1 -- 172.21.15.28:0/2191493188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0014070 con 0x55723f7a4890 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64cf7fe640 1 -- 172.21.15.28:0/2191493188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f64e0005180 con 0x55723f7a4890 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64e00047a0 con 0x55723f7a4890 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64cf7fe640 1 -- 172.21.15.28:0/2191493188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64e00053c0 con 0x55723f7a4890 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 10 client.15228 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55723f7c24b0 con 0x55723f7a4890 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64cf7fe640 1 -- 172.21.15.28:0/2191493188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e00047a0 con 0x55723f7a4890 2024-09-15T21:57:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64cf7fe640 1 -- 172.21.15.28:0/2191493188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f64e0023520 con 0x55723f7a4890 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 10 client.15228 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 10 client.15228 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 20 client.15228 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f7c48a0 0x55723f7c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 4 client.15228 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.484+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55723f718600 con 0x55723f7c48a0 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.485+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f7c48a0 0x55723f7c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.485+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f7c48a0 0x55723f7c5d10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4d7c8a4320c19672 server_cookie=da0b537fd3041326 in_seq=0 out_seq=0 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.485+0000 7f64cf7fe640 10 client.15228 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64cf7fe640 1 -- 172.21.15.28:0/2191493188 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55723f718600 con 0x55723f7c48a0 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64cf7fe640 10 client.15228 handle_command_reply: tid=0 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 client.15228 shutdown 2024-09-15T21:57:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 msgr2=0x55723f7b3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7b3860 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f64e0000c00 tx=0x7f64e0004ec0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 shutdown_connections 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f7c48a0 0x55723f7c5d10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7b3fc0 0x55723f7b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a51b0 0x55723f7b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 --2- 172.21.15.28:0/2191493188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7b3860 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 >> 172.21.15.28:0/2191493188 conn(0x55723f7a0f60 msgr2=0x55723f7a6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.486+0000 7f64cf7fe640 0 client.15228 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.487+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 shutdown_connections 2024-09-15T21:57:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.487+0000 7f64f8ded740 1 -- 172.21.15.28:0/2191493188 wait complete. 2024-09-15T21:57:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.487+0000 7f64f8ded740 20 client.15228 trim_cache size 0 max 16384 2024-09-15T21:57:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 Processor -- start 2024-09-15T21:57:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 -- start start 2024-09-15T21:57:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7ade60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7b3fc0 0x55723f7ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55723f7c4050 con 0x55723f7a4890 2024-09-15T21:57:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55723f7c4720 con 0x55723f7b3fc0 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f8ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55723f7c4df0 con 0x55723f7a51b0 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40070/0 (socket says 172.21.15.28:40070) 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 -- 172.21.15.28:0/204659451 learned_addr learned my addr 172.21.15.28:0/204659451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7b3fc0 0x55723f7ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 -- 172.21.15.28:0/204659451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 msgr2=0x55723f7ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 -- 172.21.15.28:0/204659451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7b3fc0 msgr2=0x55723f7ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7b3fc0 0x55723f7ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64effff640 1 -- 172.21.15.28:0/204659451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55723f7af150 con 0x55723f7a4890 2024-09-15T21:57:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.505+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64effff640 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7c5640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f64e0023910 tx=0x7f64e0008430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef20f326fbfef8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64d5ffb640 1 -- 172.21.15.28:0/204659451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0014040 con 0x55723f7a4890 2024-09-15T21:57:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64d5ffb640 1 -- 172.21.15.28:0/204659451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e0008510 con 0x55723f7a4890 2024-09-15T21:57:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64d5ffb640 1 -- 172.21.15.28:0/204659451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0023420 con 0x55723f7a4890 2024-09-15T21:57:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64f8ded740 1 -- 172.21.15.28:0/204659451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 msgr2=0x55723f7c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64f8ded740 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7c5640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f64e0023910 tx=0x7f64e0008430 comp rx=0 tx=0).stop 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64f8ded740 1 -- 172.21.15.28:0/204659451 shutdown_connections 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64f8ded740 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7b3fc0 0x55723f7ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64f8ded740 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64f8ded740 1 --2- 172.21.15.28:0/204659451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7a4890 0x55723f7c5640 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.506+0000 7f64f8ded740 1 -- 172.21.15.28:0/204659451 >> 172.21.15.28:0/204659451 conn(0x55723f7a0f60 msgr2=0x55723f79bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 -- 172.21.15.28:0/204659451 shutdown_connections 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 -- 172.21.15.28:0/204659451 wait complete. 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 -- start start 2024-09-15T21:57:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 Processor -- start 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7b3fc0 0x55723f7c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55723f7c5b90 con 0x55723f7b3fc0 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55723f7c54c0 con 0x55723f7a4890 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55723f7c6260 con 0x55723f7a51b0 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f8ded740 10 client.0 _ll_register_callbacks cb 0x7f64e8009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f4fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7b3fc0 0x55723f7c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f4fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7b3fc0 0x55723f7c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40074/0 (socket says 172.21.15.28:40074) 2024-09-15T21:57:07.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.507+0000 7f64f4fb1640 1 -- 172.21.15.28:0/4036156189 learned_addr learned my addr 172.21.15.28:0/4036156189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64effff640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64f4fb1640 1 -- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 msgr2=0x55723f7c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64f4fb1640 1 -- 172.21.15.28:0/4036156189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 msgr2=0x55723f7c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64f4fb1640 1 -- 172.21.15.28:0/4036156189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55723f82d0d0 con 0x55723f7b3fc0 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64ef7fe640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64f4fb1640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7b3fc0 0x55723f7c7a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f64e8005fd0 tx=0x7f64e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da413f9b7a6c65d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64cf7fe640 1 -- 172.21.15.28:0/4036156189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e8010be0 con 0x55723f7b3fc0 2024-09-15T21:57:07.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64cf7fe640 1 -- 172.21.15.28:0/4036156189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f64e801b070 con 0x55723f7b3fc0 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.508+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64e0008510 con 0x55723f7b3fc0 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64cf7fe640 1 -- 172.21.15.28:0/4036156189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64e8035050 con 0x55723f7b3fc0 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 10 client.15234 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55723f82e3b0 con 0x55723f7b3fc0 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64cf7fe640 1 -- 172.21.15.28:0/4036156189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e0008510 con 0x55723f7b3fc0 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64cf7fe640 1 -- 172.21.15.28:0/4036156189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f64e801e040 con 0x55723f7b3fc0 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 10 client.15234 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 10 client.15234 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 20 client.15234 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:07.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f833070 0x55723f831d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 4 client.15234 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "status"}) -- 0x55723f712b10 con 0x55723f833070 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.509+0000 7f64effff640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f833070 0x55723f831d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64cf7fe640 10 client.15234 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64effff640 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f833070 0x55723f831d80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ca4078437456605 server_cookie=688b89011f72d0c5 in_seq=0 out_seq=0 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64cf7fe640 1 -- 172.21.15.28:0/4036156189 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55723f712b10 con 0x55723f833070 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64cf7fe640 10 client.15234 handle_command_reply: tid=0 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 client.15234 shutdown 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7b3fc0 msgr2=0x55723f7c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7b3fc0 0x55723f7c7a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f64e8005fd0 tx=0x7f64e8004a40 comp rx=0 tx=0).stop 2024-09-15T21:57:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 shutdown_connections 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55723f833070 0x55723f831d80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55723f7b3fc0 0x55723f7c7a00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55723f7a51b0 0x55723f7c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 --2- 172.21.15.28:0/4036156189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55723f7a4890 0x55723f7c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64cf7fe640 0 client.15234 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 >> 172.21.15.28:0/4036156189 conn(0x55723f7a0f60 msgr2=0x55723f7aba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 shutdown_connections 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 1 -- 172.21.15.28:0/4036156189 wait complete. 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.510+0000 7f64f8ded740 20 client.15234 trim_cache size 0 max 16384 2024-09-15T21:57:07.518 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24454, 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 89.958618678999997, 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 156.317609191 2024-09-15T21:57:07.519 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f64d80a3240 msgr2=0x7f64d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f64d80a3240 0x7f64d80a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f64e8005c40 tx=0x7f64e8009280 comp rx=0 tx=0).stop 2024-09-15T21:57:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 msgr2=0x7f64f01988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01988e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f64e4006010 tx=0x7f64e40042f0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 shutdown_connections 2024-09-15T21:57:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f64d80a3240 0x7f64d80a5700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64f0106f10 0x7f64f019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f0102790 0x7f64f01988e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 --2- 172.21.15.28:0/799683632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64f0101de0 0x7f64f01983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 >> 172.21.15.28:0/799683632 conn(0x7f64f00fd8c0 msgr2=0x7f64f00fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 shutdown_connections 2024-09-15T21:57:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:07.511+0000 7f64f6a14640 1 -- 172.21.15.28:0/799683632 wait complete. 2024-09-15T21:57:08.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T21:57:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 -- 172.21.15.28:0/3915379839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518108290 msgr2=0x7f251810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 --2- 172.21.15.28:0/3915379839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518108290 0x7f251810ec30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2510009cd0 tx=0x7f251002a8e0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 -- 172.21.15.28:0/3915379839 shutdown_connections 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 --2- 172.21.15.28:0/3915379839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518108290 0x7f251810ec30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 --2- 172.21.15.28:0/3915379839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2518103740 0x7f2518107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 --2- 172.21.15.28:0/3915379839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2518102d90 0x7f2518103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 -- 172.21.15.28:0/3915379839 >> 172.21.15.28:0/3915379839 conn(0x7f25180fc7d0 msgr2=0x7f25180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 -- 172.21.15.28:0/3915379839 shutdown_connections 2024-09-15T21:57:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.998+0000 7f251c889640 1 -- 172.21.15.28:0/3915379839 wait complete. 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 Processor -- start 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 -- start start 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2518102d90 0x7f251819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518103740 0x7f251819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2518108290 0x7f251819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2518112090 con 0x7f2518103740 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2518111f10 con 0x7f2518102d90 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:08.999+0000 7f251c889640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2518112210 con 0x7f2518108290 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518103740 0x7f251819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518103740 0x7f251819abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40100/0 (socket says 172.21.15.28:40100) 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 -- 172.21.15.28:0/1543374483 learned_addr learned my addr 172.21.15.28:0/1543374483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f251659c640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2518102d90 0x7f251819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 -- 172.21.15.28:0/1543374483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2518108290 msgr2=0x7f251819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2518108290 0x7f251819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 -- 172.21.15.28:0/1543374483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2518102d90 msgr2=0x7f251819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2518102d90 0x7f251819a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 -- 172.21.15.28:0/1543374483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251819f650 con 0x7f2518103740 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f251659c640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2518102d90 0x7f251819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.000+0000 7f2515d9b640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518103740 0x7f251819abe0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f250c00c970 tx=0x7f250c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.001+0000 7f25077fe640 1 -- 172.21.15.28:0/1543374483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c014070 con 0x7f2518103740 2024-09-15T21:57:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.001+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251819f940 con 0x7f2518103740 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.001+0000 7f25077fe640 1 -- 172.21.15.28:0/1543374483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f250c004440 con 0x7f2518103740 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.001+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25181a7220 con 0x7f2518103740 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.001+0000 7f25077fe640 1 -- 172.21.15.28:0/1543374483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c00f450 con 0x7f2518103740 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.003+0000 7f251ec62740 1 Processor -- start 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.003+0000 7f25077fe640 1 -- 172.21.15.28:0/1543374483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f250c017070 con 0x7f2518103740 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251ec62740 1 -- start start 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251ec62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87a9d0 0x55b5ea87adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251ec62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87b2f0 0x55b5ea885670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251ec62740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea887fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251ec62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea7e1090 con 0x55b5ea87b2f0 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251ec62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea788a70 con 0x55b5ea87a9d0 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251ec62740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea7dd050 con 0x55b5ea885bb0 2024-09-15T21:57:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f251659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87b2f0 0x55b5ea885670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f2515d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea887fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f2516d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87a9d0 0x55b5ea87adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f2515d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea887fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:60470/0 (socket says 172.21.15.28:60470) 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.004+0000 7f2515d9b640 1 -- 172.21.15.28:0/2222989680 learned_addr learned my addr 172.21.15.28:0/2222989680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f25077fe640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f25000a3240 0x7f25000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f251659c640 1 -- 172.21.15.28:0/2222989680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 msgr2=0x55b5ea887fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f251659c640 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea887fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f251659c640 1 -- 172.21.15.28:0/2222989680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87a9d0 msgr2=0x55b5ea87adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f251659c640 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87a9d0 0x55b5ea87adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f251659c640 1 -- 172.21.15.28:0/2222989680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ea795130 con 0x55b5ea87b2f0 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f2515d9b640 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea887fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f2516d9d640 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87a9d0 0x55b5ea87adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f2516d9d640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f25000a3240 0x7f25000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f25077fe640 1 -- 172.21.15.28:0/1543374483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f250c0cfb10 con 0x7f2518103740 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f251659c640 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87b2f0 0x55b5ea885670 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f250800cbd0 tx=0x7f250800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395ee8b626856667 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f2504ff9640 1 -- 172.21.15.28:0/2222989680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2508005220 con 0x55b5ea87b2f0 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f2504ff9640 1 -- 172.21.15.28:0/2222989680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2508004c80 con 0x55b5ea87b2f0 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.005+0000 7f2504ff9640 1 -- 172.21.15.28:0/2222989680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2508010790 con 0x55b5ea87b2f0 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 -- 172.21.15.28:0/2222989680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87b2f0 msgr2=0x55b5ea885670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f2516d9d640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f25000a3240 0x7f25000a5700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2510009ca0 tx=0x7f2510031430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87b2f0 0x55b5ea885670 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f250800cbd0 tx=0x7f250800e720 comp rx=0 tx=0).stop 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 -- 172.21.15.28:0/2222989680 shutdown_connections 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea887fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87b2f0 0x55b5ea885670 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 --2- 172.21.15.28:0/2222989680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87a9d0 0x55b5ea87adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 -- 172.21.15.28:0/2222989680 >> 172.21.15.28:0/2222989680 conn(0x55b5ea876f60 msgr2=0x55b5ea877340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 -- 172.21.15.28:0/2222989680 shutdown_connections 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.006+0000 7f251ec62740 1 -- 172.21.15.28:0/2222989680 wait complete. 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 -- start start 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 Processor -- start 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea885420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87b2f0 0x55b5ea882770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea882cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea888780 con 0x55b5ea87a9d0 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea79bf20 con 0x55b5ea87b2f0 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea888900 con 0x55b5ea885bb0 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251ec62740 10 client.0 _ll_register_callbacks cb 0x55b5ea765030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea885420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea885420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40124/0 (socket says 172.21.15.28:40124) 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 -- 172.21.15.28:0/1748393794 learned_addr learned my addr 172.21.15.28:0/1748393794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f2515d9b640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87b2f0 0x55b5ea882770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f2516d9d640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea882cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 -- 172.21.15.28:0/1748393794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 msgr2=0x55b5ea882cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea882cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 -- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87b2f0 msgr2=0x55b5ea882770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87b2f0 0x55b5ea882770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f251659c640 1 -- 172.21.15.28:0/1748393794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ea5522b0 con 0x55b5ea87a9d0 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.007+0000 7f2515d9b640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87b2f0 0x55b5ea882770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251659c640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea885420 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f250800c940 tx=0x7f250800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72ee7c0f42989e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f24fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f24fdffb640 1 -- 172.21.15.28:0/1748393794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250802a060 con 0x55b5ea87a9d0 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2508004c80 con 0x55b5ea87a9d0 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f24fdffb640 1 -- 172.21.15.28:0/1748393794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f25080048d0 con 0x55b5ea87a9d0 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f24fdffb640 1 -- 172.21.15.28:0/1748393794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2508031050 con 0x55b5ea87a9d0 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 10 client.15246 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b5ea894070 con 0x55b5ea87a9d0 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f24fdffb640 1 -- 172.21.15.28:0/1748393794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2508004c80 con 0x55b5ea87a9d0 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f24fdffb640 1 -- 172.21.15.28:0/1748393794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f250801a460 con 0x55b5ea87a9d0 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 10 client.15246 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 20 client.15246 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea8962a0 0x55b5ea897710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 4 client.15246 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.008+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5ea7ee600 con 0x55b5ea8962a0 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.009+0000 7f2515d9b640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea8962a0 0x55b5ea897710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.009+0000 7f24fdffb640 10 client.15246 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.009+0000 7f2515d9b640 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea8962a0 0x55b5ea897710 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1191402467cf8bb9 server_cookie=1ece34466c5b9121 in_seq=0 out_seq=0 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f24fdffb640 1 -- 172.21.15.28:0/1748393794 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5ea7ee600 con 0x55b5ea8962a0 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f24fdffb640 10 client.15246 handle_command_reply: tid=0 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 client.15246 shutdown 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 msgr2=0x55b5ea885420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea885420 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f250800c940 tx=0x7f250800ced0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 shutdown_connections 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea885bb0 0x55b5ea882cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea87b2f0 0x55b5ea882770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea8962a0 0x55b5ea897710 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f24fdffb640 0 client.15246 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 --2- 172.21.15.28:0/1748393794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea885420 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 >> 172.21.15.28:0/1748393794 conn(0x55b5ea876f60 msgr2=0x55b5ea87cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 shutdown_connections 2024-09-15T21:57:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 1 -- 172.21.15.28:0/1748393794 wait complete. 2024-09-15T21:57:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.010+0000 7f251ec62740 20 client.15246 trim_cache size 0 max 16384 2024-09-15T21:57:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 Processor -- start 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 -- start start 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea897040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea881040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 0x55b5ea87f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea895a50 con 0x55b5ea87a9d0 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea896120 con 0x55b5ea885bb0 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251ec62740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea8967f0 con 0x55b5ea87b2f0 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea897040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea897040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40132/0 (socket says 172.21.15.28:40132) 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251659c640 1 -- 172.21.15.28:0/3184961633 learned_addr learned my addr 172.21.15.28:0/3184961633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f2515d9b640 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea881040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251659c640 1 -- 172.21.15.28:0/3184961633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 msgr2=0x55b5ea881040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251659c640 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea881040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251659c640 1 -- 172.21.15.28:0/3184961633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 msgr2=0x55b5ea87f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.046+0000 7f251659c640 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 0x55b5ea87f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.047+0000 7f251659c640 1 -- 172.21.15.28:0/3184961633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ea881720 con 0x55b5ea87a9d0 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.047+0000 7f2515d9b640 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea881040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.047+0000 7f251659c640 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea897040 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2508005cb0 tx=0x7f250800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24442e65df740256 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.047+0000 7f24fffff640 1 -- 172.21.15.28:0/3184961633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250802a040 con 0x55b5ea87a9d0 2024-09-15T21:57:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.047+0000 7f24fffff640 1 -- 172.21.15.28:0/3184961633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f250800ec20 con 0x55b5ea87a9d0 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.047+0000 7f24fffff640 1 -- 172.21.15.28:0/3184961633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25080048d0 con 0x55b5ea87a9d0 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 -- 172.21.15.28:0/3184961633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 msgr2=0x55b5ea897040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea897040 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2508005cb0 tx=0x7f250800cb80 comp rx=0 tx=0).stop 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 -- 172.21.15.28:0/3184961633 shutdown_connections 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 0x55b5ea87f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea881040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 --2- 172.21.15.28:0/3184961633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea897040 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 -- 172.21.15.28:0/3184961633 >> 172.21.15.28:0/3184961633 conn(0x55b5ea876f60 msgr2=0x55b5ea876000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 -- 172.21.15.28:0/3184961633 shutdown_connections 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 -- 172.21.15.28:0/3184961633 wait complete. 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.048+0000 7f251ec62740 1 -- start start 2024-09-15T21:57:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 1 Processor -- start 2024-09-15T21:57:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea89fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea89ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 0x55b5ea89af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea897590 con 0x55b5ea87a9d0 2024-09-15T21:57:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea896ec0 con 0x55b5ea885bb0 2024-09-15T21:57:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b5ea897c60 con 0x55b5ea87b2f0 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251ec62740 10 client.0 _ll_register_callbacks cb 0x7f2508014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea89fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea89fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40138/0 (socket says 172.21.15.28:40138) 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 -- 172.21.15.28:0/2846985537 learned_addr learned my addr 172.21.15.28:0/2846985537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f2515d9b640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea89ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f2516d9d640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 0x55b5ea89af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 -- 172.21.15.28:0/2846985537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 msgr2=0x55b5ea89ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea89ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 -- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 msgr2=0x55b5ea89af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 0x55b5ea89af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f251659c640 1 -- 172.21.15.28:0/2846985537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ea89b780 con 0x55b5ea87a9d0 2024-09-15T21:57:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.049+0000 7f2515d9b640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea89ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251659c640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea89fa90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f250801ae90 tx=0x7f250801a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791f64661374fbf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f24fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f24fdffb640 1 -- 172.21.15.28:0/2846985537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250802a030 con 0x55b5ea87a9d0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f24fdffb640 1 -- 172.21.15.28:0/2846985537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f2508022040 con 0x55b5ea87a9d0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f250800ec20 con 0x55b5ea87a9d0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f24fdffb640 1 -- 172.21.15.28:0/2846985537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f250801a580 con 0x55b5ea87a9d0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251ec62740 10 client.15252 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b5ea9004b0 con 0x55b5ea87a9d0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f24fdffb640 1 -- 172.21.15.28:0/2846985537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f250800ec20 con 0x55b5ea87a9d0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f24fdffb640 1 -- 172.21.15.28:0/2846985537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f250803c070 con 0x55b5ea87a9d0 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251ec62740 10 client.15252 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251ec62740 10 client.15252 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251ec62740 20 client.15252 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.050+0000 7f251ec62740 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea904dc0 0x55b5ea903ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.051+0000 7f251ec62740 4 client.15252 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.051+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "status"}) -- 0x55b5ea7e8b10 con 0x55b5ea904dc0 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.051+0000 7f2515d9b640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea904dc0 0x55b5ea903ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.051+0000 7f24fdffb640 10 client.15252 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.051+0000 7f2515d9b640 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea904dc0 0x55b5ea903ad0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bad055c01c1d17f6 server_cookie=bdf1dfc6ad70548d in_seq=0 out_seq=0 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f24fdffb640 1 -- 172.21.15.28:0/2846985537 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b5ea7e8b10 con 0x55b5ea904dc0 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f24fdffb640 10 client.15252 handle_command_reply: tid=0 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 client.15252 shutdown 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 msgr2=0x55b5ea89fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea89fa90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f250801ae90 tx=0x7f250801a8b0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 shutdown_connections 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5ea885bb0 0x55b5ea89af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5ea87b2f0 0x55b5ea89ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b5ea904dc0 0x55b5ea903ad0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 --2- 172.21.15.28:0/2846985537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5ea87a9d0 0x55b5ea89fa90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f24fdffb640 0 client.15252 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 >> 172.21.15.28:0/2846985537 conn(0x55b5ea876f60 msgr2=0x55b5ea876000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 shutdown_connections 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 1 -- 172.21.15.28:0/2846985537 wait complete. 2024-09-15T21:57:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251ec62740 20 client.15252 trim_cache size 0 max 16384 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24458, 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 92.505439351000007, 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-09-15T21:57:09.059 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:57:09.060 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 152.35765294500001 2024-09-15T21:57:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f25000a3240 msgr2=0x7f25000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251c889640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f25000a3240 0x7f25000a5700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2510009ca0 tx=0x7f2510031430 comp rx=0 tx=0).stop 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.052+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518103740 msgr2=0x7f251819abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518103740 0x7f251819abe0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f250c00c970 tx=0x7f250c00ce30 comp rx=0 tx=0).stop 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 shutdown_connections 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f25000a3240 0x7f25000a5700 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2518108290 0x7f251819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2518103740 0x7f251819abe0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 --2- 172.21.15.28:0/1543374483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2518102d90 0x7f251819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 >> 172.21.15.28:0/1543374483 conn(0x7f25180fc7d0 msgr2=0x7f25180fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 shutdown_connections 2024-09-15T21:57:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:09.053+0000 7f251c889640 1 -- 172.21.15.28:0/1543374483 wait complete. 2024-09-15T21:57:09.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:08 smithi028 ceph-mon[30969]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 742 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:09.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:08 smithi203 ceph-mon[35546]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 742 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:09.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:08 smithi174 ceph-mon[35244]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 742 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:09.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-15T21:57:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 -- 172.21.15.28:0/2168713227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc750103660 msgr2=0x7fc75010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 --2- 172.21.15.28:0/2168713227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc750103660 0x7fc75010bb50 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc738009900 tx=0x7fc73802a480 comp rx=0 tx=0).stop 2024-09-15T21:57:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 -- 172.21.15.28:0/2168713227 shutdown_connections 2024-09-15T21:57:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 --2- 172.21.15.28:0/2168713227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc75010c170 0x7fc75010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 --2- 172.21.15.28:0/2168713227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc750103660 0x7fc75010bb50 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 --2- 172.21.15.28:0/2168713227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc750102d40 0x7fc750103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 -- 172.21.15.28:0/2168713227 >> 172.21.15.28:0/2168713227 conn(0x7fc7500fc7a0 msgr2=0x7fc7500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 -- 172.21.15.28:0/2168713227 shutdown_connections 2024-09-15T21:57:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.581+0000 7fc754bbe640 1 -- 172.21.15.28:0/2168713227 wait complete. 2024-09-15T21:57:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 Processor -- start 2024-09-15T21:57:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 -- start start 2024-09-15T21:57:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc750102d40 0x7fc750198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc750103660 0x7fc7501989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc75010c170 0x7fc75019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc750111f70 con 0x7fc75010c170 2024-09-15T21:57:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc750111df0 con 0x7fc750102d40 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc754bbe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501120f0 con 0x7fc750103660 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc74ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc75010c170 0x7fc75019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc74ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc75010c170 0x7fc75019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35196/0 (socket says 172.21.15.28:35196) 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc74ed9d640 1 -- 172.21.15.28:0/3731951830 learned_addr learned my addr 172.21.15.28:0/3731951830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc745d9b640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc750103660 0x7fc7501989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.582+0000 7fc74e59c640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc750102d40 0x7fc750198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc74ed9d640 1 -- 172.21.15.28:0/3731951830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc750103660 msgr2=0x7fc7501989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc74ed9d640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc750103660 0x7fc7501989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc74ed9d640 1 -- 172.21.15.28:0/3731951830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc750102d40 msgr2=0x7fc750198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc74ed9d640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc750102d40 0x7fc750198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc74ed9d640 1 -- 172.21.15.28:0/3731951830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75019d420 con 0x7fc75010c170 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc74ed9d640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc75010c170 0x7fc75019cd40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc74800eaa0 tx=0x7fc74800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc747fff640 1 -- 172.21.15.28:0/3731951830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74800cdb0 con 0x7fc75010c170 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75019d710 con 0x7fc75010c170 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc747fff640 1 -- 172.21.15.28:0/3731951830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc748005020 con 0x7fc75010c170 2024-09-15T21:57:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc747fff640 1 -- 172.21.15.28:0/3731951830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7480116d0 con 0x7fc75010c170 2024-09-15T21:57:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.583+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7501a4ff0 con 0x7fc75010c170 2024-09-15T21:57:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.586+0000 7fc756f97740 1 Processor -- start 2024-09-15T21:57:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.586+0000 7fc747fff640 1 -- 172.21.15.28:0/3731951830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc7480054d0 con 0x7fc75010c170 2024-09-15T21:57:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.586+0000 7fc756f97740 1 -- start start 2024-09-15T21:57:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc756f97740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919439d0 0x558c91943db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc756f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc756f97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9194ebb0 0x558c91950fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc756f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c918aa090 con 0x558c919442f0 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc756f97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c91851a70 con 0x558c9194ebb0 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc756f97740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c918a6050 con 0x558c919439d0 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35208/0 (socket says 172.21.15.28:35208) 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 -- 172.21.15.28:0/3320172424 learned_addr learned my addr 172.21.15.28:0/3320172424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc745d9b640 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919439d0 0x558c91943db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc745d9b640 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9194ebb0 0x558c91950fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc747fff640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc73c0a3240 0x7fc73c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 -- 172.21.15.28:0/3320172424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919439d0 msgr2=0x558c91943db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919439d0 0x558c91943db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 -- 172.21.15.28:0/3320172424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9194ebb0 msgr2=0x558c91950fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9194ebb0 0x558c91950fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 -- 172.21.15.28:0/3320172424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c9185e130 con 0x558c919442f0 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc745d9b640 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919439d0 0x558c91943db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74ed9d640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc73c0a3240 0x7fc73c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc747fff640 1 -- 172.21.15.28:0/3731951830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fc7480cfdc0 con 0x7fc75010c170 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.587+0000 7fc74e59c640 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194e670 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc74000bd10 tx=0x7fc7400107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81cab4603f1b1fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.588+0000 7fc744d99640 1 -- 172.21.15.28:0/3320172424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740004030 con 0x558c919442f0 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.588+0000 7fc744d99640 1 -- 172.21.15.28:0/3320172424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7400041d0 con 0x558c919442f0 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.588+0000 7fc744d99640 1 -- 172.21.15.28:0/3320172424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7400117e0 con 0x558c919442f0 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.588+0000 7fc756f97740 1 -- 172.21.15.28:0/3320172424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 msgr2=0x558c9194e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.588+0000 7fc74ed9d640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc73c0a3240 0x7fc73c0a5700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc7480cd540 tx=0x7fc74801eb40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.588+0000 7fc756f97740 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194e670 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc74000bd10 tx=0x7fc7400107a0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- 172.21.15.28:0/3320172424 shutdown_connections 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9194ebb0 0x558c91950fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194e670 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 --2- 172.21.15.28:0/3320172424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919439d0 0x558c91943db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- 172.21.15.28:0/3320172424 >> 172.21.15.28:0/3320172424 conn(0x558c9193ff60 msgr2=0x558c91940340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- 172.21.15.28:0/3320172424 shutdown_connections 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- 172.21.15.28:0/3320172424 wait complete. 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- start start 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 Processor -- start 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9194e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c9194ebb0 0x558c9194bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c91951780 con 0x558c919442f0 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c91864f20 con 0x558c919439d0 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.589+0000 7fc756f97740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c91951900 con 0x558c9194ebb0 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc756f97740 10 client.0 _ll_register_callbacks cb 0x558c9182e030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35224/0 (socket says 172.21.15.28:35224) 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 -- 172.21.15.28:0/1815707385 learned_addr learned my addr 172.21.15.28:0/1815707385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc745d9b640 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9194e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc745d9b640 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c9194ebb0 0x558c9194bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 -- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c9194ebb0 msgr2=0x558c9194bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c9194ebb0 0x558c9194bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 -- 172.21.15.28:0/1815707385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 msgr2=0x558c9194e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9194e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 -- 172.21.15.28:0/1815707385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c9161b2b0 con 0x558c919442f0 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc74e59c640 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194b780 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc740011480 tx=0x7fc740003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8267ed66e088646f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc735ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc735ffb640 1 -- 172.21.15.28:0/1815707385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740029060 con 0x558c919442f0 2024-09-15T21:57:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc735ffb640 1 -- 172.21.15.28:0/1815707385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fc7400035f0 con 0x558c919442f0 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7400041d0 con 0x558c919442f0 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc735ffb640 1 -- 172.21.15.28:0/1815707385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc740030050 con 0x558c919442f0 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc756f97740 10 client.15264 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.590+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c9195d000 con 0x558c919442f0 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc735ffb640 1 -- 172.21.15.28:0/1815707385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7400041d0 con 0x558c919442f0 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc735ffb640 1 -- 172.21.15.28:0/1815707385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc7400046c0 con 0x558c919442f0 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc756f97740 10 client.15264 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc756f97740 10 client.15264 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc756f97740 20 client.15264 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc756f97740 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c9195f230 0x558c919606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc756f97740 4 client.15264 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.591+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c918b7600 con 0x558c9195f230 2024-09-15T21:57:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.592+0000 7fc745d9b640 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c9195f230 0x558c919606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.592+0000 7fc745d9b640 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c9195f230 0x558c919606a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54fcee577f150991 server_cookie=a277d074c6509304 in_seq=0 out_seq=0 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.592+0000 7fc735ffb640 10 client.15264 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc735ffb640 1 -- 172.21.15.28:0/1815707385 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558c918b7600 con 0x558c9195f230 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc735ffb640 10 client.15264 handle_command_reply: tid=0 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 client.15264 shutdown 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 msgr2=0x558c9194b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194b780 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc740011480 tx=0x7fc740003f70 comp rx=0 tx=0).stop 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 shutdown_connections 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c9195f230 0x558c919606a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c9194ebb0 0x558c9194bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c919442f0 0x558c9194b780 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc735ffb640 0 client.15264 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 --2- 172.21.15.28:0/1815707385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9194e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 >> 172.21.15.28:0/1815707385 conn(0x558c9193ff60 msgr2=0x558c91945c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 shutdown_connections 2024-09-15T21:57:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 1 -- 172.21.15.28:0/1815707385 wait complete. 2024-09-15T21:57:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.593+0000 7fc756f97740 20 client.15264 trim_cache size 0 max 16384 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.611+0000 7fc756f97740 1 Processor -- start 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.611+0000 7fc756f97740 1 -- start start 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.611+0000 7fc756f97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9195ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.611+0000 7fc756f97740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c919489e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.611+0000 7fc756f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c91948f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc756f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c9195e9e0 con 0x558c9194ebb0 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc756f97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c9195f0b0 con 0x558c919439d0 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc756f97740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c9195f780 con 0x558c919442f0 2024-09-15T21:57:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c91948f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c91948f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35238/0 (socket says 172.21.15.28:35238) 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 -- 172.21.15.28:0/2455895794 learned_addr learned my addr 172.21.15.28:0/2455895794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc74e59c640 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c919489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9195ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 -- 172.21.15.28:0/2455895794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 msgr2=0x558c919489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c919489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 -- 172.21.15.28:0/2455895794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 msgr2=0x558c9195ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9195ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 -- 172.21.15.28:0/2455895794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c91943800 con 0x558c9194ebb0 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c91948f20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc738009900 tx=0x7fc73802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b474e2c9f9e601d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc737fff640 1 -- 172.21.15.28:0/2455895794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738035040 con 0x558c9194ebb0 2024-09-15T21:57:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc737fff640 1 -- 172.21.15.28:0/2455895794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc73804f070 con 0x558c9194ebb0 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.612+0000 7fc737fff640 1 -- 172.21.15.28:0/2455895794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738004a40 con 0x558c9194ebb0 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 -- 172.21.15.28:0/2455895794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 msgr2=0x558c91948f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c91948f20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc738009900 tx=0x7fc73802a480 comp rx=0 tx=0).stop 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 -- 172.21.15.28:0/2455895794 shutdown_connections 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c91948f20 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c919489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 --2- 172.21.15.28:0/2455895794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c9195ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 -- 172.21.15.28:0/2455895794 >> 172.21.15.28:0/2455895794 conn(0x558c9193ff60 msgr2=0x558c9193f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 -- 172.21.15.28:0/2455895794 shutdown_connections 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 -- 172.21.15.28:0/2455895794 wait complete. 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 -- start start 2024-09-15T21:57:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.613+0000 7fc756f97740 1 Processor -- start 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc756f97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c91964620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc756f97740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c91964b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc756f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c919650a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc756f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c91960520 con 0x558c9194ebb0 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc756f97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c9195fe50 con 0x558c919439d0 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc756f97740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c91960bf0 con 0x558c919442f0 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc756f97740 10 client.0 _ll_register_callbacks cb 0x7fc74000bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c919650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc74e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c91964b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c919650a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35248/0 (socket says 172.21.15.28:35248) 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 -- 172.21.15.28:0/2558912117 learned_addr learned my addr 172.21.15.28:0/2558912117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c91964620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 -- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 msgr2=0x558c91964b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c91964b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 -- 172.21.15.28:0/2558912117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 msgr2=0x558c91964620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c91964620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 -- 172.21.15.28:0/2558912117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c919c8940 con 0x558c9194ebb0 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc74e59c640 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c91964b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc745d9b640 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c919650a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc738033000 tx=0x7fc738004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5717e25665e1c96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.614+0000 7fc735ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc735ffb640 1 -- 172.21.15.28:0/2558912117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738035040 con 0x558c9194ebb0 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc735ffb640 1 -- 172.21.15.28:0/2558912117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fc738032250 con 0x558c9194ebb0 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc73804f070 con 0x558c9194ebb0 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc735ffb640 1 -- 172.21.15.28:0/2558912117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc738032450 con 0x558c9194ebb0 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 10 client.15270 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c919c9b80 con 0x558c9194ebb0 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc735ffb640 1 -- 172.21.15.28:0/2558912117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc73804f070 con 0x558c9194ebb0 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc735ffb640 1 -- 172.21.15.28:0/2558912117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc7380318b0 con 0x558c9194ebb0 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 10 client.15270 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 10 client.15270 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 20 client.15270 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c919ce200 0x558c919ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 4 client.15270 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.615+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "status"}) -- 0x558c918b1b10 con 0x558c919ce200 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc74e59c640 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c919ce200 0x558c919ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc74e59c640 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c919ce200 0x558c919ccf10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ecf74c990bc883e server_cookie=267d423ab919bf30 in_seq=0 out_seq=0 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc735ffb640 10 client.15270 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc735ffb640 1 -- 172.21.15.28:0/2558912117 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558c918b1b10 con 0x558c919ce200 2024-09-15T21:57:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc735ffb640 10 client.15270 handle_command_reply: tid=0 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 client.15270 shutdown 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 msgr2=0x558c919650a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c919650a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc738033000 tx=0x7fc738004840 comp rx=0 tx=0).stop 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 shutdown_connections 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558c919ce200 0x558c919ccf10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c9194ebb0 0x558c919650a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c919442f0 0x558c91964b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 --2- 172.21.15.28:0/2558912117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c919439d0 0x558c91964620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 >> 172.21.15.28:0/2558912117 conn(0x558c9193ff60 msgr2=0x558c9193e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc735ffb640 0 client.15270 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 shutdown_connections 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.616+0000 7fc756f97740 1 -- 172.21.15.28:0/2558912117 wait complete. 2024-09-15T21:57:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc756f97740 20 client.15270 trim_cache size 0 max 16384 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24460, 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:10.633 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 95.082732375000006, 2024-09-15T21:57:10.634 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:57:10.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-09-15T21:57:10.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:57:10.634 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 156.79283875199999 2024-09-15T21:57:10.634 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc73c0a3240 msgr2=0x7fc73c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc73c0a3240 0x7fc73c0a5700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc7480cd540 tx=0x7fc74801eb40 comp rx=0 tx=0).stop 2024-09-15T21:57:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc75010c170 msgr2=0x7fc75019cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc75010c170 0x7fc75019cd40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc74800eaa0 tx=0x7fc74800ef60 comp rx=0 tx=0).stop 2024-09-15T21:57:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 shutdown_connections 2024-09-15T21:57:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc73c0a3240 0x7fc73c0a5700 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc75010c170 0x7fc75019cd40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc750103660 0x7fc7501989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 --2- 172.21.15.28:0/3731951830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc750102d40 0x7fc750198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 >> 172.21.15.28:0/3731951830 conn(0x7fc7500fc7a0 msgr2=0x7fc7500fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 shutdown_connections 2024-09-15T21:57:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:10.617+0000 7fc754bbe640 1 -- 172.21.15.28:0/3731951830 wait complete. 2024-09-15T21:57:11.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:10 smithi028 ceph-mon[30969]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 576 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:11.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:10 smithi203 ceph-mon[35546]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 576 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:11.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:10 smithi174 ceph-mon[35244]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 576 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:11.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T21:57:11.604 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:12.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 -- 172.21.15.28:0/3145036643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140102d40 msgr2=0x7fc140103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 --2- 172.21.15.28:0/3145036643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140102d40 0x7fc140103120 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc130009900 tx=0x7fc13002a480 comp rx=0 tx=0).stop 2024-09-15T21:57:12.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 -- 172.21.15.28:0/3145036643 shutdown_connections 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 --2- 172.21.15.28:0/3145036643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc14010c170 0x7fc14010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 --2- 172.21.15.28:0/3145036643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc140103660 0x7fc14010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 --2- 172.21.15.28:0/3145036643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc140102d40 0x7fc140103120 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 -- 172.21.15.28:0/3145036643 >> 172.21.15.28:0/3145036643 conn(0x7fc1400fc7a0 msgr2=0x7fc1400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 -- 172.21.15.28:0/3145036643 shutdown_connections 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.048+0000 7fc145b5e640 1 -- 172.21.15.28:0/3145036643 wait complete. 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.049+0000 7fc145b5e640 1 Processor -- start 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.049+0000 7fc145b5e640 1 -- start start 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc145b5e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc140102d40 0x7fc140198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc145b5e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc140103660 0x7fc140198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc145b5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14010c170 0x7fc14019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc145b5e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc140111f70 con 0x7fc14010c170 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc145b5e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc140111df0 con 0x7fc140103660 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc145b5e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401120f0 con 0x7fc140102d40 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14010c170 0x7fc14019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc140102d40 0x7fc140198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14010c170 0x7fc14019cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35268/0 (socket says 172.21.15.28:35268) 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 -- 172.21.15.28:0/2991761416 learned_addr learned my addr 172.21.15.28:0/2991761416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13effd640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc140103660 0x7fc140198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 -- 172.21.15.28:0/2991761416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc140102d40 msgr2=0x7fc140198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc140102d40 0x7fc140198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 -- 172.21.15.28:0/2991761416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc140103660 msgr2=0x7fc140198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc140103660 0x7fc140198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 -- 172.21.15.28:0/2991761416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14019d3f0 con 0x7fc14010c170 2024-09-15T21:57:12.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14010c170 0x7fc14019cd10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc13800eaa0 tx=0x7fc13800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.050+0000 7fc13cff9640 1 -- 172.21.15.28:0/2991761416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13800cdb0 con 0x7fc14010c170 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.051+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14019d6e0 con 0x7fc14010c170 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.051+0000 7fc13cff9640 1 -- 172.21.15.28:0/2991761416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc138005020 con 0x7fc14010c170 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.051+0000 7fc13cff9640 1 -- 172.21.15.28:0/2991761416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1380116b0 con 0x7fc14010c170 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.051+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1401a4fc0 con 0x7fc14010c170 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.053+0000 7fc13cff9640 1 -- 172.21.15.28:0/2991761416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc138005560 con 0x7fc14010c170 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.053+0000 7fc147f37740 1 Processor -- start 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.053+0000 7fc147f37740 1 -- start start 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc147f37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f628818db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc147f37740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f628823670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc147f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628825fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc147f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f62877f090 con 0x55f628823bb0 2024-09-15T21:57:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc147f37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f628726a70 con 0x55f6288189d0 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc147f37740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f62877b050 con 0x55f6288192f0 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13cff9640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc1280a3240 0x7fc1280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628825fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628825fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35282/0 (socket says 172.21.15.28:35282) 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13effd640 1 -- 172.21.15.28:0/2057788052 learned_addr learned my addr 172.21.15.28:0/2057788052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f628823670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13effd640 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f628818db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc1280a3240 0x7fc1280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13cff9640 1 -- 172.21.15.28:0/2991761416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fc1380cfee0 con 0x7fc14010c170 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13f7fe640 1 -- 172.21.15.28:0/2057788052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 msgr2=0x55f628818db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f628818db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13f7fe640 1 -- 172.21.15.28:0/2057788052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 msgr2=0x55f628825fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628825fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13f7fe640 1 -- 172.21.15.28:0/2057788052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f628733130 con 0x55f6288192f0 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.054+0000 7fc13effd640 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628825fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.055+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f628823670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc130005940 tx=0x7fc130031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c859b864d34e188c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.055+0000 7fc13ffff640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc1280a3240 0x7fc1280a5700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc13800efd0 tx=0x7fc138066b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.055+0000 7fc125ffb640 1 -- 172.21.15.28:0/2057788052 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130040070 con 0x55f6288192f0 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.055+0000 7fc125ffb640 1 -- 172.21.15.28:0/2057788052 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc130005020 con 0x55f6288192f0 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.055+0000 7fc125ffb640 1 -- 172.21.15.28:0/2057788052 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130035040 con 0x55f6288192f0 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 -- 172.21.15.28:0/2057788052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 msgr2=0x55f628823670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f628823670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc130005940 tx=0x7fc130031630 comp rx=0 tx=0).stop 2024-09-15T21:57:12.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 -- 172.21.15.28:0/2057788052 shutdown_connections 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628825fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f628823670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 --2- 172.21.15.28:0/2057788052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f628818db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 -- 172.21.15.28:0/2057788052 >> 172.21.15.28:0/2057788052 conn(0x55f628814f60 msgr2=0x55f628815340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 -- 172.21.15.28:0/2057788052 shutdown_connections 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 -- 172.21.15.28:0/2057788052 wait complete. 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 -- start start 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.056+0000 7fc147f37740 1 Processor -- start 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.057+0000 7fc147f37740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288189d0 0x55f628823360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.057+0000 7fc147f37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288192f0 0x55f6288206b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.057+0000 7fc147f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628820bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.057+0000 7fc147f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f628826780 con 0x55f628823bb0 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.057+0000 7fc147f37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f628739f20 con 0x55f6288192f0 2024-09-15T21:57:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.057+0000 7fc147f37740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f628826900 con 0x55f6288189d0 2024-09-15T21:57:12.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.057+0000 7fc147f37740 10 client.0 _ll_register_callbacks cb 0x55f628703030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:12.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628820bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628820bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35290/0 (socket says 172.21.15.28:35290) 2024-09-15T21:57:12.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 -- 172.21.15.28:0/397430285 learned_addr learned my addr 172.21.15.28:0/397430285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:12.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288189d0 0x55f628823360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288192f0 0x55f6288206b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 -- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288189d0 msgr2=0x55f628823360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288189d0 0x55f628823360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 -- 172.21.15.28:0/397430285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288192f0 msgr2=0x55f6288206b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288192f0 0x55f6288206b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 -- 172.21.15.28:0/397430285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6284f02b0 con 0x55f628823bb0 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc13effd640 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628820bf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc1340107b0 tx=0x7fc13400bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515557e52ea6f138 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc11effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.058+0000 7fc11effd640 1 -- 172.21.15.28:0/397430285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134013310 con 0x55f628823bb0 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc130005020 con 0x55f628823bb0 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc11effd640 1 -- 172.21.15.28:0/397430285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fc13400e7b0 con 0x55f628823bb0 2024-09-15T21:57:12.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc11effd640 1 -- 172.21.15.28:0/397430285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc13402b050 con 0x55f628823bb0 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 10 client.15282 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f6288363a0 con 0x55f628823bb0 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc11effd640 1 -- 172.21.15.28:0/397430285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc130005020 con 0x55f628823bb0 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc11effd640 1 -- 172.21.15.28:0/397430285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc134011b10 con 0x55f628823bb0 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 10 client.15282 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 10 client.15282 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 20 client.15282 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288389c0 0x55f628839e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 4 client.15282 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.059+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f62878c600 con 0x55f6288389c0 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.060+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288389c0 0x55f628839e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.060+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288389c0 0x55f628839e30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5794464830ff9e3 server_cookie=8d95b66909a0d220 in_seq=0 out_seq=0 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.060+0000 7fc11effd640 10 client.15282 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc11effd640 1 -- 172.21.15.28:0/397430285 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f62878c600 con 0x55f6288389c0 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc11effd640 10 client.15282 handle_command_reply: tid=0 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 client.15282 shutdown 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 msgr2=0x55f628820bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628820bf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc1340107b0 tx=0x7fc13400bbb0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 shutdown_connections 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288389c0 0x55f628839e30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f628823bb0 0x55f628820bf0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288192f0 0x55f6288206b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 --2- 172.21.15.28:0/397430285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288189d0 0x55f628823360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 >> 172.21.15.28:0/397430285 conn(0x55f628814f60 msgr2=0x55f62881ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:12.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc11effd640 0 client.15282 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:12.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 shutdown_connections 2024-09-15T21:57:12.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 1 -- 172.21.15.28:0/397430285 wait complete. 2024-09-15T21:57:12.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.061+0000 7fc147f37740 20 client.15282 trim_cache size 0 max 16384 2024-09-15T21:57:12.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.082+0000 7fc147f37740 1 Processor -- start 2024-09-15T21:57:12.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc147f37740 1 -- start start 2024-09-15T21:57:12.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc147f37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f6288382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc147f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288192f0 0x55f628839760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc147f37740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f628823bb0 0x55f62881ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc147f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f628838170 con 0x55f6288192f0 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc147f37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f628838840 con 0x55f6288189d0 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc147f37740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f628838f10 con 0x55f628823bb0 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288192f0 0x55f628839760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288192f0 0x55f628839760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35306/0 (socket says 172.21.15.28:35306) 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 -- 172.21.15.28:0/3848040648 learned_addr learned my addr 172.21.15.28:0/3848040648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13effd640 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f6288382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13effd640 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f628823bb0 0x55f62881ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 -- 172.21.15.28:0/3848040648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f628823bb0 msgr2=0x55f62881ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f628823bb0 0x55f62881ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 -- 172.21.15.28:0/3848040648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 msgr2=0x55f6288382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f6288382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 -- 172.21.15.28:0/3848040648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f628814d90 con 0x55f6288192f0 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13effd640 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f6288382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.083+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288192f0 0x55f628839760 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc130004650 tx=0x7fc130005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2dad74faf8c96a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc1257fa640 1 -- 172.21.15.28:0/3848040648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130054070 con 0x55f6288192f0 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc1257fa640 1 -- 172.21.15.28:0/3848040648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc130004350 con 0x55f6288192f0 2024-09-15T21:57:12.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc1257fa640 1 -- 172.21.15.28:0/3848040648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130040040 con 0x55f6288192f0 2024-09-15T21:57:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 -- 172.21.15.28:0/3848040648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288192f0 msgr2=0x55f628839760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288192f0 0x55f628839760 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc130004650 tx=0x7fc130005fd0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 -- 172.21.15.28:0/3848040648 shutdown_connections 2024-09-15T21:57:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f628823bb0 0x55f62881ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288192f0 0x55f628839760 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 --2- 172.21.15.28:0/3848040648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6288189d0 0x55f6288382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 -- 172.21.15.28:0/3848040648 >> 172.21.15.28:0/3848040648 conn(0x55f628814f60 msgr2=0x55f628815340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.084+0000 7fc147f37740 1 -- 172.21.15.28:0/3848040648 shutdown_connections 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 -- 172.21.15.28:0/3848040648 wait complete. 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 -- start start 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 Processor -- start 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288189d0 0x55f628842910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f62883cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f628823bb0 0x55f62883d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f628839cb0 con 0x55f6288189d0 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6288395e0 con 0x55f628823bb0 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f62883a380 con 0x55f6288192f0 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc147f37740 10 client.0 _ll_register_callbacks cb 0x7fc1340103f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:12.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288189d0 0x55f628842910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc13effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288189d0 0x55f628842910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35318/0 (socket says 172.21.15.28:35318) 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.085+0000 7fc13effd640 1 -- 172.21.15.28:0/1303054146 learned_addr learned my addr 172.21.15.28:0/1303054146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f62883cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13effd640 1 -- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 msgr2=0x55f62883cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13effd640 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f62883cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13effd640 1 -- 172.21.15.28:0/1303054146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f628823bb0 msgr2=0x55f62883d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13effd640 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f628823bb0 0x55f62883d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13effd640 1 -- 172.21.15.28:0/1303054146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f62883db70 con 0x55f6288189d0 2024-09-15T21:57:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f62883cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc13effd640 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288189d0 0x55f628842910 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc134015010 tx=0x7fc13400ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc068b045340772 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc11effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc11effd640 1 -- 172.21.15.28:0/1303054146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134035070 con 0x55f6288189d0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc11effd640 1 -- 172.21.15.28:0/1303054146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fc134013320 con 0x55f6288189d0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc130004350 con 0x55f6288189d0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc11effd640 1 -- 172.21.15.28:0/1303054146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc134041050 con 0x55f6288189d0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc147f37740 10 client.15288 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.086+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f6288a2100 con 0x55f6288189d0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc11effd640 1 -- 172.21.15.28:0/1303054146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc130004350 con 0x55f6288189d0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc11effd640 1 -- 172.21.15.28:0/1303054146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fc134005470 con 0x55f6288189d0 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc147f37740 10 client.15288 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc147f37740 10 client.15288 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T21:57:12.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc147f37740 20 client.15288 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc147f37740 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288a6c50 0x55f6288a5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc147f37740 4 client.15288 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "status"}) -- 0x55f628786b10 con 0x55f6288a6c50 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.087+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288a6c50 0x55f6288a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc13f7fe640 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288a6c50 0x55f6288a5960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=47c3d807762998e1 server_cookie=5d037c03ab794b2a in_seq=0 out_seq=0 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc11effd640 10 client.15288 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc11effd640 1 -- 172.21.15.28:0/1303054146 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f628786b10 con 0x55f6288a6c50 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc11effd640 10 client.15288 handle_command_reply: tid=0 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 client.15288 shutdown 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288189d0 msgr2=0x55f628842910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288189d0 0x55f628842910 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc134015010 tx=0x7fc13400ba00 comp rx=0 tx=0).stop 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 shutdown_connections 2024-09-15T21:57:12.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6288a6c50 0x55f6288a5960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f628823bb0 0x55f62883d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6288192f0 0x55f62883cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc11effd640 0 client.15288 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 --2- 172.21.15.28:0/1303054146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6288189d0 0x55f628842910 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 >> 172.21.15.28:0/1303054146 conn(0x55f628814f60 msgr2=0x55f628820670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 shutdown_connections 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 1 -- 172.21.15.28:0/1303054146 wait complete. 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.088+0000 7fc147f37740 20 client.15288 trim_cache size 0 max 16384 2024-09-15T21:57:12.096 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24481, 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 97.559837170999998, 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-09-15T21:57:12.097 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:57:12.098 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 152.792791693 2024-09-15T21:57:12.098 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc1280a3240 msgr2=0x7fc1280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc1280a3240 0x7fc1280a5700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc13800efd0 tx=0x7fc138066b60 comp rx=0 tx=0).stop 2024-09-15T21:57:12.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14010c170 msgr2=0x7fc14019cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14010c170 0x7fc14019cd10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc13800eaa0 tx=0x7fc13800ef60 comp rx=0 tx=0).stop 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 shutdown_connections 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fc1280a3240 0x7fc1280a5700 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14010c170 0x7fc14019cd10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc140103660 0x7fc140198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 --2- 172.21.15.28:0/2991761416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc140102d40 0x7fc140198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 >> 172.21.15.28:0/2991761416 conn(0x7fc1400fc7a0 msgr2=0x7fc1400fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 shutdown_connections 2024-09-15T21:57:12.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:12.089+0000 7fc145b5e640 1 -- 172.21.15.28:0/2991761416 wait complete. 2024-09-15T21:57:12.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T21:57:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:13.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:12 smithi028 ceph-mon[30969]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:13.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:12 smithi203 ceph-mon[35546]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:13.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:12 smithi174 ceph-mon[35244]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 -- 172.21.15.28:0/942880740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c102570 msgr2=0x7f6d0c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 --2- 172.21.15.28:0/942880740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c102570 0x7f6d0c1029b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009900 tx=0x7f6cf402a480 comp rx=0 tx=0).stop 2024-09-15T21:57:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 -- 172.21.15.28:0/942880740 shutdown_connections 2024-09-15T21:57:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 --2- 172.21.15.28:0/942880740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d0c102ef0 0x7f6d0c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 --2- 172.21.15.28:0/942880740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c102570 0x7f6d0c1029b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 --2- 172.21.15.28:0/942880740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c100e20 0x7f6d0c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 -- 172.21.15.28:0/942880740 >> 172.21.15.28:0/942880740 conn(0x7f6d0c0fc7d0 msgr2=0x7f6d0c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 -- 172.21.15.28:0/942880740 shutdown_connections 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 -- 172.21.15.28:0/942880740 wait complete. 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.565+0000 7f6d113ce640 1 Processor -- start 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d113ce640 1 -- start start 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d113ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c100e20 0x7f6d0c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d113ce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d0c102570 0x7f6d0c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d113ce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c102ef0 0x7f6d0c1a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d113ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c110f50 con 0x7f6d0c100e20 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d113ce640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c110dd0 con 0x7f6d0c102570 2024-09-15T21:57:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d113ce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c1110d0 con 0x7f6d0c102ef0 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c100e20 0x7f6d0c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c100e20 0x7f6d0c199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35346/0 (socket says 172.21.15.28:35346) 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 -- 172.21.15.28:0/3256277061 learned_addr learned my addr 172.21.15.28:0/3256277061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d03fff640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d0c102570 0x7f6d0c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c102ef0 0x7f6d0c1a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 -- 172.21.15.28:0/3256277061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c102ef0 msgr2=0x7f6d0c1a43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c102ef0 0x7f6d0c1a43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 -- 172.21.15.28:0/3256277061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d0c102570 msgr2=0x7f6d0c1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d0c102570 0x7f6d0c1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.566+0000 7f6d0affd640 1 -- 172.21.15.28:0/3256277061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c1a4920 con 0x7f6d0c100e20 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.567+0000 7f6d0affd640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c100e20 0x7f6d0c199360 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00d950 tx=0x7f6cfc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.567+0000 7f6d08ff9640 1 -- 172.21.15.28:0/3256277061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc017030 con 0x7f6d0c100e20 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.567+0000 7f6d08ff9640 1 -- 172.21.15.28:0/3256277061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cfc004020 con 0x7f6d0c100e20 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.567+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c1a4af0 con 0x7f6d0c100e20 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.567+0000 7f6d08ff9640 1 -- 172.21.15.28:0/3256277061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc0047a0 con 0x7f6d0c100e20 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.567+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0c1a5000 con 0x7f6d0c100e20 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.569+0000 7f6d137a7740 1 Processor -- start 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.569+0000 7f6d08ff9640 1 -- 172.21.15.28:0/3256277061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6cfc004940 con 0x7f6d0c100e20 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.569+0000 7f6d137a7740 1 -- start start 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4d8d0 0x563693c4dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c5ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d137a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563693bb4090 con 0x563693c4d8d0 2024-09-15T21:57:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d137a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563693b5ba70 con 0x563693c4e1f0 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d137a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563693bb0050 con 0x563693c5cfc0 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4d8d0 0x563693c4dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4d8d0 0x563693c4dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35354/0 (socket says 172.21.15.28:35354) 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/260662706 learned_addr learned my addr 172.21.15.28:0/260662706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d0affd640 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c5ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.570+0000 7f6d03fff640 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/260662706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 msgr2=0x563693c5f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/260662706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 msgr2=0x563693c5ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c5ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/260662706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563693b68130 con 0x563693c4d8d0 2024-09-15T21:57:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d08ff9640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6cf80a32c0 0x7f6cf80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4d8d0 0x563693c4dcb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009f70 tx=0x7f6d04010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8ccea78c75d1b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d0affd640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6cf80a32c0 0x7f6cf80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d08ff9640 1 -- 172.21.15.28:0/3256277061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f6cfc0cf600 con 0x7f6d0c100e20 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d017fa640 1 -- 172.21.15.28:0/260662706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04016070 con 0x563693c4d8d0 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d017fa640 1 -- 172.21.15.28:0/260662706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d04004d20 con 0x563693c4d8d0 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.571+0000 7f6d017fa640 1 -- 172.21.15.28:0/260662706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04012850 con 0x563693c4d8d0 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 -- 172.21.15.28:0/260662706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4d8d0 msgr2=0x563693c4dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d0affd640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6cf80a32c0 0x7f6cf80a5780 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009fd0 tx=0x7f6cfc0d50e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4d8d0 0x563693c4dcb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009f70 tx=0x7f6d04010800 comp rx=0 tx=0).stop 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 -- 172.21.15.28:0/260662706 shutdown_connections 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c5ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 --2- 172.21.15.28:0/260662706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4d8d0 0x563693c4dcb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 -- 172.21.15.28:0/260662706 >> 172.21.15.28:0/260662706 conn(0x563693c49f60 msgr2=0x563693c4a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 -- 172.21.15.28:0/260662706 shutdown_connections 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 -- 172.21.15.28:0/260662706 wait complete. 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.572+0000 7f6d137a7740 1 -- start start 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 1 Processor -- start 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4d8d0 0x563693c5c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4e1f0 0x563693c59bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563693c5fbf0 con 0x563693c4e1f0 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563693b6ef20 con 0x563693c4d8d0 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563693c5fd70 con 0x563693c5cfc0 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d137a7740 10 client.0 _ll_register_callbacks cb 0x563693b38030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4e1f0 0x563693c59bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4e1f0 0x563693c59bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35364/0 (socket says 172.21.15.28:35364) 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 -- 172.21.15.28:0/1421896095 learned_addr learned my addr 172.21.15.28:0/1421896095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4d8d0 0x563693c5c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 -- 172.21.15.28:0/1421896095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 msgr2=0x563693c5a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 -- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4d8d0 msgr2=0x563693c5c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4d8d0 0x563693c5c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 -- 172.21.15.28:0/1421896095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636939252b0 con 0x563693c4e1f0 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4d8d0 0x563693c5c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6d03fff640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4e1f0 0x563693c59bc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4000c00 tx=0x7f6cf4004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f892f1f18c4c556b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.573+0000 7f6cf27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6cf27fc640 1 -- 172.21.15.28:0/1421896095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4035040 con 0x563693c4e1f0 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6cf27fc640 1 -- 172.21.15.28:0/1421896095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f6cf4003ac0 con 0x563693c4e1f0 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d04004d20 con 0x563693c4e1f0 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6cf27fc640 1 -- 172.21.15.28:0/1421896095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6cf404e050 con 0x563693c4e1f0 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 10 client.15300 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563693c6b550 con 0x563693c4e1f0 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6cf27fc640 1 -- 172.21.15.28:0/1421896095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d04004d20 con 0x563693c4e1f0 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6cf27fc640 1 -- 172.21.15.28:0/1421896095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f6cf403c4d0 con 0x563693c4e1f0 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 10 client.15300 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 10 client.15300 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 20 client.15300 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693c6dba0 0x563693c6f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 4 client.15300 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.574+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563693bc1600 con 0x563693c6dba0 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.575+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693c6dba0 0x563693c6f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.575+0000 7f6cf27fc640 10 client.15300 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.575+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693c6dba0 0x563693c6f010 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51eaab194364dc7f server_cookie=1ff8c60a8b4142a1 in_seq=0 out_seq=0 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6cf27fc640 1 -- 172.21.15.28:0/1421896095 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563693bc1600 con 0x563693c6dba0 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6cf27fc640 10 client.15300 handle_command_reply: tid=0 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 client.15300 shutdown 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4e1f0 msgr2=0x563693c59bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4e1f0 0x563693c59bc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4000c00 tx=0x7f6cf4004440 comp rx=0 tx=0).stop 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 shutdown_connections 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693c6dba0 0x563693c6f010 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c5cfc0 0x563693c5a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c4e1f0 0x563693c59bc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6cf27fc640 0 client.15300 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1421896095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4d8d0 0x563693c5c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 >> 172.21.15.28:0/1421896095 conn(0x563693c49f60 msgr2=0x563693c4fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 shutdown_connections 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 1 -- 172.21.15.28:0/1421896095 wait complete. 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.576+0000 7f6d137a7740 20 client.15300 trim_cache size 0 max 16384 2024-09-15T21:57:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.594+0000 7f6d137a7740 1 Processor -- start 2024-09-15T21:57:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.594+0000 7f6d137a7740 1 -- start start 2024-09-15T21:57:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 0x563693c55130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c56e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693c57360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d137a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563693c6d350 con 0x563693c5cfc0 2024-09-15T21:57:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d137a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563693c6da20 con 0x563693c4e1f0 2024-09-15T21:57:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d137a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563693c6e0f0 con 0x563693c4d8d0 2024-09-15T21:57:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693c57360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693c57360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35368/0 (socket says 172.21.15.28:35368) 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/1028910813 learned_addr learned my addr 172.21.15.28:0/1028910813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 0x563693c55130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d03fff640 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c56e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/1028910813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 msgr2=0x563693c55130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 0x563693c55130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/1028910813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 msgr2=0x563693c56e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c56e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/1028910813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563693c4b0f0 con 0x563693c5cfc0 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693c57360 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009ec0 tx=0x7f6d04014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9fd0ea641c19ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d00ff9640 1 -- 172.21.15.28:0/1028910813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04016040 con 0x563693c5cfc0 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.595+0000 7f6d00ff9640 1 -- 172.21.15.28:0/1028910813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d04012c10 con 0x563693c5cfc0 2024-09-15T21:57:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d00ff9640 1 -- 172.21.15.28:0/1028910813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04023040 con 0x563693c5cfc0 2024-09-15T21:57:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 -- 172.21.15.28:0/1028910813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 msgr2=0x563693c57360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693c57360 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009ec0 tx=0x7f6d04014040 comp rx=0 tx=0).stop 2024-09-15T21:57:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 -- 172.21.15.28:0/1028910813 shutdown_connections 2024-09-15T21:57:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693c57360 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c56e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 --2- 172.21.15.28:0/1028910813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 0x563693c55130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 -- 172.21.15.28:0/1028910813 >> 172.21.15.28:0/1028910813 conn(0x563693c49f60 msgr2=0x563693c54a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 -- 172.21.15.28:0/1028910813 shutdown_connections 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.596+0000 7f6d137a7740 1 -- 172.21.15.28:0/1028910813 wait complete. 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 -- start start 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 Processor -- start 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 0x563693c70800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c70d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693cd62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563693c6ee90 con 0x563693c5cfc0 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563693c6e7c0 con 0x563693c4e1f0 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563693c6f560 con 0x563693c4d8d0 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d137a7740 10 client.0 _ll_register_callbacks cb 0x7f6cf4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693cd62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693cd62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35374/0 (socket says 172.21.15.28:35374) 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/3363755442 learned_addr learned my addr 172.21.15.28:0/3363755442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d03fff640 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c70d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/3363755442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 msgr2=0x563693c70800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 0x563693c70800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 msgr2=0x563693c70d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.597+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c70d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6d0b7fe640 1 -- 172.21.15.28:0/3363755442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563693cd6860 con 0x563693c5cfc0 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6d03fff640 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c70d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6d0b7fe640 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693cd62a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6d04000f80 tx=0x7f6d04003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e3c44b614abf06 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6cf27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6cf27fc640 1 -- 172.21.15.28:0/3363755442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d04039020 con 0x563693c5cfc0 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d04012c10 con 0x563693c5cfc0 2024-09-15T21:57:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6cf27fc640 1 -- 172.21.15.28:0/3363755442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f6d04003b70 con 0x563693c5cfc0 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6cf27fc640 1 -- 172.21.15.28:0/3363755442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6d04041050 con 0x563693c5cfc0 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6d137a7740 10 client.15306 fetch_fsmap learned FSMap version 28 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563693cd7b40 con 0x563693c5cfc0 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.598+0000 7f6cf27fc640 1 -- 172.21.15.28:0/3363755442 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d04012c10 con 0x563693c5cfc0 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6cf27fc640 1 -- 172.21.15.28:0/3363755442 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f6d04030050 con 0x563693c5cfc0 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d137a7740 10 client.15306 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d137a7740 10 client.15306 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d137a7740 20 client.15306 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d137a7740 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693cdbf90 0x563693cdaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d137a7740 4 client.15306 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "status"}) -- 0x563693bbbb10 con 0x563693cdbf90 2024-09-15T21:57:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d03fff640 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693cdbf90 0x563693cdaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6d03fff640 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693cdbf90 0x563693cdaca0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8284f596441d10c8 server_cookie=5922de9c1bffd0c5 in_seq=0 out_seq=0 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6cf27fc640 10 client.15306 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6cf27fc640 1 -- 172.21.15.28:0/3363755442 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563693bbbb10 con 0x563693cdbf90 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.599+0000 7f6cf27fc640 10 client.15306 handle_command_reply: tid=0 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 client.15306 shutdown 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 msgr2=0x563693cd62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693cd62a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6d04000f80 tx=0x7f6d04003f20 comp rx=0 tx=0).stop 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 shutdown_connections 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563693cdbf90 0x563693cdaca0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563693c5cfc0 0x563693cd62a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563693c4e1f0 0x563693c70d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6cf27fc640 0 client.15306 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 --2- 172.21.15.28:0/3363755442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563693c4d8d0 0x563693c70800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 >> 172.21.15.28:0/3363755442 conn(0x563693c49f60 msgr2=0x563693c55240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 shutdown_connections 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 1 -- 172.21.15.28:0/3363755442 wait complete. 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d137a7740 20 client.15306 trim_cache size 0 max 16384 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24485, 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:13.607 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 134.175350511, 2024-09-15T21:57:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 28, 2024-09-15T21:57:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-09-15T21:57:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T21:57:13.608 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 145.32409064699999 2024-09-15T21:57:13.608 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6cf80a32c0 msgr2=0x7f6cf80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6cf80a32c0 0x7f6cf80a5780 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009fd0 tx=0x7f6cfc0d50e0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c100e20 msgr2=0x7f6d0c199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c100e20 0x7f6d0c199360 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00d950 tx=0x7f6cfc00de10 comp rx=0 tx=0).stop 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 shutdown_connections 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f6cf80a32c0 0x7f6cf80a5780 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d0c102ef0 0x7f6d0c1a43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d0c102570 0x7f6d0c1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 --2- 172.21.15.28:0/3256277061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d0c100e20 0x7f6d0c199360 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.600+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 >> 172.21.15.28:0/3256277061 conn(0x7f6d0c0fc7d0 msgr2=0x7f6d0c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.601+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 shutdown_connections 2024-09-15T21:57:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:13.601+0000 7f6d113ce640 1 -- 172.21.15.28:0/3256277061 wait complete. 2024-09-15T21:57:14.236 INFO:teuthology.run_tasks:Running task sequential... 2024-09-15T21:57:14.245 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-15T21:57:14.245 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-15T21:57:14.246 INFO:teuthology.task.print:Enabling mgr modules 2024-09-15T21:57:14.246 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-09-15T21:57:14.246 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T21:57:14.247 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:57:14.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-09-15T21:57:14.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.351+0000 7f54d1e17640 1 -- 172.21.15.28:0/3356897683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1001e0 msgr2=0x7f54cc1005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.351+0000 7f54d1e17640 1 --2- 172.21.15.28:0/3356897683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1001e0 0x7f54cc1005e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f54bc009900 tx=0x7f54bc02a410 comp rx=0 tx=0).stop 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.351+0000 7f54d1e17640 1 -- 172.21.15.28:0/3356897683 shutdown_connections 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.351+0000 7f54d1e17640 1 --2- 172.21.15.28:0/3356897683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54cc101d90 0x7f54cc108620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.351+0000 7f54d1e17640 1 --2- 172.21.15.28:0/3356897683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54cc1013d0 0x7f54cc101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.351+0000 7f54d1e17640 1 --2- 172.21.15.28:0/3356897683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1001e0 0x7f54cc1005e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.351+0000 7f54d1e17640 1 -- 172.21.15.28:0/3356897683 >> 172.21.15.28:0/3356897683 conn(0x7f54cc0fb980 msgr2=0x7f54cc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.352+0000 7f54d1e17640 1 -- 172.21.15.28:0/3356897683 shutdown_connections 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.352+0000 7f54d1e17640 1 -- 172.21.15.28:0/3356897683 wait complete. 2024-09-15T21:57:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.352+0000 7f54d1e17640 1 Processor -- start 2024-09-15T21:57:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.352+0000 7f54d1e17640 1 -- start start 2024-09-15T21:57:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54cc1001e0 0x7f54cc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1013d0 0x7f54cc1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54cc101d90 0x7f54cc19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc10d8d0 con 0x7f54cc1013d0 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc10d750 con 0x7f54cc1001e0 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc10da50 con 0x7f54cc101d90 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1013d0 0x7f54cc1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1013d0 0x7f54cc1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35380/0 (socket says 172.21.15.28:35380) 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 -- 172.21.15.28:0/693521204 learned_addr learned my addr 172.21.15.28:0/693521204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54cb7fe640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54cc1001e0 0x7f54cc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54cbfff640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54cc101d90 0x7f54cc19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 -- 172.21.15.28:0/693521204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54cc101d90 msgr2=0x7f54cc19f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54cc101d90 0x7f54cc19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 -- 172.21.15.28:0/693521204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54cc1001e0 msgr2=0x7f54cc197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54cc1001e0 0x7f54cc197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 -- 172.21.15.28:0/693521204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54cc19fbd0 con 0x7f54cc1013d0 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54cb7fe640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54cc1001e0 0x7f54cc197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54caffd640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1013d0 0x7f54cc1984f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f54c000b4a0 tx=0x7f54c000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54c8ff9640 1 -- 172.21.15.28:0/693521204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0014070 con 0x7f54cc1013d0 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54c8ff9640 1 -- 172.21.15.28:0/693521204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c0010af0 con 0x7f54cc1013d0 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54cc19fec0 con 0x7f54cc1013d0 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54c8ff9640 1 -- 172.21.15.28:0/693521204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0017d10 con 0x7f54cc1013d0 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.353+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54cc1a0450 con 0x7f54cc1013d0 2024-09-15T21:57:14.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.355+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54cc107620 con 0x7f54cc1013d0 2024-09-15T21:57:14.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.356+0000 7f54c8ff9640 1 -- 172.21.15.28:0/693521204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f54c0017070 con 0x7f54cc1013d0 2024-09-15T21:57:14.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.357+0000 7f54c8ff9640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54b40a3370 0x7f54b40a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:14.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.357+0000 7f54cb7fe640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54b40a3370 0x7f54b40a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:14.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.357+0000 7f54c8ff9640 1 -- 172.21.15.28:0/693521204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f54c00cf070 con 0x7f54cc1013d0 2024-09-15T21:57:14.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.358+0000 7f54cb7fe640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54b40a3370 0x7f54b40a5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f54bc000c00 tx=0x7f54bc031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:14.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.361+0000 7f54c8ff9640 1 -- 172.21.15.28:0/693521204 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f54c009b9b0 con 0x7f54cc1013d0 2024-09-15T21:57:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.590+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f54cc062e90 con 0x7f54cc1013d0 2024-09-15T21:57:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.947+0000 7f54c8ff9640 1 -- 172.21.15.28:0/693521204 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f54c0095e20 con 0x7f54cc1013d0 2024-09-15T21:57:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54b40a3370 msgr2=0x7f54b40a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54b40a3370 0x7f54b40a5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f54bc000c00 tx=0x7f54bc031400 comp rx=0 tx=0).stop 2024-09-15T21:57:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1013d0 msgr2=0x7f54cc1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1013d0 0x7f54cc1984f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f54c000b4a0 tx=0x7f54c000b960 comp rx=0 tx=0).stop 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 shutdown_connections 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f54b40a3370 0x7f54b40a5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54cc101d90 0x7f54cc19f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc1013d0 0x7f54cc1984f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 --2- 172.21.15.28:0/693521204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54cc1001e0 0x7f54cc197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 >> 172.21.15.28:0/693521204 conn(0x7f54cc0fb980 msgr2=0x7f54cc1047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.950+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 shutdown_connections 2024-09-15T21:57:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:14.951+0000 7f54d1e17640 1 -- 172.21.15.28:0/693521204 wait complete. 2024-09-15T21:57:14.970 INFO:teuthology.run_tasks:Running task kclient... 2024-09-15T21:57:14.985 INFO:tasks.kclient:Mounting kernel clients... 2024-09-15T21:57:14.986 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-15T21:57:14.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:14.986 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-15T21:57:15.003 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-15T21:57:15.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:15.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T21:57:15.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:15.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-15T21:57:15.187 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-09-15T21:57:15.187 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-15T21:57:15.187 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:15.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-15T21:57:15.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:14 smithi028 ceph-mon[30969]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:15.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:14 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/693521204' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-15T21:57:15.214 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ignoring --setuser ceph since I am not root 2024-09-15T21:57:15.214 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ignoring --setgroup ceph since I am not root 2024-09-15T21:57:15.214 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:15.111+0000 7fb9a7dd1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:57:15.214 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:15.172+0000 7fb9a7dd1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:57:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:14 smithi203 ceph-mon[35546]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:14 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/693521204' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-15T21:57:15.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: ignoring --setuser ceph since I am not root 2024-09-15T21:57:15.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: ignoring --setgroup ceph since I am not root 2024-09-15T21:57:15.219 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-mgr[37357]: -- 172.21.15.203:0/1962183731 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5577801fd0e0 con 0x5577801db400 2024-09-15T21:57:15.219 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:15.103+0000 7f14f25e0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:57:15.219 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:15.159+0000 7f14f25e0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:57:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:14 smithi174 ceph-mon[35244]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T21:57:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:14 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/693521204' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-09-15T21:57:15.260 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: ignoring --setuser ceph since I am not root 2024-09-15T21:57:15.260 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: ignoring --setgroup ceph since I am not root 2024-09-15T21:57:15.261 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-mgr[36610]: -- 172.21.15.174:0/2013516393 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e89448b0e0 con 0x55e894469400 2024-09-15T21:57:15.261 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:15.101+0000 7f489e441040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T21:57:15.261 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:15.158+0000 7f489e441040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T21:57:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:15.577 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:15.336+0000 7fb9a7dd1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:57:15.578 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:15.414+0000 7fb9a7dd1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:57:15.578 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:15.576+0000 7fb9a7dd1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:57:15.664 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:15.306+0000 7f14f25e0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:57:15.664 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:15.372+0000 7f14f25e0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:57:15.664 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:15.521+0000 7f14f25e0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:57:15.664 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:15.662+0000 7f14f25e0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:57:15.665 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:15.306+0000 7f489e441040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T21:57:15.665 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:15.372+0000 7f489e441040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T21:57:15.665 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:15.520+0000 7f489e441040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:57:15.871 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:15.727+0000 7fb9a7dd1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:57:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1879409641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 msgr2=0x7fb4b8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1879409641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b8103110 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8009930 tx=0x7fb4a802a420 comp rx=0 tx=0).stop 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1879409641 shutdown_connections 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1879409641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4b81082a0 0x7fb4b810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1879409641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4b81036e0 0x7fb4b8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1879409641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b8103110 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1879409641 >> 172.21.15.28:0/1879409641 conn(0x7fb4b80fc7d0 msgr2=0x7fb4b80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1879409641 shutdown_connections 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.953+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1879409641 wait complete. 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 Processor -- start 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 -- start start 2024-09-15T21:57:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4b81036e0 0x7fb4b8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4b81082a0 0x7fb4b819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b81120e0 con 0x7fb4b8102d30 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b8111f60 con 0x7fb4b81082a0 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4bcf64640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b8112260 con 0x7fb4b81036e0 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4b659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4b659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b81985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35412/0 (socket says 172.21.15.28:35412) 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4b659c640 1 -- 172.21.15.28:0/1865303203 learned_addr learned my addr 172.21.15.28:0/1865303203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4add9b640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4b81036e0 0x7fb4b8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.954+0000 7fb4b6d9d640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4b81082a0 0x7fb4b819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4b659c640 1 -- 172.21.15.28:0/1865303203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4b81036e0 msgr2=0x7fb4b8198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4b659c640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4b81036e0 0x7fb4b8198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4b659c640 1 -- 172.21.15.28:0/1865303203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4b81082a0 msgr2=0x7fb4b819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4b659c640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4b81082a0 0x7fb4b819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4b659c640 1 -- 172.21.15.28:0/1865303203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b819d550 con 0x7fb4b8102d30 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4add9b640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4b81036e0 0x7fb4b8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4b659c640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b81985a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb4a802a940 tx=0x7fb4a8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4affff640 1 -- 172.21.15.28:0/1865303203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a803f070 con 0x7fb4b8102d30 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4affff640 1 -- 172.21.15.28:0/1865303203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4a8031c90 con 0x7fb4b8102d30 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b819d7e0 con 0x7fb4b8102d30 2024-09-15T21:57:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4affff640 1 -- 172.21.15.28:0/1865303203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8035030 con 0x7fb4b8102d30 2024-09-15T21:57:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.955+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b81a5120 con 0x7fb4b8102d30 2024-09-15T21:57:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.956+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b8103110 con 0x7fb4b8102d30 2024-09-15T21:57:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.957+0000 7fb4affff640 1 -- 172.21.15.28:0/1865303203 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 138372+0+0 (secure 0 0 0) 0x7fb4a8042070 con 0x7fb4b8102d30 2024-09-15T21:57:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.958+0000 7fb4affff640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 0x7fb4a40a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.958+0000 7fb4add9b640 1 -- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 msgr2=0x7fb4a40a5750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1670367809 2024-09-15T21:57:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.958+0000 7fb4affff640 1 -- 172.21.15.28:0/1865303203 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fb4a80f0a90 con 0x7fb4b8102d30 2024-09-15T21:57:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.958+0000 7fb4add9b640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 0x7fb4a40a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:57:15.967 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:15.663+0000 7f489e441040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T21:57:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:15.961+0000 7fb4affff640 1 -- 172.21.15.28:0/1865303203 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb4a80bcde0 con 0x7fb4b8102d30 2024-09-15T21:57:16.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/693521204' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-15T21:57:16.131 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:15 smithi028 ceph-mon[30969]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-15T21:57:16.131 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:16 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:16.065+0000 7fb9a7dd1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:57:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.159+0000 7fb4add9b640 1 -- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 msgr2=0x7fb4a40a5750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1670367809 2024-09-15T21:57:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.159+0000 7fb4add9b640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 0x7fb4a40a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T21:57:16.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.180+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb4b810a2b0 con 0x7fb4b8102d30 2024-09-15T21:57:16.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.180+0000 7fb4affff640 1 -- 172.21.15.28:0/1865303203 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7fb4a80c10e0 con 0x7fb4b8102d30 2024-09-15T21:57:16.182 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-15T21:57:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 msgr2=0x7fb4a40a5750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:57:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 0x7fb4a40a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 msgr2=0x7fb4b81985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b81985a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb4a802a940 tx=0x7fb4a8003b90 comp rx=0 tx=0).stop 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 shutdown_connections 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fb4a40a3290 0x7fb4a40a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4b81082a0 0x7fb4b819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4b81036e0 0x7fb4b8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.182+0000 7fb4bcf64640 1 --2- 172.21.15.28:0/1865303203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b8102d30 0x7fb4b81985a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.183+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 >> 172.21.15.28:0/1865303203 conn(0x7fb4b80fc7d0 msgr2=0x7fb4b8106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.183+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 shutdown_connections 2024-09-15T21:57:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:16.183+0000 7fb4bcf64640 1 -- 172.21.15.28:0/1865303203 wait complete. 2024-09-15T21:57:16.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/693521204' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-15T21:57:16.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-mon[35546]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-15T21:57:16.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:15.978+0000 7f14f25e0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:57:16.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:16 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:16.039+0000 7f14f25e0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:57:16.221 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/693521204' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-09-15T21:57:16.221 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-mon[35244]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-15T21:57:16.221 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:15.976+0000 7f489e441040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T21:57:16.221 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:16 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:16.036+0000 7f489e441040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:57:16.221 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:16 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:16.219+0000 7f489e441040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:57:16.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:16 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:16.129+0000 7fb9a7dd1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T21:57:16.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:16 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:16.335+0000 7fb9a7dd1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:57:16.718 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:16 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:16.222+0000 7f14f25e0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T21:57:16.855 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-15T21:57:16.855 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-15T21:57:16.856 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-15T21:57:16.856 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-15T21:57:16.856 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:16.856 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-15T21:57:16.856 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-15T21:57:16.856 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-15T21:57:16.856 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-15T21:57:16.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:16.856 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-15T21:57:16.912 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-15T21:57:16.912 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-15T21:57:16.913 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42162sec preferred_lft 42162sec 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-15T21:57:16.914 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-15T21:57:16.914 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-15T21:57:16.915 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:57:16.915 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:57:16.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-09-15T21:57:16.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-09-15T21:57:16.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-09-15T21:57:16.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-15T21:57:16.915 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:57:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:57:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:17.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:17.059 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-15T21:57:17.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:16 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1865303203' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T21:57:17.085 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-15T21:57:17.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:17.086 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-15T21:57:17.144 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-15T21:57:17.144 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-15T21:57:17.144 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-15T21:57:17.144 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-15T21:57:17.144 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-15T21:57:17.145 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:57:17.145 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:57:17.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-15T21:57:17.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-15T21:57:17.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-15T21:57:17.145 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:57:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:16 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1865303203' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T21:57:17.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:17 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:17.105+0000 7f14f25e0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:57:17.219 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:17 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:17.173+0000 7f14f25e0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:57:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:57:17.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:16 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1865303203' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T21:57:17.260 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:17 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:17.098+0000 7f489e441040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:57:17.260 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:17 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:17.165+0000 7f489e441040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:57:17.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:17.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:17.277 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-15T21:57:17.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:17.293 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-15T21:57:17.348 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:57:17.349 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:57:17.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-15T21:57:17.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-15T21:57:17.349 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:57:17.364 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:17 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:17.290+0000 7fb9a7dd1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T21:57:17.364 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:17 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:17.362+0000 7fb9a7dd1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T21:57:17.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:57:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:17.445 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-15T21:57:17.445 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:57:17.445 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:57:17.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-15T21:57:17.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-15T21:57:17.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-15T21:57:17.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-15T21:57:17.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-15T21:57:17.446 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:57:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:57:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:17.566 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T21:57:17.566 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-15T21:57:17.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-15T21:57:17.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-09-15T21:57:17.566 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-15T21:57:17.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:57:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:17.648 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T21:57:17.649 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:17.666 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-15T21:57:17.666 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T21:57:17.666 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:17.722 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-15T21:57:17.722 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-15T21:57:17.722 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/485a13b0-7b8e-4b77-ac19-edf83448c265 2024-09-15T21:57:17.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:17.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/485a13b0-7b8e-4b77-ac19-edf83448c265 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-09-15T21:57:17.807 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-09-15T21:57:17.807 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-09-15T21:57:17.808 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-15T21:57:17.955 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-09-15T21:57:17.955 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-09-15T21:57:17.955 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-09-15T21:57:17.955 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.174:3300,172.21.15.203:3300" 2024-09-15T21:57:17.956 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:3300,172.21.15.174:3300,172.21.15.203:3300:/volumes/qa/sv_1/485a13b0-7b8e-4b77-ac19-edf83448c265 2024-09-15T21:57:17.956 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=e78b7764-73ab-11ef-bceb-c7b262605968" will pass to kernel 2024-09-15T21:57:17.957 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-15T21:57:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-15T21:57:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-15T21:57:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-15T21:57:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-15T21:57:17.958 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-15T21:57:17.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T21:57:17.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:17.989 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T21:57:18.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T21:57:18.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:18.089 INFO:teuthology.orchestra.run.smithi028.stdout:client24976 2024-09-15T21:57:18.089 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:57:18.089 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T21:57:18.107 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T21:57:18.129 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:17 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:17.869+0000 7f489e441040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:57:18.129 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:17 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:17.984+0000 7f489e441040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:57:18.147 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:17 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:17.884+0000 7f14f25e0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:57:18.147 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.002+0000 7f14f25e0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:57:18.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T21:57:18.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:18.206 INFO:teuthology.orchestra.run.smithi028.stdout:client24976 2024-09-15T21:57:18.206 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T21:57:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T21:57:18.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:18.265 INFO:teuthology.orchestra.run.smithi028.stdout:client24976 2024-09-15T21:57:18.265 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e78b7764-73ab-11ef-bceb-c7b262605968.client24976/status') 2024-09-15T21:57:18.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.140+0000 7fb9a7dd1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T21:57:18.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.265+0000 7fb9a7dd1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T21:57:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T21:57:18.326 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.24976 (3)172.21.15.28:0/181312182 2024-09-15T21:57:18.326 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-09-15T21:57:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-09-15T21:57:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-09-15T21:57:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:70 bytes copied, 2.593e-05 s, 2.7 MB/s 2024-09-15T21:57:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T21:57:18.334 INFO:teuthology.run_tasks:Running task exec... 2024-09-15T21:57:18.343 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T21:57:18.343 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:57:18.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-09-15T21:57:18.407 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.127+0000 7f489e441040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:57:18.407 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.188+0000 7f489e441040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:57:18.407 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.338+0000 7f489e441040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:57:18.426 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.145+0000 7f14f25e0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:57:18.426 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.206+0000 7f14f25e0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:57:18.427 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.357+0000 7f14f25e0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:57:18.427 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.424+0000 7f14f25e0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:57:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 -- 172.21.15.28:0/3462581035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c101c40 msgr2=0x7fe73c1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 --2- 172.21.15.28:0/3462581035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c101c40 0x7fe73c1020c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe72c009900 tx=0x7fe72c02a410 comp rx=0 tx=0).stop 2024-09-15T21:57:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 -- 172.21.15.28:0/3462581035 shutdown_connections 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 --2- 172.21.15.28:0/3462581035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe73c102600 0x7fe73c108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 --2- 172.21.15.28:0/3462581035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c101c40 0x7fe73c1020c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 --2- 172.21.15.28:0/3462581035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe73c100a50 0x7fe73c100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 -- 172.21.15.28:0/3462581035 >> 172.21.15.28:0/3462581035 conn(0x7fe73c0fc1b0 msgr2=0x7fe73c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 -- 172.21.15.28:0/3462581035 shutdown_connections 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 -- 172.21.15.28:0/3462581035 wait complete. 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 Processor -- start 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.447+0000 7fe744037640 1 -- start start 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe744037640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe73c100a50 0x7fe73c19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe744037640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe73c101c40 0x7fe73c19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe744037640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c102600 0x7fe73c1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe744037640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c10c6b0 con 0x7fe73c102600 2024-09-15T21:57:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe744037640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c10c530 con 0x7fe73c101c40 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe744037640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c10c830 con 0x7fe73c100a50 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c102600 0x7fe73c1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c102600 0x7fe73c1a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35428/0 (socket says 172.21.15.28:35428) 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 -- 172.21.15.28:0/571076209 learned_addr learned my addr 172.21.15.28:0/571076209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe741dd3640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe73c100a50 0x7fe73c19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7415d2640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe73c101c40 0x7fe73c19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 -- 172.21.15.28:0/571076209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe73c100a50 msgr2=0x7fe73c19a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe73c100a50 0x7fe73c19a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 -- 172.21.15.28:0/571076209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe73c101c40 msgr2=0x7fe73c19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe73c101c40 0x7fe73c19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 -- 172.21.15.28:0/571076209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73c1a1bd0 con 0x7fe73c102600 2024-09-15T21:57:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe7425d4640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c102600 0x7fe73c1a1560 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe73400eb10 tx=0x7fe73400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.448+0000 7fe732ffd640 1 -- 172.21.15.28:0/571076209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734004b20 con 0x7fe73c102600 2024-09-15T21:57:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.449+0000 7fe732ffd640 1 -- 172.21.15.28:0/571076209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe734004cc0 con 0x7fe73c102600 2024-09-15T21:57:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.449+0000 7fe732ffd640 1 -- 172.21.15.28:0/571076209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734010760 con 0x7fe73c102600 2024-09-15T21:57:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.449+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73c1a1ec0 con 0x7fe73c102600 2024-09-15T21:57:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.449+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe73c1a2380 con 0x7fe73c102600 2024-09-15T21:57:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.450+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe73c1a2610 con 0x7fe73c102600 2024-09-15T21:57:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.451+0000 7fe732ffd640 1 -- 172.21.15.28:0/571076209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 138372+0+0 (secure 0 0 0) 0x7fe734010900 con 0x7fe73c102600 2024-09-15T21:57:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.452+0000 7fe732ffd640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 0x7fe7280a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.452+0000 7fe741dd3640 1 -- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 msgr2=0x7fe7280a5880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1670367809 2024-09-15T21:57:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.452+0000 7fe732ffd640 1 -- 172.21.15.28:0/571076209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7fe7340d0460 con 0x7fe73c102600 2024-09-15T21:57:18.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.452+0000 7fe741dd3640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 0x7fe7280a5880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:57:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.455+0000 7fe732ffd640 1 -- 172.21.15.28:0/571076209 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe7340541b0 con 0x7fe73c102600 2024-09-15T21:57:18.648 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.420+0000 7fb9a7dd1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T21:57:18.648 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.487+0000 7fb9a7dd1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T21:57:18.649 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.646+0000 7fb9a7dd1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T21:57:18.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.653+0000 7fe741dd3640 1 -- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 msgr2=0x7fe7280a5880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1670367809 2024-09-15T21:57:18.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.653+0000 7fe741dd3640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 0x7fe7280a5880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T21:57:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.673+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fe73c1a2a00 con 0x7fe73c102600 2024-09-15T21:57:18.689 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.405+0000 7f489e441040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:57:18.689 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.466+0000 7f489e441040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:57:18.689 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.623+0000 7f489e441040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:57:18.689 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.687+0000 7f489e441040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:57:18.712 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.485+0000 7f14f25e0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:57:18.712 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.645+0000 7f14f25e0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:57:18.713 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.710+0000 7f14f25e0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:57:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.725+0000 7fe732ffd640 1 -- 172.21.15.28:0/571076209 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fe73409cda0 con 0x7fe73c102600 2024-09-15T21:57:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 msgr2=0x7fe7280a5880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:57:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 0x7fe7280a5880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c102600 msgr2=0x7fe73c1a1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c102600 0x7fe73c1a1560 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe73400eb10 tx=0x7fe73400c2d0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 shutdown_connections 2024-09-15T21:57:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7fe7280a33c0 0x7fe7280a5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73c102600 0x7fe73c1a1560 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe73c101c40 0x7fe73c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 --2- 172.21.15.28:0/571076209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe73c100a50 0x7fe73c19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 >> 172.21.15.28:0/571076209 conn(0x7fe73c0fc1b0 msgr2=0x7fe73c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 shutdown_connections 2024-09-15T21:57:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.727+0000 7fe744037640 1 -- 172.21.15.28:0/571076209 wait complete. 2024-09-15T21:57:18.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-09-15T21:57:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 -- 172.21.15.28:0/790102062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c101c40 msgr2=0x7f982c1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 --2- 172.21.15.28:0/790102062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c101c40 0x7f982c1020c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f981c009900 tx=0x7f981c02a410 comp rx=0 tx=0).stop 2024-09-15T21:57:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 -- 172.21.15.28:0/790102062 shutdown_connections 2024-09-15T21:57:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 --2- 172.21.15.28:0/790102062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f982c102600 0x7f982c108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 --2- 172.21.15.28:0/790102062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c101c40 0x7f982c1020c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 --2- 172.21.15.28:0/790102062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982c100a50 0x7f982c100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 -- 172.21.15.28:0/790102062 >> 172.21.15.28:0/790102062 conn(0x7f982c0fc1b0 msgr2=0x7f982c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 -- 172.21.15.28:0/790102062 shutdown_connections 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.846+0000 7f98344c0640 1 -- 172.21.15.28:0/790102062 wait complete. 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 Processor -- start 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 -- start start 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c100a50 0x7f982c1049e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f982c101c40 0x7f982c103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982c102600 0x7f982c103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f982c10c6d0 con 0x7f982c100a50 2024-09-15T21:57:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f982c10c550 con 0x7f982c102600 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f98344c0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f982c10c850 con 0x7f982c101c40 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f983225c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c100a50 0x7f982c1049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f9831a5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f982c101c40 0x7f982c103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f983225c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c100a50 0x7f982c1049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35460/0 (socket says 172.21.15.28:35460) 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f983225c640 1 -- 172.21.15.28:0/1745499572 learned_addr learned my addr 172.21.15.28:0/1745499572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f9832a5d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982c102600 0x7f982c103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.847+0000 7f9831a5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f982c101c40 0x7f982c103020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58816/0 (socket says 172.21.15.28:58816) 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f983225c640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f982c101c40 msgr2=0x7f982c103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f983225c640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f982c101c40 0x7f982c103020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f983225c640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982c102600 msgr2=0x7f982c103560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f983225c640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f982c102600 0x7f982c103560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f983225c640 1 -- 172.21.15.28:0/1745499572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c103df0 con 0x7f982c100a50 2024-09-15T21:57:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f983225c640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c100a50 0x7f982c1049e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f982800e9c0 tx=0x7f982800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982800cd80 con 0x7f982c100a50 2024-09-15T21:57:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98280035f0 con 0x7f982c100a50 2024-09-15T21:57:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982c1a1f40 con 0x7f982c100a50 2024-09-15T21:57:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828010690 con 0x7f982c100a50 2024-09-15T21:57:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.848+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982c1a2340 con 0x7f982c100a50 2024-09-15T21:57:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.849+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f982c1a2630 con 0x7f982c100a50 2024-09-15T21:57:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.850+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 138372+0+0 (secure 0 0 0) 0x7f98280037a0 con 0x7f982c100a50 2024-09-15T21:57:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.851+0000 7f98237fe640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 0x7f98180a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.851+0000 7f9831a5b640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 msgr2=0x7f98180a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1670367809 2024-09-15T21:57:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.852+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9121+0+0 (secure 0 0 0) 0x7f98280d0050 con 0x7f982c100a50 2024-09-15T21:57:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.852+0000 7f9831a5b640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 0x7f98180a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T21:57:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:18.856+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f982809c9f0 con 0x7f982c100a50 2024-09-15T21:57:18.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/571076209' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-15T21:57:18.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.720+0000 7fb9a7dd1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T21:57:18.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.786+0000 7fb9a7dd1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T21:57:18.957 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/571076209' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-15T21:57:18.958 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.756+0000 7f489e441040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:57:18.958 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.837+0000 7f489e441040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:57:18.958 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.897+0000 7f489e441040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:57:18.958 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:18 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:18.955+0000 7f489e441040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:57:18.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/571076209' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-15T21:57:18.968 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.779+0000 7f14f25e0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:57:18.968 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.861+0000 7f14f25e0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:57:18.968 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.921+0000 7f14f25e0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:57:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:19.053+0000 7f9831a5b640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 msgr2=0x7f98180a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1670367809 2024-09-15T21:57:19.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:19.053+0000 7f9831a5b640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 0x7f98180a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T21:57:19.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:19.061+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 --> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f982c0fe440 con 0x7f98180a3340 2024-09-15T21:57:19.267 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:18 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:18.959+0000 7fb9a7dd1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T21:57:19.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:19 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:19.046+0000 7fb9a7dd1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T21:57:19.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:19 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:19.116+0000 7fb9a7dd1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T21:57:19.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:19 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:19.203+0000 7fb9a7dd1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T21:57:19.268 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:19 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:19.265+0000 7fb9a7dd1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T21:57:19.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:19.454+0000 7f9831a5b640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 msgr2=0x7f98180a5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1670367809 2024-09-15T21:57:19.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:19.454+0000 7f9831a5b640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 0x7f98180a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T21:57:19.468 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:18 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:18.980+0000 7f14f25e0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:57:19.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:19 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:19.333+0000 7fb9a7dd1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T21:57:20.010 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 21:57:19 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y[36606]: 2024-09-15T21:57:19.742+0000 7f489e441040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:57:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:19 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/571076209' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-15T21:57:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:19 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:57:20.078 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:19 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/571076209' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-15T21:57:20.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:19 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:57:20.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:20.190+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 20) ==== 138040+0+0 (secure 0 0 0) 0x7f9828098960 con 0x7f982c100a50 2024-09-15T21:57:20.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:20.190+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 msgr2=0x7f98180a5800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:57:20.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:20.190+0000 7f98237fe640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1670367809,v1:172.21.15.28:6801/1670367809] conn(0x7f98180a3340 0x7f98180a5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:20.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 21:57:19 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z[37353]: 2024-09-15T21:57:19.771+0000 7f14f25e0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:57:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:19 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/571076209' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-15T21:57:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:19 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:57:20.421 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.178+0000 7fb9a7dd1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb9123a9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb9123a9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb9123a9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb9123a9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb9123a9640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb90f3a3640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb90f3a3640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb90f3a3640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb90f3a3640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:20.456+0000 7fb90f3a3640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: Standby manager daemon y restarted 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: Standby manager daemon y started 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: Standby manager daemon z restarted 2024-09-15T21:57:20.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: Standby manager daemon z started 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: Active manager daemon x restarted 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: Activating manager daemon x 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: mgrmap e20: x(active, starting, since 0.0116971s), standbys: z, y 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:57:20.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T21:57:20.971 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:57:20.972 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: Manager daemon x is now available 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:20.973 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T21:57:20.974 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:20 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T21:57:21.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: Standby manager daemon y restarted 2024-09-15T21:57:21.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: Standby manager daemon y started 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: Standby manager daemon z restarted 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: Standby manager daemon z started 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: Active manager daemon x restarted 2024-09-15T21:57:21.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: Activating manager daemon x 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: mgrmap e20: x(active, starting, since 0.0116971s), standbys: z, y 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T21:57:21.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:57:21.013 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T21:57:21.014 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T21:57:21.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: Manager daemon x is now available 2024-09-15T21:57:21.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:21.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T21:57:21.015 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:20 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: Standby manager daemon y restarted 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: Standby manager daemon y started 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.174:0/2399149737' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: Standby manager daemon z restarted 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: Standby manager daemon z started 2024-09-15T21:57:21.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.? 172.21.15.203:0/4153292601' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: Active manager daemon x restarted 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: Activating manager daemon x 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: mgrmap e20: x(active, starting, since 0.0116971s), standbys: z, y 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T21:57:21.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T21:57:21.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T21:57:21.175 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T21:57:21.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T21:57:21.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T21:57:21.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: Manager daemon x is now available 2024-09-15T21:57:21.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:21.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T21:57:21.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:20 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T21:57:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.195+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 21) ==== 138167+0+0 (secure 0 0 0) 0x7f982809c550 con 0x7f982c100a50 2024-09-15T21:57:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.196+0000 7f98237fe640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9818051dd0 0x7f98180521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.196+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f9818034b90 con 0x7f9818051dd0 2024-09-15T21:57:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.199+0000 7f9831a5b640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9818051dd0 0x7f98180521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.203+0000 7f9831a5b640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9818051dd0 0x7f98180521b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f981c0314e0 tx=0x7f981c000bc0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.219+0000 7f98237fe640 1 -- 172.21.15.28:0/1745499572 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f9818034b90 con 0x7f9818051dd0 2024-09-15T21:57:21.221 INFO:teuthology.orchestra.run.smithi028.stdout:{} 2024-09-15T21:57:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9818051dd0 msgr2=0x7f98180521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9818051dd0 0x7f98180521b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f981c0314e0 tx=0x7f981c000bc0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c100a50 msgr2=0x7f982c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 --2- 172.21.15.28:0/1745499572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f982c100a50 0x7f982c1049e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f982800e9c0 tx=0x7f982800ee80 comp rx=0 tx=0).stop 2024-09-15T21:57:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f9832a5d640 1 -- 172.21.15.28:0/1745499572 reap_dead start 2024-09-15T21:57:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 shutdown_connections 2024-09-15T21:57:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 >> 172.21.15.28:0/1745499572 conn(0x7f982c0fc1b0 msgr2=0x7f982c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 shutdown_connections 2024-09-15T21:57:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.222+0000 7f98344c0640 1 -- 172.21.15.28:0/1745499572 wait complete. 2024-09-15T21:57:21.237 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-15T21:57:21.251 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-09-15T21:57:21.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:57:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 -- 172.21.15.28:0/2727550696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72841025c0 msgr2=0x7f7284108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 --2- 172.21.15.28:0/2727550696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72841025c0 0x7f7284108e50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f727c009ca0 tx=0x7f727c02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 -- 172.21.15.28:0/2727550696 shutdown_connections 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 --2- 172.21.15.28:0/2727550696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72841025c0 0x7f7284108e50 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 --2- 172.21.15.28:0/2727550696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7284101c00 0x7f7284102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 --2- 172.21.15.28:0/2727550696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7284100a10 0x7f7284100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 -- 172.21.15.28:0/2727550696 >> 172.21.15.28:0/2727550696 conn(0x7f72840fc170 msgr2=0x7f72840fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 -- 172.21.15.28:0/2727550696 shutdown_connections 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.376+0000 7f7289245640 1 -- 172.21.15.28:0/2727550696 wait complete. 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.377+0000 7f7289245640 1 Processor -- start 2024-09-15T21:57:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.377+0000 7f7289245640 1 -- start start 2024-09-15T21:57:21.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f7289245640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7284100a10 0x7f72841049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f7289245640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7284101c00 0x7f7284102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f7289245640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72841025c0 0x7f7284103520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f7289245640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f728410c690 con 0x7f7284101c00 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f7289245640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f728410c510 con 0x7f72841025c0 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f7289245640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f728410c810 con 0x7f7284100a10 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7284101c00 0x7f7284102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7284101c00 0x7f7284102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59582/0 (socket says 172.21.15.28:59582) 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 -- 172.21.15.28:0/402763051 learned_addr learned my addr 172.21.15.28:0/402763051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f7282d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7284100a10 0x7f72841049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728359e640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72841025c0 0x7f7284103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 -- 172.21.15.28:0/402763051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7284100a10 msgr2=0x7f72841049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7284100a10 0x7f72841049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 -- 172.21.15.28:0/402763051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72841025c0 msgr2=0x7f7284103520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72841025c0 0x7f7284103520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 -- 172.21.15.28:0/402763051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7284103db0 con 0x7f7284101c00 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728359e640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72841025c0 0x7f7284103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f728259c640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7284101c00 0x7f7284102fe0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f727400e9f0 tx=0x7f727400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f727bfff640 1 -- 172.21.15.28:0/402763051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727400cd80 con 0x7f7284101c00 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.378+0000 7f727bfff640 1 -- 172.21.15.28:0/402763051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7274010910 con 0x7f7284101c00 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.379+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72841a1f30 con 0x7f7284101c00 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.379+0000 7f727bfff640 1 -- 172.21.15.28:0/402763051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274018990 con 0x7f7284101c00 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.379+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72841a2390 con 0x7f7284101c00 2024-09-15T21:57:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.380+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72841a2680 con 0x7f7284101c00 2024-09-15T21:57:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.382+0000 7f727bfff640 1 -- 172.21.15.28:0/402763051 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 138167+0+0 (secure 0 0 0) 0x7f7274010ac0 con 0x7f7284101c00 2024-09-15T21:57:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.383+0000 7f727bfff640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f72700a3050 0x7f72700a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.383+0000 7f7282d9d640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f72700a3050 0x7f72700a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.383+0000 7f727bfff640 1 -- 172.21.15.28:0/402763051 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f72740cfc20 con 0x7f7284101c00 2024-09-15T21:57:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.383+0000 7f7282d9d640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f72700a3050 0x7f72700a5510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7284101a60 tx=0x7f726c005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:21.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.387+0000 7f727bfff640 1 -- 172.21.15.28:0/402763051 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f727409c160 con 0x7f7284101c00 2024-09-15T21:57:21.474 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:21.209+0000 7fb90c39d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:21.474 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:21.209+0000 7fb90c39d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:21.474 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:21.209+0000 7fb90c39d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:21.474 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:21.209+0000 7fb90c39d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:21.474 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:21.209+0000 7fb90c39d640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:57:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.569+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72841a2f40 con 0x7f7284101c00 2024-09-15T21:57:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.571+0000 7f727bfff640 1 -- 172.21.15.28:0/402763051 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+25138 (secure 0 0 0) 0x7f72740a0460 con 0x7f7284101c00 2024-09-15T21:57:21.574 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:21.574 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:57:20.180529+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/972903625":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/3676950981":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/4057362541":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/955882315":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/1670367809":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6801/1670367809":"2024-09-16T21:57:20.180486+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:57:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f72700a3050 msgr2=0x7f72700a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f72700a3050 0x7f72700a5510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7284101a60 tx=0x7f726c005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7284101c00 msgr2=0x7f7284102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7284101c00 0x7f7284102fe0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f727400e9f0 tx=0x7f727400eeb0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 shutdown_connections 2024-09-15T21:57:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f72700a3050 0x7f72700a5510 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72841025c0 0x7f7284103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7284101c00 0x7f7284102fe0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 --2- 172.21.15.28:0/402763051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7284100a10 0x7f72841049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 >> 172.21.15.28:0/402763051 conn(0x7f72840fc170 msgr2=0x7f72840693b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 shutdown_connections 2024-09-15T21:57:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.573+0000 7f7289245640 1 -- 172.21.15.28:0/402763051 wait complete. 2024-09-15T21:57:21.586 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T21:50:13.957276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-15T21:54:54.805516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-15T21:54:57.410111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-15T21:55:03.884115+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-15T21:57:21.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 -- 172.21.15.28:0/2933120923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 msgr2=0x7feae0107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 --2- 172.21.15.28:0/2933120923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae0107320 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fead4009900 tx=0x7fead402a410 comp rx=0 tx=0).stop 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 -- 172.21.15.28:0/2933120923 shutdown_connections 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 --2- 172.21.15.28:0/2933120923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae0107860 0x7feae0109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 --2- 172.21.15.28:0/2933120923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae0107320 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 --2- 172.21.15.28:0/2933120923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feae0102600 0x7feae01049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 -- 172.21.15.28:0/2933120923 >> 172.21.15.28:0/2933120923 conn(0x7feae00fc140 msgr2=0x7feae00fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 -- 172.21.15.28:0/2933120923 shutdown_connections 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 -- 172.21.15.28:0/2933120923 wait complete. 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 Processor -- start 2024-09-15T21:57:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.694+0000 7feae83c4640 1 -- start start 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae83c4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feae0102600 0x7feae0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae83c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae83c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae0107860 0x7feae019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae83c4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feae010d7b0 con 0x7feae0104f30 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae83c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feae010d630 con 0x7feae0107860 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae83c4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feae010d930 con 0x7feae0102600 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59600/0 (socket says 172.21.15.28:59600) 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 -- 172.21.15.28:0/1607927749 learned_addr learned my addr 172.21.15.28:0/1607927749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae6961640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae0107860 0x7feae019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 -- 172.21.15.28:0/1607927749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feae0102600 msgr2=0x7feae0197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feae0102600 0x7feae0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 -- 172.21.15.28:0/1607927749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae0107860 msgr2=0x7feae019f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae0107860 0x7feae019f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 -- 172.21.15.28:0/1607927749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae019fab0 con 0x7feae0104f30 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae595f640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae01983d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fead4031610 tx=0x7fead40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7fead37fe640 1 -- 172.21.15.28:0/1607927749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead403f030 con 0x7feae0104f30 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7fead37fe640 1 -- 172.21.15.28:0/1607927749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead4003ce0 con 0x7feae0104f30 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.695+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae019fd40 con 0x7feae0104f30 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.696+0000 7fead37fe640 1 -- 172.21.15.28:0/1607927749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4035070 con 0x7feae0104f30 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.696+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae01a0220 con 0x7feae0104f30 2024-09-15T21:57:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.696+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae00fdc30 con 0x7feae0104f30 2024-09-15T21:57:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.699+0000 7fead37fe640 1 -- 172.21.15.28:0/1607927749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 138167+0+0 (secure 0 0 0) 0x7fead4042030 con 0x7feae0104f30 2024-09-15T21:57:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.699+0000 7fead37fe640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feacc0a30d0 0x7feacc0a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.699+0000 7feae6160640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feacc0a30d0 0x7feacc0a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.699+0000 7fead37fe640 1 -- 172.21.15.28:0/1607927749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fead40f0210 con 0x7feae0104f30 2024-09-15T21:57:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.700+0000 7feae6160640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feacc0a30d0 0x7feacc0a5590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feadc005fd0 tx=0x7feadc005850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:21.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.702+0000 7fead37fe640 1 -- 172.21.15.28:0/1607927749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fead40bcda0 con 0x7feae0104f30 2024-09-15T21:57:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.900+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feae002da10 con 0x7feae0104f30 2024-09-15T21:57:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.901+0000 7fead37fe640 1 -- 172.21.15.28:0/1607927749 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fead40bc6a0 con 0x7feae0104f30 2024-09-15T21:57:21.903 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feacc0a30d0 msgr2=0x7feacc0a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feacc0a30d0 0x7feacc0a5590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feadc005fd0 tx=0x7feadc005850 comp rx=0 tx=0).stop 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 msgr2=0x7feae01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae01983d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fead4031610 tx=0x7fead40043b0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 shutdown_connections 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feacc0a30d0 0x7feacc0a5590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae0107860 0x7feae019f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feae0104f30 0x7feae01983d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 --2- 172.21.15.28:0/1607927749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feae0102600 0x7feae0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 >> 172.21.15.28:0/1607927749 conn(0x7feae00fc140 msgr2=0x7feae0102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 shutdown_connections 2024-09-15T21:57:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:21.903+0000 7feae83c4640 1 -- 172.21.15.28:0/1607927749 wait complete. 2024-09-15T21:57:21.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-15T21:57:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.040+0000 7f8ffd453640 1 -- 172.21.15.28:0/526632152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8102020 msgr2=0x7f8ff8102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.040+0000 7f8ffd453640 1 --2- 172.21.15.28:0/526632152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8102020 0x7f8ff8102420 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009900 tx=0x7f8fe802a410 comp rx=0 tx=0).stop 2024-09-15T21:57:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.040+0000 7f8ffd453640 1 -- 172.21.15.28:0/526632152 shutdown_connections 2024-09-15T21:57:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.040+0000 7f8ffd453640 1 --2- 172.21.15.28:0/526632152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ff8101630 0x7f8ff8108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.040+0000 7f8ffd453640 1 --2- 172.21.15.28:0/526632152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff8100c70 0x7f8ff81010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.040+0000 7f8ffd453640 1 --2- 172.21.15.28:0/526632152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8102020 0x7f8ff8102420 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.040+0000 7f8ffd453640 1 -- 172.21.15.28:0/526632152 >> 172.21.15.28:0/526632152 conn(0x7f8ff80fb910 msgr2=0x7f8ff80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.041+0000 7f8ffd453640 1 -- 172.21.15.28:0/526632152 shutdown_connections 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.041+0000 7f8ffd453640 1 -- 172.21.15.28:0/526632152 wait complete. 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.041+0000 7f8ffd453640 1 Processor -- start 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.041+0000 7f8ffd453640 1 -- start start 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.041+0000 7f8ffd453640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8100c70 0x7f8ff8104a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.041+0000 7f8ffd453640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff8101630 0x7f8ff8103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.041+0000 7f8ffd453640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ff8102020 0x7f8ff81035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ffd453640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810c6a0 con 0x7f8ff8100c70 2024-09-15T21:57:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ffd453640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810c520 con 0x7f8ff8102020 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ffd453640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810c820 con 0x7f8ff8101630 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8100c70 0x7f8ff8104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8feffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff8101630 0x7f8ff8103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8100c70 0x7f8ff8104a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59616/0 (socket says 172.21.15.28:59616) 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ff8102020 0x7f8ff81035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff6ffd640 1 -- 172.21.15.28:0/2913258663 learned_addr learned my addr 172.21.15.28:0/2913258663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8feffff640 1 -- 172.21.15.28:0/2913258663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ff8102020 msgr2=0x7f8ff81035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8feffff640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ff8102020 0x7f8ff81035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8feffff640 1 -- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8100c70 msgr2=0x7f8ff8104a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8feffff640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8100c70 0x7f8ff8104a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8feffff640 1 -- 172.21.15.28:0/2913258663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff8103e30 con 0x7f8ff8101630 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff77fe640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ff8102020 0x7f8ff81035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8feffff640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff8101630 0x7f8ff8103060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000e9c0 tx=0x7f8fe000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:22.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe000ccc0 con 0x7f8ff8101630 2024-09-15T21:57:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe00035f0 con 0x7f8ff8101630 2024-09-15T21:57:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a1fc0 con 0x7f8ff8101630 2024-09-15T21:57:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.042+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe000ce60 con 0x7f8ff8101630 2024-09-15T21:57:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.043+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a2430 con 0x7f8ff8101630 2024-09-15T21:57:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.043+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff81a2720 con 0x7f8ff8101630 2024-09-15T21:57:22.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.045+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 21) ==== 138167+0+0 (secure 0 0 0) 0x7f8fe00037a0 con 0x7f8ff8101630 2024-09-15T21:57:22.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.046+0000 7f8ff4ff9640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8fe40a3050 0x7f8fe40a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.047+0000 7f8ff6ffd640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8fe40a3050 0x7f8fe40a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.047+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f8fe00cff50 con 0x7f8ff8101630 2024-09-15T21:57:22.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.047+0000 7f8ff6ffd640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8fe40a3050 0x7f8fe40a5510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80046c0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.050+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8fe009caa0 con 0x7f8ff8101630 2024-09-15T21:57:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.206+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mgrmap(e 22) ==== 138526+0+0 (secure 0 0 0) 0x7f8fe0098a10 con 0x7f8ff8101630 2024-09-15T21:57:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.281+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8ff81a2a10 con 0x7f8ff8101630 2024-09-15T21:57:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.281+0000 7f8ff4ff9640 1 -- 172.21.15.28:0/2913258663 <== mon.1 v2:172.21.15.203:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f8fe00583c0 con 0x7f8ff8101630 2024-09-15T21:57:22.283 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8fe40a3050 msgr2=0x7f8fe40a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8fe40a3050 0x7f8fe40a5510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80046c0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff8101630 msgr2=0x7f8ff8103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff8101630 0x7f8ff8103060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000e9c0 tx=0x7f8fe000ee80 comp rx=0 tx=0).stop 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 shutdown_connections 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8fe40a3050 0x7f8fe40a5510 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ff8102020 0x7f8ff81035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff8101630 0x7f8ff8103060 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 --2- 172.21.15.28:0/2913258663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ff8100c70 0x7f8ff8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 >> 172.21.15.28:0/2913258663 conn(0x7f8ff80fb910 msgr2=0x7f8ff80ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 shutdown_connections 2024-09-15T21:57:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.285+0000 7f8ffd453640 1 -- 172.21.15.28:0/2913258663 wait complete. 2024-09-15T21:57:22.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-15T21:57:22.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.407+0000 7fedf7fff640 1 -- 172.21.15.28:0/544668070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8105570 msgr2=0x7fedf8105a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.407+0000 7fedf7fff640 1 --2- 172.21.15.28:0/544668070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8105570 0x7fedf8105a20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009ca0 tx=0x7fedf002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.407+0000 7fedf7fff640 1 -- 172.21.15.28:0/544668070 shutdown_connections 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.407+0000 7fedf7fff640 1 --2- 172.21.15.28:0/544668070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8105570 0x7fedf8105a20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.407+0000 7fedf7fff640 1 --2- 172.21.15.28:0/544668070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedf8102bc0 0x7fedf8105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.407+0000 7fedf7fff640 1 --2- 172.21.15.28:0/544668070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedf8100270 0x7fedf8102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.408+0000 7fedf7fff640 1 -- 172.21.15.28:0/544668070 >> 172.21.15.28:0/544668070 conn(0x7fedf80fc0a0 msgr2=0x7fedf80fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.408+0000 7fedf7fff640 1 -- 172.21.15.28:0/544668070 shutdown_connections 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.408+0000 7fedf7fff640 1 -- 172.21.15.28:0/544668070 wait complete. 2024-09-15T21:57:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.408+0000 7fedf7fff640 1 Processor -- start 2024-09-15T21:57:22.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.408+0000 7fedf7fff640 1 -- start start 2024-09-15T21:57:22.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedf8100270 0x7fedf819e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8102bc0 0x7fedf819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedf8105570 0x7fedf81a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf7fff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedf8113cb0 con 0x7fedf8102bc0 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf7fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fedf8113b30 con 0x7fedf8100270 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf7fff640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fedf8113e30 con 0x7fedf8105570 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8102bc0 0x7fedf819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedf8100270 0x7fedf819e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8102bc0 0x7fedf819e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59650/0 (socket says 172.21.15.28:59650) 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf77fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedf8105570 0x7fedf81a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 -- 172.21.15.28:0/3648204761 learned_addr learned my addr 172.21.15.28:0/3648204761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 -- 172.21.15.28:0/3648204761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedf8105570 msgr2=0x7fedf81a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedf8105570 0x7fedf81a5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 -- 172.21.15.28:0/3648204761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedf8100270 msgr2=0x7fedf819e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedf8100270 0x7fedf819e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 -- 172.21.15.28:0/3648204761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf81a5e80 con 0x7fedf8102bc0 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf77fe640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedf8105570 0x7fedf81a5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.409+0000 7fedf67fc640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8102bc0 0x7fedf819e7a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fedec00e9f0 tx=0x7fedec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.410+0000 7feddffff640 1 -- 172.21.15.28:0/3648204761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec00cd80 con 0x7fedf8102bc0 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.410+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf81a6170 con 0x7fedf8102bc0 2024-09-15T21:57:22.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.410+0000 7feddffff640 1 -- 172.21.15.28:0/3648204761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec010910 con 0x7fedf8102bc0 2024-09-15T21:57:22.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.410+0000 7feddffff640 1 -- 172.21.15.28:0/3648204761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec0188f0 con 0x7fedf8102bc0 2024-09-15T21:57:22.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.410+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf81a6710 con 0x7fedf8102bc0 2024-09-15T21:57:22.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.411+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf80fdb90 con 0x7fedf8102bc0 2024-09-15T21:57:22.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.413+0000 7feddffff640 1 -- 172.21.15.28:0/3648204761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 138526+0+0 (secure 0 0 0) 0x7fedec010ac0 con 0x7fedf8102bc0 2024-09-15T21:57:22.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.414+0000 7feddffff640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fede00a3450 0x7fede00a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.414+0000 7feddffff640 1 -- 172.21.15.28:0/3648204761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fedec0d0970 con 0x7fedf8102bc0 2024-09-15T21:57:22.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.414+0000 7fedf6ffd640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fede00a3450 0x7fede00a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.414+0000 7fedf6ffd640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fede00a3450 0x7fede00a5910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fede80095f0 tx=0x7fede8005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.420+0000 7feddffff640 1 -- 172.21.15.28:0/3648204761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fedec09c830 con 0x7fedf8102bc0 2024-09-15T21:57:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:22 smithi203 ceph-mon[35546]: mgrmap e21: x(active, since 1.01883s), standbys: z, y 2024-09-15T21:57:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:22 smithi203 ceph-mon[35546]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:22 smithi203 ceph-mon[35546]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:57:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:22 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/402763051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:22 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1607927749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T21:57:22.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:22 smithi174 ceph-mon[35244]: mgrmap e21: x(active, since 1.01883s), standbys: z, y 2024-09-15T21:57:22.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:22 smithi174 ceph-mon[35244]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:22.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:22 smithi174 ceph-mon[35244]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:57:22.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:22 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/402763051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:22.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:22 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1607927749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T21:57:22.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.614+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fedf802d7c0 con 0x7fedf8102bc0 2024-09-15T21:57:22.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.615+0000 7feddffff640 1 -- 172.21.15.28:0/3648204761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fedec0a0b30 con 0x7fedf8102bc0 2024-09-15T21:57:22.616 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fede00a3450 msgr2=0x7fede00a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fede00a3450 0x7fede00a5910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fede80095f0 tx=0x7fede8005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8102bc0 msgr2=0x7fedf819e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8102bc0 0x7fedf819e7a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fedec00e9f0 tx=0x7fedec00eeb0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 shutdown_connections 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fede00a3450 0x7fede00a5910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedf8105570 0x7fedf81a5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedf8102bc0 0x7fedf819e7a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 --2- 172.21.15.28:0/3648204761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedf8100270 0x7fedf819e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 >> 172.21.15.28:0/3648204761 conn(0x7fedf80fc0a0 msgr2=0x7fedf8100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:22.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 shutdown_connections 2024-09-15T21:57:22.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.617+0000 7fedf7fff640 1 -- 172.21.15.28:0/3648204761 wait complete. 2024-09-15T21:57:22.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-mon[30969]: mgrmap e21: x(active, since 1.01883s), standbys: z, y 2024-09-15T21:57:22.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-mon[30969]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:22.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-mon[30969]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:57:22.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/402763051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:22.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1607927749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T21:57:22.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-15T21:57:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.739+0000 7f5a76877640 1 -- 172.21.15.28:0/3043582528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a70101c20 msgr2=0x7f5a701020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.739+0000 7f5a76877640 1 --2- 172.21.15.28:0/3043582528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a70101c20 0x7f5a701020a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009900 tx=0x7f5a6402a410 comp rx=0 tx=0).stop 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.739+0000 7f5a76877640 1 -- 172.21.15.28:0/3043582528 shutdown_connections 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.739+0000 7f5a76877640 1 --2- 172.21.15.28:0/3043582528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a701025e0 0x7f5a70108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.740+0000 7f5a76877640 1 --2- 172.21.15.28:0/3043582528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a70101c20 0x7f5a701020a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.740+0000 7f5a76877640 1 --2- 172.21.15.28:0/3043582528 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70100a30 0x7f5a70100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.740+0000 7f5a76877640 1 -- 172.21.15.28:0/3043582528 >> 172.21.15.28:0/3043582528 conn(0x7f5a700fc1b0 msgr2=0x7f5a700fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.740+0000 7f5a76877640 1 -- 172.21.15.28:0/3043582528 shutdown_connections 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.740+0000 7f5a76877640 1 -- 172.21.15.28:0/3043582528 wait complete. 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.740+0000 7f5a76877640 1 Processor -- start 2024-09-15T21:57:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.740+0000 7f5a76877640 1 -- start start 2024-09-15T21:57:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a76877640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70100a30 0x7f5a701049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a76877640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a70101c20 0x7f5a70103000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a76877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a701025e0 0x7f5a70103540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a76877640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010c700 con 0x7f5a701025e0 2024-09-15T21:57:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a76877640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010c580 con 0x7f5a70101c20 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a76877640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010c880 con 0x7f5a70100a30 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a701025e0 0x7f5a70103540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a701025e0 0x7f5a70103540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59672/0 (socket says 172.21.15.28:59672) 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 -- 172.21.15.28:0/3022317998 learned_addr learned my addr 172.21.15.28:0/3022317998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70100a30 0x7f5a701049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a70101c20 0x7f5a70103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 -- 172.21.15.28:0/3022317998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70100a30 msgr2=0x7f5a701049c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70100a30 0x7f5a701049c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 -- 172.21.15.28:0/3022317998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a70101c20 msgr2=0x7f5a70103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a70101c20 0x7f5a70103000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 -- 172.21.15.28:0/3022317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70103dd0 con 0x7f5a701025e0 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70100a30 0x7f5a701049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a74e14640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a701025e0 0x7f5a70103540 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800eb10 tx=0x7f5a6800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.741+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/3022317998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68004b20 con 0x7f5a701025e0 2024-09-15T21:57:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.742+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a701a1fc0 con 0x7f5a701025e0 2024-09-15T21:57:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.742+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/3022317998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a68004cc0 con 0x7f5a701025e0 2024-09-15T21:57:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.742+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a701a2480 con 0x7f5a701025e0 2024-09-15T21:57:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.742+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/3022317998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68010700 con 0x7f5a701025e0 2024-09-15T21:57:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.743+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a701a2770 con 0x7f5a701025e0 2024-09-15T21:57:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.744+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/3022317998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 138526+0+0 (secure 0 0 0) 0x7f5a680108a0 con 0x7f5a701025e0 2024-09-15T21:57:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.745+0000 7f5a6d7fa640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a580a3450 0x7f5a580a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.745+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/3022317998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f5a680d0fb0 con 0x7f5a701025e0 2024-09-15T21:57:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.745+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a580a3450 0x7f5a580a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.745+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a580a3450 0x7f5a580a5910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a70101a80 tx=0x7f5a60005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.748+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/3022317998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5a6809d460 con 0x7f5a701025e0 2024-09-15T21:57:22.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:22.831+0000 7fb8f2deb640 -1 log_channel(cephadm) log [ERR] : [15/Sep/2024:21:57:22] ENGINE Error in HTTPServer.serve 2024-09-15T21:57:22.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: Traceback (most recent call last): 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._connections.run(self.expiration_interval) 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._run(expiration_interval) 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: s = self.context.wrap_socket( 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: return self.sslsocket_class._create( 2024-09-15T21:57:22.922 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:57:22.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self.do_handshake() 2024-09-15T21:57:22.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:57:22.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: self._sslobj.do_handshake() 2024-09-15T21:57:22.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:57:22.923 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:57:22 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:57:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.952+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5a701a2a60 con 0x7f5a701025e0 2024-09-15T21:57:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.952+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/3022317998 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f5a6809cd60 con 0x7f5a701025e0 2024-09-15T21:57:22.954 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-15T21:57:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a580a3450 msgr2=0x7f5a580a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a580a3450 0x7f5a580a5910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5a70101a80 tx=0x7f5a60005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a701025e0 msgr2=0x7f5a70103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a701025e0 0x7f5a70103540 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800eb10 tx=0x7f5a6800c2d0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 shutdown_connections 2024-09-15T21:57:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a580a3450 0x7f5a580a5910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a701025e0 0x7f5a70103540 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a70101c20 0x7f5a70103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 --2- 172.21.15.28:0/3022317998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a70100a30 0x7f5a701049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 >> 172.21.15.28:0/3022317998 conn(0x7f5a700fc1b0 msgr2=0x7f5a7006a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 shutdown_connections 2024-09-15T21:57:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:22.954+0000 7f5a76877640 1 -- 172.21.15.28:0/3022317998 wait complete. 2024-09-15T21:57:22.967 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-09-15T21:57:22.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:57:23.243 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2913258663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3648204761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: [15/Sep/2024:21:57:22] ENGINE Bus STARTING 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: [15/Sep/2024:21:57:22] ENGINE Serving on https://172.21.15.28:7150 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: [15/Sep/2024:21:57:22] ENGINE Error in HTTPServer.serve 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: Traceback (most recent call last): 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._connections.run(self.expiration_interval) 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._run(expiration_interval) 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:57:23.369 INFO:journalctl@ceph.mon.a.smithi028.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: s = self.context.wrap_socket( 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: return self.sslsocket_class._create( 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: self.do_handshake() 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: self._sslobj.do_handshake() 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: [15/Sep/2024:21:57:22] ENGINE Serving on http://172.21.15.28:8765 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: [15/Sep/2024:21:57:22] ENGINE Bus STARTED 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3022317998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.371 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:23.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:23 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:23.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2913258663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3648204761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: [15/Sep/2024:21:57:22] ENGINE Bus STARTING 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: [15/Sep/2024:21:57:22] ENGINE Serving on https://172.21.15.28:7150 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: [15/Sep/2024:21:57:22] ENGINE Error in HTTPServer.serve 2024-09-15T21:57:23.469 INFO:journalctl@ceph.mon.c.smithi203.stdout: Traceback (most recent call last): 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: self._connections.run(self.expiration_interval) 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: self._run(expiration_interval) 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: s = self.context.wrap_socket( 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:57:23.470 INFO:journalctl@ceph.mon.c.smithi203.stdout: return self.sslsocket_class._create( 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout: self.do_handshake() 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout: self._sslobj.do_handshake() 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: [15/Sep/2024:21:57:22] ENGINE Serving on http://172.21.15.28:8765 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: [15/Sep/2024:21:57:22] ENGINE Bus STARTED 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3022317998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.471 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.472 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.473 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.473 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:23.473 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:23 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:23.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-09-15T21:57:23.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2913258663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3648204761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: [15/Sep/2024:21:57:22] ENGINE Bus STARTING 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: [15/Sep/2024:21:57:22] ENGINE Serving on https://172.21.15.28:7150 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: [15/Sep/2024:21:57:22] ENGINE Error in HTTPServer.serve 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout: Traceback (most recent call last): 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout: self._connections.run(self.expiration_interval) 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout: self._run(expiration_interval) 2024-09-15T21:57:23.511 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: s = self.context.wrap_socket( 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: return self.sslsocket_class._create( 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: self.do_handshake() 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: self._sslobj.do_handshake() 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T21:57:23.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: [15/Sep/2024:21:57:22] ENGINE Serving on http://172.21.15.28:8765 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: [15/Sep/2024:21:57:22] ENGINE Bus STARTED 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3022317998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:23.514 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:23 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 -- 172.21.15.28:0/3085087961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc10e3b0 msgr2=0x7f84bc110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 --2- 172.21.15.28:0/3085087961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc10e3b0 0x7f84bc110800 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009ca0 tx=0x7f84b402a950 comp rx=0 tx=0).stop 2024-09-15T21:57:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 -- 172.21.15.28:0/3085087961 shutdown_connections 2024-09-15T21:57:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 --2- 172.21.15.28:0/3085087961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc10e3b0 0x7f84bc110800 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 --2- 172.21.15.28:0/3085087961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 0x7f84bc10de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 --2- 172.21.15.28:0/3085087961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc074620 0x7f84bc074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 -- 172.21.15.28:0/3085087961 >> 172.21.15.28:0/3085087961 conn(0x7f84bc0fd8b0 msgr2=0x7f84bc0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 -- 172.21.15.28:0/3085087961 shutdown_connections 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.837+0000 7f84c0a79640 1 -- 172.21.15.28:0/3085087961 wait complete. 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 Processor -- start 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 -- start start 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc074620 0x7f84bc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 0x7f84bc19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc10e3b0 0x7f84bc19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc114140 con 0x7f84bc074620 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc113fc0 con 0x7f84bc10e3b0 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84c0a79640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc1142c0 con 0x7f84bc074f40 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc074620 0x7f84bc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:23.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84bad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc10e3b0 0x7f84bc19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84b9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 0x7f84bc19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84bad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc10e3b0 0x7f84bc19eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56668/0 (socket says 172.21.15.28:56668) 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84b9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 0x7f84bc19ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:60622/0 (socket says 172.21.15.28:60622) 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.838+0000 7f84bad9d640 1 -- 172.21.15.28:0/1035486229 learned_addr learned my addr 172.21.15.28:0/1035486229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84bad9d640 1 -- 172.21.15.28:0/1035486229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 msgr2=0x7f84bc19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84bad9d640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 0x7f84bc19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84bad9d640 1 -- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc074620 msgr2=0x7f84bc19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84bad9d640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc074620 0x7f84bc19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84bad9d640 1 -- 172.21.15.28:0/1035486229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84bc19f5d0 con 0x7f84bc10e3b0 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84b9d9b640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 0x7f84bc19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84bad9d640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc10e3b0 0x7f84bc19eef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84b4004af0 tx=0x7f84b402f3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1035486229 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4034040 con 0x7f84bc10e3b0 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1035486229 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b4030920 con 0x7f84bc10e3b0 2024-09-15T21:57:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84bc19f860 con 0x7f84bc10e3b0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1035486229 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b4004670 con 0x7f84bc10e3b0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.839+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84bc1a71a0 con 0x7f84bc10e3b0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.840+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84bc075cd0 con 0x7f84bc10e3b0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.842+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1035486229 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 22) ==== 138526+0+0 (secure 0 0 0) 0x7f84b4037020 con 0x7f84bc10e3b0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.842+0000 7f84ab7fe640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f84a40a3440 0x7f84a40a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.842+0000 7f84ba59c640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f84a40a3440 0x7f84a40a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.843+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1035486229 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f84b40f0fc0 con 0x7f84bc10e3b0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.843+0000 7f84ba59c640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f84a40a3440 0x7f84a40a5900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009bd0 tx=0x7f84ac005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:23.845+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1035486229 <== mon.2 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f84b40bd580 con 0x7f84bc10e3b0 2024-09-15T21:57:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.062+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84bc074a00 con 0x7f84bc10e3b0 2024-09-15T21:57:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.063+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1035486229 <== mon.2 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f84b40bce80 con 0x7f84bc10e3b0 2024-09-15T21:57:24.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:24.065 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:57:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:57:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f84a40a3440 msgr2=0x7f84a40a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f84a40a3440 0x7f84a40a5900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009bd0 tx=0x7f84ac005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc10e3b0 msgr2=0x7f84bc19eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc10e3b0 0x7f84bc19eef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f84b4004af0 tx=0x7f84b402f3a0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 shutdown_connections 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f84a40a3440 0x7f84a40a5900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84bc10e3b0 0x7f84bc19eef0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84bc074f40 0x7f84bc19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 --2- 172.21.15.28:0/1035486229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84bc074620 0x7f84bc19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 >> 172.21.15.28:0/1035486229 conn(0x7f84bc0fd8b0 msgr2=0x7f84bc0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 shutdown_connections 2024-09-15T21:57:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.066+0000 7f84c0a79640 1 -- 172.21.15.28:0/1035486229 wait complete. 2024-09-15T21:57:24.746 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-09-15T21:57:24.746 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-09-15T21:57:24.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:57:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 -- 172.21.15.28:0/144927929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8101320 msgr2=0x7f3ce810d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/144927929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8101320 0x7f3ce810d850 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009c10 tx=0x7f3ce002a6e0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 -- 172.21.15.28:0/144927929 shutdown_connections 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/144927929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8101320 0x7f3ce810d850 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/144927929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ce8100980 0x7f3ce8100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/144927929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce8106990 0x7f3ce8106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 -- 172.21.15.28:0/144927929 >> 172.21.15.28:0/144927929 conn(0x7f3ce8077a10 msgr2=0x7f3ce80fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 -- 172.21.15.28:0/144927929 shutdown_connections 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.853+0000 7f3ceeaac640 1 -- 172.21.15.28:0/144927929 wait complete. 2024-09-15T21:57:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 Processor -- start 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 -- start start 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce8100980 0x7f3ce8199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ce8101320 0x7f3ce81997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8106990 0x7f3ce81a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce8110d80 con 0x7f3ce8106990 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce8110c00 con 0x7f3ce8101320 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ceeaac640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce8110f00 con 0x7f3ce8100980 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8106990 0x7f3ce81a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8106990 0x7f3ce81a42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59714/0 (socket says 172.21.15.28:59714) 2024-09-15T21:57:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 -- 172.21.15.28:0/4098750408 learned_addr learned my addr 172.21.15.28:0/4098750408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3cec848640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce8100980 0x7f3ce8199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ce7fff640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ce8101320 0x7f3ce81997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 -- 172.21.15.28:0/4098750408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce8100980 msgr2=0x7f3ce8199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce8100980 0x7f3ce8199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 -- 172.21.15.28:0/4098750408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ce8101320 msgr2=0x7f3ce81997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ce8101320 0x7f3ce81997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ced049640 1 -- 172.21.15.28:0/4098750408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce81a4830 con 0x7f3ce8106990 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.854+0000 7f3ce7fff640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ce8101320 0x7f3ce81997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.855+0000 7f3ced049640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8106990 0x7f3ce81a42f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00305d0 tx=0x7f3ce0030600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.855+0000 7f3ce5ffb640 1 -- 172.21.15.28:0/4098750408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0041020 con 0x7f3ce8106990 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.855+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce81a4a00 con 0x7f3ce8106990 2024-09-15T21:57:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.855+0000 7f3ce5ffb640 1 -- 172.21.15.28:0/4098750408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ce0037040 con 0x7f3ce8106990 2024-09-15T21:57:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.855+0000 7f3ce5ffb640 1 -- 172.21.15.28:0/4098750408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0036070 con 0x7f3ce8106990 2024-09-15T21:57:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.855+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce81a4ec0 con 0x7f3ce8106990 2024-09-15T21:57:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.856+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce8100de0 con 0x7f3ce8106990 2024-09-15T21:57:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.857+0000 7f3ce5ffb640 1 -- 172.21.15.28:0/4098750408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 138526+0+0 (secure 0 0 0) 0x7f3ce0044020 con 0x7f3ce8106990 2024-09-15T21:57:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.858+0000 7f3ce5ffb640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3cd00a3450 0x7f3cd00a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.858+0000 7f3cec848640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3cd00a3450 0x7f3cd00a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.858+0000 7f3ce5ffb640 1 -- 172.21.15.28:0/4098750408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f3ce00f11e0 con 0x7f3ce8106990 2024-09-15T21:57:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.858+0000 7f3cec848640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3cd00a3450 0x7f3cd00a5910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009ae0 tx=0x7f3cd8005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:24.861+0000 7f3ce5ffb640 1 -- 172.21.15.28:0/4098750408 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ce004b020 con 0x7f3ce8106990 2024-09-15T21:57:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.038+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ce8062630 con 0x7f3cd00a3450 2024-09-15T21:57:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.047+0000 7f3ce5ffb640 1 -- 172.21.15.28:0/4098750408 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339443 (secure 0 0 0) 0x7f3ce8062630 con 0x7f3cd00a3450 2024-09-15T21:57:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.050+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3cd00a3450 msgr2=0x7f3cd00a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.050+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3cd00a3450 0x7f3cd00a5910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009ae0 tx=0x7f3cd8005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.050+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8106990 msgr2=0x7f3ce81a42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.050+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8106990 0x7f3ce81a42f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00305d0 tx=0x7f3ce0030600 comp rx=0 tx=0).stop 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 shutdown_connections 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3cd00a3450 0x7f3cd00a5910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ce8106990 0x7f3ce81a42f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ce8101320 0x7f3ce81997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 --2- 172.21.15.28:0/4098750408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ce8100980 0x7f3ce8199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 >> 172.21.15.28:0/4098750408 conn(0x7f3ce8077a10 msgr2=0x7f3ce8103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 shutdown_connections 2024-09-15T21:57:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.051+0000 7f3ceeaac640 1 -- 172.21.15.28:0/4098750408 wait complete. 2024-09-15T21:57:25.063 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-09-15T21:57:24.202295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193769,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1308,"num_read_kb":823,"num_write":572,"num_write_kb":4417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":952,"ondisk_log_size":952,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365756,"kb_used_data":16296,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576068,"statfs":{"total":1151940427776,"available":1151565893632,"internally_reserved":0,"allocated":16687104,"data_stored":9942839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993652"},"pg_stats":[{"pgid":"2.1f","version":"93'14","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.469302+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:57:20.469302+0000","last_peered":"2024-09-15T21:57:20.469302+0000","last_clean":"2024-09-15T21:57:20.469302+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:57:20.469302+0000","last_undegraded":"2024-09-15T21:57:20.469302+0000","last_fullsized":"2024-09-15T21:57:20.469302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280359+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:57:20.280359+0000","last_peered":"2024-09-15T21:57:20.280359+0000","last_clean":"2024-09-15T21:57:20.280359+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:57:20.280359+0000","last_undegraded":"2024-09-15T21:57:20.280359+0000","last_fullsized":"2024-09-15T21:57:20.280359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395705+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:57:20.395705+0000","last_peered":"2024-09-15T21:57:20.395705+0000","last_clean":"2024-09-15T21:57:20.395705+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:57:20.395705+0000","last_undegraded":"2024-09-15T21:57:20.395705+0000","last_fullsized":"2024-09-15T21:57:20.395705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280756+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:57:20.280756+0000","last_peered":"2024-09-15T21:57:20.280756+0000","last_clean":"2024-09-15T21:57:20.280756+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:57:20.280756+0000","last_undegraded":"2024-09-15T21:57:20.280756+0000","last_fullsized":"2024-09-15T21:57:20.280756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467940+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:57:20.467940+0000","last_peered":"2024-09-15T21:57:20.467940+0000","last_clean":"2024-09-15T21:57:20.467940+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:57:20.467940+0000","last_undegraded":"2024-09-15T21:57:20.467940+0000","last_fullsized":"2024-09-15T21:57:20.467940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190047+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:57:20.190047+0000","last_peered":"2024-09-15T21:57:20.190047+0000","last_clean":"2024-09-15T21:57:20.190047+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:57:20.190047+0000","last_undegraded":"2024-09-15T21:57:20.190047+0000","last_fullsized":"2024-09-15T21:57:20.190047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278626+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:57:20.278626+0000","last_peered":"2024-09-15T21:57:20.278626+0000","last_clean":"2024-09-15T21:57:20.278626+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:57:20.278626+0000","last_undegraded":"2024-09-15T21:57:20.278626+0000","last_fullsized":"2024-09-15T21:57:20.278626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463402+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:57:20.463402+0000","last_peered":"2024-09-15T21:57:20.463402+0000","last_clean":"2024-09-15T21:57:20.463402+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:57:20.463402+0000","last_undegraded":"2024-09-15T21:57:20.463402+0000","last_fullsized":"2024-09-15T21:57:20.463402+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190387+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:57:20.190387+0000","last_peered":"2024-09-15T21:57:20.190387+0000","last_clean":"2024-09-15T21:57:20.190387+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:57:20.190387+0000","last_undegraded":"2024-09-15T21:57:20.190387+0000","last_fullsized":"2024-09-15T21:57:20.190387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472029+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:57:20.472029+0000","last_peered":"2024-09-15T21:57:20.472029+0000","last_clean":"2024-09-15T21:57:20.472029+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:57:20.472029+0000","last_undegraded":"2024-09-15T21:57:20.472029+0000","last_fullsized":"2024-09-15T21:57:20.472029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188165+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:57:20.188165+0000","last_peered":"2024-09-15T21:57:20.188165+0000","last_clean":"2024-09-15T21:57:20.188165+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:57:20.188165+0000","last_undegraded":"2024-09-15T21:57:20.188165+0000","last_fullsized":"2024-09-15T21:57:20.188165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190320+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:57:20.190320+0000","last_peered":"2024-09-15T21:57:20.190320+0000","last_clean":"2024-09-15T21:57:20.190320+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:57:20.190320+0000","last_undegraded":"2024-09-15T21:57:20.190320+0000","last_fullsized":"2024-09-15T21:57:20.190320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460655+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:57:20.460655+0000","last_peered":"2024-09-15T21:57:20.460655+0000","last_clean":"2024-09-15T21:57:20.460655+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:57:20.460655+0000","last_undegraded":"2024-09-15T21:57:20.460655+0000","last_fullsized":"2024-09-15T21:57:20.460655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278262+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:57:20.278262+0000","last_peered":"2024-09-15T21:57:20.278262+0000","last_clean":"2024-09-15T21:57:20.278262+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:57:20.278262+0000","last_undegraded":"2024-09-15T21:57:20.278262+0000","last_fullsized":"2024-09-15T21:57:20.278262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190739+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:57:20.190739+0000","last_peered":"2024-09-15T21:57:20.190739+0000","last_clean":"2024-09-15T21:57:20.190739+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:57:20.190739+0000","last_undegraded":"2024-09-15T21:57:20.190739+0000","last_fullsized":"2024-09-15T21:57:20.190739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279728+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:57:20.279728+0000","last_peered":"2024-09-15T21:57:20.279728+0000","last_clean":"2024-09-15T21:57:20.279728+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:57:20.279728+0000","last_undegraded":"2024-09-15T21:57:20.279728+0000","last_fullsized":"2024-09-15T21:57:20.279728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"92'4","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471831+0000","last_change":"2024-09-15T21:56:58.926427+0000","last_active":"2024-09-15T21:57:20.471831+0000","last_peered":"2024-09-15T21:57:20.471831+0000","last_clean":"2024-09-15T21:57:20.471831+0000","last_became_active":"2024-09-15T21:56:58.922755+0000","last_became_peered":"2024-09-15T21:56:58.922755+0000","last_unstale":"2024-09-15T21:57:20.471831+0000","last_undegraded":"2024-09-15T21:57:20.471831+0000","last_fullsized":"2024-09-15T21:57:20.471831+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:22.033354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:57:20.190660+0000","last_peered":"2024-09-15T21:57:20.190660+0000","last_clean":"2024-09-15T21:57:20.190660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:57:20.190660+0000","last_undegraded":"2024-09-15T21:57:20.190660+0000","last_fullsized":"2024-09-15T21:57:20.190660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471869+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:57:20.471869+0000","last_peered":"2024-09-15T21:57:20.471869+0000","last_clean":"2024-09-15T21:57:20.471869+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:57:20.471869+0000","last_undegraded":"2024-09-15T21:57:20.471869+0000","last_fullsized":"2024-09-15T21:57:20.471869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279173+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:57:20.279173+0000","last_peered":"2024-09-15T21:57:20.279173+0000","last_clean":"2024-09-15T21:57:20.279173+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:57:20.279173+0000","last_undegraded":"2024-09-15T21:57:20.279173+0000","last_fullsized":"2024-09-15T21:57:20.279173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276125+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:57:20.276125+0000","last_peered":"2024-09-15T21:57:20.276125+0000","last_clean":"2024-09-15T21:57:20.276125+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:57:20.276125+0000","last_undegraded":"2024-09-15T21:57:20.276125+0000","last_fullsized":"2024-09-15T21:57:20.276125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278075+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:57:20.278075+0000","last_peered":"2024-09-15T21:57:20.278075+0000","last_clean":"2024-09-15T21:57:20.278075+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:57:20.278075+0000","last_undegraded":"2024-09-15T21:57:20.278075+0000","last_fullsized":"2024-09-15T21:57:20.278075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462853+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:57:20.462853+0000","last_peered":"2024-09-15T21:57:20.462853+0000","last_clean":"2024-09-15T21:57:20.462853+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:57:20.462853+0000","last_undegraded":"2024-09-15T21:57:20.462853+0000","last_fullsized":"2024-09-15T21:57:20.462853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190763+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:57:20.190763+0000","last_peered":"2024-09-15T21:57:20.190763+0000","last_clean":"2024-09-15T21:57:20.190763+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:57:20.190763+0000","last_undegraded":"2024-09-15T21:57:20.190763+0000","last_fullsized":"2024-09-15T21:57:20.190763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467562+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:57:20.467562+0000","last_peered":"2024-09-15T21:57:20.467562+0000","last_clean":"2024-09-15T21:57:20.467562+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:57:20.467562+0000","last_undegraded":"2024-09-15T21:57:20.467562+0000","last_fullsized":"2024-09-15T21:57:20.467562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190932+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:57:20.190932+0000","last_peered":"2024-09-15T21:57:20.190932+0000","last_clean":"2024-09-15T21:57:20.190932+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:57:20.190932+0000","last_undegraded":"2024-09-15T21:57:20.190932+0000","last_fullsized":"2024-09-15T21:57:20.190932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395988+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:57:20.395988+0000","last_peered":"2024-09-15T21:57:20.395988+0000","last_clean":"2024-09-15T21:57:20.395988+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:57:20.395988+0000","last_undegraded":"2024-09-15T21:57:20.395988+0000","last_fullsized":"2024-09-15T21:57:20.395988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280166+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:57:20.280166+0000","last_peered":"2024-09-15T21:57:20.280166+0000","last_clean":"2024-09-15T21:57:20.280166+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:57:20.280166+0000","last_undegraded":"2024-09-15T21:57:20.280166+0000","last_fullsized":"2024-09-15T21:57:20.280166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'53","reported_seq":117,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.416465+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:57:20.416465+0000","last_peered":"2024-09-15T21:57:20.416465+0000","last_clean":"2024-09-15T21:57:20.416465+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:57:20.416465+0000","last_undegraded":"2024-09-15T21:57:20.416465+0000","last_fullsized":"2024-09-15T21:57:20.416465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":94,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.277996+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:57:20.277996+0000","last_peered":"2024-09-15T21:57:20.277996+0000","last_clean":"2024-09-15T21:57:20.277996+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:57:20.277996+0000","last_undegraded":"2024-09-15T21:57:20.277996+0000","last_fullsized":"2024-09-15T21:57:20.277996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186161+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:57:20.186161+0000","last_peered":"2024-09-15T21:57:20.186161+0000","last_clean":"2024-09-15T21:57:20.186161+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:57:20.186161+0000","last_undegraded":"2024-09-15T21:57:20.186161+0000","last_fullsized":"2024-09-15T21:57:20.186161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278630+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:57:20.278630+0000","last_peered":"2024-09-15T21:57:20.278630+0000","last_clean":"2024-09-15T21:57:20.278630+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:57:20.278630+0000","last_undegraded":"2024-09-15T21:57:20.278630+0000","last_fullsized":"2024-09-15T21:57:20.278630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472019+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:57:20.472019+0000","last_peered":"2024-09-15T21:57:20.472019+0000","last_clean":"2024-09-15T21:57:20.472019+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:57:20.472019+0000","last_undegraded":"2024-09-15T21:57:20.472019+0000","last_fullsized":"2024-09-15T21:57:20.472019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285785+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:57:20.285785+0000","last_peered":"2024-09-15T21:57:20.285785+0000","last_clean":"2024-09-15T21:57:20.285785+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:57:20.285785+0000","last_undegraded":"2024-09-15T21:57:20.285785+0000","last_fullsized":"2024-09-15T21:57:20.285785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276329+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:57:20.276329+0000","last_peered":"2024-09-15T21:57:20.276329+0000","last_clean":"2024-09-15T21:57:20.276329+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:57:20.276329+0000","last_undegraded":"2024-09-15T21:57:20.276329+0000","last_fullsized":"2024-09-15T21:57:20.276329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278087+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:57:20.278087+0000","last_peered":"2024-09-15T21:57:20.278087+0000","last_clean":"2024-09-15T21:57:20.278087+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:57:20.278087+0000","last_undegraded":"2024-09-15T21:57:20.278087+0000","last_fullsized":"2024-09-15T21:57:20.278087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285570+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:57:20.285570+0000","last_peered":"2024-09-15T21:57:20.285570+0000","last_clean":"2024-09-15T21:57:20.285570+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:57:20.285570+0000","last_undegraded":"2024-09-15T21:57:20.285570+0000","last_fullsized":"2024-09-15T21:57:20.285570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.711889+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:57:20.711889+0000","last_peered":"2024-09-15T21:57:20.711889+0000","last_clean":"2024-09-15T21:57:20.711889+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:57:20.711889+0000","last_undegraded":"2024-09-15T21:57:20.711889+0000","last_fullsized":"2024-09-15T21:57:20.711889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185976+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:57:20.185976+0000","last_peered":"2024-09-15T21:57:20.185976+0000","last_clean":"2024-09-15T21:57:20.185976+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:57:20.185976+0000","last_undegraded":"2024-09-15T21:57:20.185976+0000","last_fullsized":"2024-09-15T21:57:20.185976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187498+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:57:20.187498+0000","last_peered":"2024-09-15T21:57:20.187498+0000","last_clean":"2024-09-15T21:57:20.187498+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:57:20.187498+0000","last_undegraded":"2024-09-15T21:57:20.187498+0000","last_fullsized":"2024-09-15T21:57:20.187498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":151,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466327+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:57:20.466327+0000","last_peered":"2024-09-15T21:57:20.466327+0000","last_clean":"2024-09-15T21:57:20.466327+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:57:20.466327+0000","last_undegraded":"2024-09-15T21:57:20.466327+0000","last_fullsized":"2024-09-15T21:57:20.466327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186270+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:57:20.186270+0000","last_peered":"2024-09-15T21:57:20.186270+0000","last_clean":"2024-09-15T21:57:20.186270+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:57:20.186270+0000","last_undegraded":"2024-09-15T21:57:20.186270+0000","last_fullsized":"2024-09-15T21:57:20.186270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185590+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:57:20.185590+0000","last_peered":"2024-09-15T21:57:20.185590+0000","last_clean":"2024-09-15T21:57:20.185590+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:57:20.185590+0000","last_undegraded":"2024-09-15T21:57:20.185590+0000","last_fullsized":"2024-09-15T21:57:20.185590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460124+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:57:20.460124+0000","last_peered":"2024-09-15T21:57:20.460124+0000","last_clean":"2024-09-15T21:57:20.460124+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:57:20.460124+0000","last_undegraded":"2024-09-15T21:57:20.460124+0000","last_fullsized":"2024-09-15T21:57:20.460124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188088+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:57:20.188088+0000","last_peered":"2024-09-15T21:57:20.188088+0000","last_clean":"2024-09-15T21:57:20.188088+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:57:20.188088+0000","last_undegraded":"2024-09-15T21:57:20.188088+0000","last_fullsized":"2024-09-15T21:57:20.188088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279569+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:57:20.279569+0000","last_peered":"2024-09-15T21:57:20.279569+0000","last_clean":"2024-09-15T21:57:20.279569+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:57:20.279569+0000","last_undegraded":"2024-09-15T21:57:20.279569+0000","last_fullsized":"2024-09-15T21:57:20.279569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190561+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:57:20.190561+0000","last_peered":"2024-09-15T21:57:20.190561+0000","last_clean":"2024-09-15T21:57:20.190561+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:57:20.190561+0000","last_undegraded":"2024-09-15T21:57:20.190561+0000","last_fullsized":"2024-09-15T21:57:20.190561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460734+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:57:20.460734+0000","last_peered":"2024-09-15T21:57:20.460734+0000","last_clean":"2024-09-15T21:57:20.460734+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:57:20.460734+0000","last_undegraded":"2024-09-15T21:57:20.460734+0000","last_fullsized":"2024-09-15T21:57:20.460734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460868+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:57:20.460868+0000","last_peered":"2024-09-15T21:57:20.460868+0000","last_clean":"2024-09-15T21:57:20.460868+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:57:20.460868+0000","last_undegraded":"2024-09-15T21:57:20.460868+0000","last_fullsized":"2024-09-15T21:57:20.460868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461076+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:57:20.461076+0000","last_peered":"2024-09-15T21:57:20.461076+0000","last_clean":"2024-09-15T21:57:20.461076+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:57:20.461076+0000","last_undegraded":"2024-09-15T21:57:20.461076+0000","last_fullsized":"2024-09-15T21:57:20.461076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.719909+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:57:20.719909+0000","last_peered":"2024-09-15T21:57:20.719909+0000","last_clean":"2024-09-15T21:57:20.719909+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:57:20.719909+0000","last_undegraded":"2024-09-15T21:57:20.719909+0000","last_fullsized":"2024-09-15T21:57:20.719909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462918+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:57:20.462918+0000","last_peered":"2024-09-15T21:57:20.462918+0000","last_clean":"2024-09-15T21:57:20.462918+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:57:20.462918+0000","last_undegraded":"2024-09-15T21:57:20.462918+0000","last_fullsized":"2024-09-15T21:57:20.462918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460824+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:57:20.460824+0000","last_peered":"2024-09-15T21:57:20.460824+0000","last_clean":"2024-09-15T21:57:20.460824+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:57:20.460824+0000","last_undegraded":"2024-09-15T21:57:20.460824+0000","last_fullsized":"2024-09-15T21:57:20.460824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'41","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.413534+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:57:20.413534+0000","last_peered":"2024-09-15T21:57:20.413534+0000","last_clean":"2024-09-15T21:57:20.413534+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:57:20.413534+0000","last_undegraded":"2024-09-15T21:57:20.413534+0000","last_fullsized":"2024-09-15T21:57:20.413534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466957+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:57:20.466957+0000","last_peered":"2024-09-15T21:57:20.466957+0000","last_clean":"2024-09-15T21:57:20.466957+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:57:20.466957+0000","last_undegraded":"2024-09-15T21:57:20.466957+0000","last_fullsized":"2024-09-15T21:57:20.466957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460987+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:57:20.460987+0000","last_peered":"2024-09-15T21:57:20.460987+0000","last_clean":"2024-09-15T21:57:20.460987+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:57:20.460987+0000","last_undegraded":"2024-09-15T21:57:20.460987+0000","last_fullsized":"2024-09-15T21:57:20.460987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186150+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:57:20.186150+0000","last_peered":"2024-09-15T21:57:20.186150+0000","last_clean":"2024-09-15T21:57:20.186150+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:57:20.186150+0000","last_undegraded":"2024-09-15T21:57:20.186150+0000","last_fullsized":"2024-09-15T21:57:20.186150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187377+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:57:20.187377+0000","last_peered":"2024-09-15T21:57:20.187377+0000","last_clean":"2024-09-15T21:57:20.187377+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:57:20.187377+0000","last_undegraded":"2024-09-15T21:57:20.187377+0000","last_fullsized":"2024-09-15T21:57:20.187377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472329+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:57:20.472329+0000","last_peered":"2024-09-15T21:57:20.472329+0000","last_clean":"2024-09-15T21:57:20.472329+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:57:20.472329+0000","last_undegraded":"2024-09-15T21:57:20.472329+0000","last_fullsized":"2024-09-15T21:57:20.472329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278620+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:57:20.278620+0000","last_peered":"2024-09-15T21:57:20.278620+0000","last_clean":"2024-09-15T21:57:20.278620+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:57:20.278620+0000","last_undegraded":"2024-09-15T21:57:20.278620+0000","last_fullsized":"2024-09-15T21:57:20.278620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187448+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:57:20.187448+0000","last_peered":"2024-09-15T21:57:20.187448+0000","last_clean":"2024-09-15T21:57:20.187448+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:57:20.187448+0000","last_undegraded":"2024-09-15T21:57:20.187448+0000","last_fullsized":"2024-09-15T21:57:20.187448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187552+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:57:20.187552+0000","last_peered":"2024-09-15T21:57:20.187552+0000","last_clean":"2024-09-15T21:57:20.187552+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:57:20.187552+0000","last_undegraded":"2024-09-15T21:57:20.187552+0000","last_fullsized":"2024-09-15T21:57:20.187552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278144+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:57:20.278144+0000","last_peered":"2024-09-15T21:57:20.278144+0000","last_clean":"2024-09-15T21:57:20.278144+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:57:20.278144+0000","last_undegraded":"2024-09-15T21:57:20.278144+0000","last_fullsized":"2024-09-15T21:57:20.278144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186058+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:57:20.186058+0000","last_peered":"2024-09-15T21:57:20.186058+0000","last_clean":"2024-09-15T21:57:20.186058+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:57:20.186058+0000","last_undegraded":"2024-09-15T21:57:20.186058+0000","last_fullsized":"2024-09-15T21:57:20.186058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460653+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:57:20.460653+0000","last_peered":"2024-09-15T21:57:20.460653+0000","last_clean":"2024-09-15T21:57:20.460653+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:57:20.460653+0000","last_undegraded":"2024-09-15T21:57:20.460653+0000","last_fullsized":"2024-09-15T21:57:20.460653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":154,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.720943+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:57:20.720943+0000","last_peered":"2024-09-15T21:57:20.720943+0000","last_clean":"2024-09-15T21:57:20.720943+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:57:20.720943+0000","last_undegraded":"2024-09-15T21:57:20.720943+0000","last_fullsized":"2024-09-15T21:57:20.720943+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276236+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:57:20.276236+0000","last_peered":"2024-09-15T21:57:20.276236+0000","last_clean":"2024-09-15T21:57:20.276236+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:57:20.276236+0000","last_undegraded":"2024-09-15T21:57:20.276236+0000","last_fullsized":"2024-09-15T21:57:20.276236+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460781+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:57:20.460781+0000","last_peered":"2024-09-15T21:57:20.460781+0000","last_clean":"2024-09-15T21:57:20.460781+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:57:20.460781+0000","last_undegraded":"2024-09-15T21:57:20.460781+0000","last_fullsized":"2024-09-15T21:57:20.460781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187760+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:57:20.187760+0000","last_peered":"2024-09-15T21:57:20.187760+0000","last_clean":"2024-09-15T21:57:20.187760+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:57:20.187760+0000","last_undegraded":"2024-09-15T21:57:20.187760+0000","last_fullsized":"2024-09-15T21:57:20.187760+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472044+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:57:20.472044+0000","last_peered":"2024-09-15T21:57:20.472044+0000","last_clean":"2024-09-15T21:57:20.472044+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:57:20.472044+0000","last_undegraded":"2024-09-15T21:57:20.472044+0000","last_fullsized":"2024-09-15T21:57:20.472044+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185779+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:57:20.185779+0000","last_peered":"2024-09-15T21:57:20.185779+0000","last_clean":"2024-09-15T21:57:20.185779+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:57:20.185779+0000","last_undegraded":"2024-09-15T21:57:20.185779+0000","last_fullsized":"2024-09-15T21:57:20.185779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395654+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:57:20.395654+0000","last_peered":"2024-09-15T21:57:20.395654+0000","last_clean":"2024-09-15T21:57:20.395654+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:57:20.395654+0000","last_undegraded":"2024-09-15T21:57:20.395654+0000","last_fullsized":"2024-09-15T21:57:20.395654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188062+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:57:20.188062+0000","last_peered":"2024-09-15T21:57:20.188062+0000","last_clean":"2024-09-15T21:57:20.188062+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:57:20.188062+0000","last_undegraded":"2024-09-15T21:57:20.188062+0000","last_fullsized":"2024-09-15T21:57:20.188062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278543+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:57:20.278543+0000","last_peered":"2024-09-15T21:57:20.278543+0000","last_clean":"2024-09-15T21:57:20.278543+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:57:20.278543+0000","last_undegraded":"2024-09-15T21:57:20.278543+0000","last_fullsized":"2024-09-15T21:57:20.278543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'42","reported_seq":301,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712883+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:57:20.712883+0000","last_peered":"2024-09-15T21:57:20.712883+0000","last_clean":"2024-09-15T21:57:20.712883+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:57:20.712883+0000","last_undegraded":"2024-09-15T21:57:20.712883+0000","last_fullsized":"2024-09-15T21:57:20.712883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68588,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":218,"num_read_kb":175,"num_write":44,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275841+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:57:20.275841+0000","last_peered":"2024-09-15T21:57:20.275841+0000","last_clean":"2024-09-15T21:57:20.275841+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:57:20.275841+0000","last_undegraded":"2024-09-15T21:57:20.275841+0000","last_fullsized":"2024-09-15T21:57:20.275841+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463470+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:57:20.463470+0000","last_peered":"2024-09-15T21:57:20.463470+0000","last_clean":"2024-09-15T21:57:20.463470+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:57:20.463470+0000","last_undegraded":"2024-09-15T21:57:20.463470+0000","last_fullsized":"2024-09-15T21:57:20.463470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188450+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:57:20.188450+0000","last_peered":"2024-09-15T21:57:20.188450+0000","last_clean":"2024-09-15T21:57:20.188450+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:57:20.188450+0000","last_undegraded":"2024-09-15T21:57:20.188450+0000","last_fullsized":"2024-09-15T21:57:20.188450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190331+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:57:20.190331+0000","last_peered":"2024-09-15T21:57:20.190331+0000","last_clean":"2024-09-15T21:57:20.190331+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:57:20.190331+0000","last_undegraded":"2024-09-15T21:57:20.190331+0000","last_fullsized":"2024-09-15T21:57:20.190331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467936+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:57:20.467936+0000","last_peered":"2024-09-15T21:57:20.467936+0000","last_clean":"2024-09-15T21:57:20.467936+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:57:20.467936+0000","last_undegraded":"2024-09-15T21:57:20.467936+0000","last_fullsized":"2024-09-15T21:57:20.467936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461005+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:57:20.461005+0000","last_peered":"2024-09-15T21:57:20.461005+0000","last_clean":"2024-09-15T21:57:20.461005+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:57:20.461005+0000","last_undegraded":"2024-09-15T21:57:20.461005+0000","last_fullsized":"2024-09-15T21:57:20.461005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275613+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:57:20.275613+0000","last_peered":"2024-09-15T21:57:20.275613+0000","last_clean":"2024-09-15T21:57:20.275613+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:57:20.275613+0000","last_undegraded":"2024-09-15T21:57:20.275613+0000","last_fullsized":"2024-09-15T21:57:20.275613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463309+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:57:20.463309+0000","last_peered":"2024-09-15T21:57:20.463309+0000","last_clean":"2024-09-15T21:57:20.463309+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:57:20.463309+0000","last_undegraded":"2024-09-15T21:57:20.463309+0000","last_fullsized":"2024-09-15T21:57:20.463309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'8","reported_seq":254,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712100+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:57:20.712100+0000","last_peered":"2024-09-15T21:57:20.712100+0000","last_clean":"2024-09-15T21:57:20.712100+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:57:20.712100+0000","last_undegraded":"2024-09-15T21:57:20.712100+0000","last_fullsized":"2024-09-15T21:57:20.712100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":206,"num_read_kb":206,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278726+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:57:20.278726+0000","last_peered":"2024-09-15T21:57:20.278726+0000","last_clean":"2024-09-15T21:57:20.278726+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:57:20.278726+0000","last_undegraded":"2024-09-15T21:57:20.278726+0000","last_fullsized":"2024-09-15T21:57:20.278726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467332+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:57:20.467332+0000","last_peered":"2024-09-15T21:57:20.467332+0000","last_clean":"2024-09-15T21:57:20.467332+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:57:20.467332+0000","last_undegraded":"2024-09-15T21:57:20.467332+0000","last_fullsized":"2024-09-15T21:57:20.467332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275825+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:57:20.275825+0000","last_peered":"2024-09-15T21:57:20.275825+0000","last_clean":"2024-09-15T21:57:20.275825+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:57:20.275825+0000","last_undegraded":"2024-09-15T21:57:20.275825+0000","last_fullsized":"2024-09-15T21:57:20.275825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187675+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:57:20.187675+0000","last_peered":"2024-09-15T21:57:20.187675+0000","last_clean":"2024-09-15T21:57:20.187675+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:57:20.187675+0000","last_undegraded":"2024-09-15T21:57:20.187675+0000","last_fullsized":"2024-09-15T21:57:20.187675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472373+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:57:20.472373+0000","last_peered":"2024-09-15T21:57:20.472373+0000","last_clean":"2024-09-15T21:57:20.472373+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:57:20.472373+0000","last_undegraded":"2024-09-15T21:57:20.472373+0000","last_fullsized":"2024-09-15T21:57:20.472373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278372+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:57:20.278372+0000","last_peered":"2024-09-15T21:57:20.278372+0000","last_clean":"2024-09-15T21:57:20.278372+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:57:20.278372+0000","last_undegraded":"2024-09-15T21:57:20.278372+0000","last_fullsized":"2024-09-15T21:57:20.278372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185595+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:57:20.185595+0000","last_peered":"2024-09-15T21:57:20.185595+0000","last_clean":"2024-09-15T21:57:20.185595+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:57:20.185595+0000","last_undegraded":"2024-09-15T21:57:20.185595+0000","last_fullsized":"2024-09-15T21:57:20.185595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285572+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:57:20.285572+0000","last_peered":"2024-09-15T21:57:20.285572+0000","last_clean":"2024-09-15T21:57:20.285572+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:57:20.285572+0000","last_undegraded":"2024-09-15T21:57:20.285572+0000","last_fullsized":"2024-09-15T21:57:20.285572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187909+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:57:20.187909+0000","last_peered":"2024-09-15T21:57:20.187909+0000","last_clean":"2024-09-15T21:57:20.187909+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:57:20.187909+0000","last_undegraded":"2024-09-15T21:57:20.187909+0000","last_fullsized":"2024-09-15T21:57:20.187909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396115+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:57:20.396115+0000","last_peered":"2024-09-15T21:57:20.396115+0000","last_clean":"2024-09-15T21:57:20.396115+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:57:20.396115+0000","last_undegraded":"2024-09-15T21:57:20.396115+0000","last_fullsized":"2024-09-15T21:57:20.396115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'9","reported_seq":158,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712626+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:57:20.712626+0000","last_peered":"2024-09-15T21:57:20.712626+0000","last_clean":"2024-09-15T21:57:20.712626+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:57:20.712626+0000","last_undegraded":"2024-09-15T21:57:20.712626+0000","last_fullsized":"2024-09-15T21:57:20.712626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1857,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":109,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278171+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:57:20.278171+0000","last_peered":"2024-09-15T21:57:20.278171+0000","last_clean":"2024-09-15T21:57:20.278171+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:57:20.278171+0000","last_undegraded":"2024-09-15T21:57:20.278171+0000","last_fullsized":"2024-09-15T21:57:20.278171+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396197+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:57:20.396197+0000","last_peered":"2024-09-15T21:57:20.396197+0000","last_clean":"2024-09-15T21:57:20.396197+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:57:20.396197+0000","last_undegraded":"2024-09-15T21:57:20.396197+0000","last_fullsized":"2024-09-15T21:57:20.396197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471807+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:57:20.471807+0000","last_peered":"2024-09-15T21:57:20.471807+0000","last_clean":"2024-09-15T21:57:20.471807+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:57:20.471807+0000","last_undegraded":"2024-09-15T21:57:20.471807+0000","last_fullsized":"2024-09-15T21:57:20.471807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396260+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:57:20.396260+0000","last_peered":"2024-09-15T21:57:20.396260+0000","last_clean":"2024-09-15T21:57:20.396260+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:57:20.396260+0000","last_undegraded":"2024-09-15T21:57:20.396260+0000","last_fullsized":"2024-09-15T21:57:20.396260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395966+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:57:20.395966+0000","last_peered":"2024-09-15T21:57:20.395966+0000","last_clean":"2024-09-15T21:57:20.395966+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:57:20.395966+0000","last_undegraded":"2024-09-15T21:57:20.395966+0000","last_fullsized":"2024-09-15T21:57:20.395966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466952+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:57:20.466952+0000","last_peered":"2024-09-15T21:57:20.466952+0000","last_clean":"2024-09-15T21:57:20.466952+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:57:20.466952+0000","last_undegraded":"2024-09-15T21:57:20.466952+0000","last_fullsized":"2024-09-15T21:57:20.466952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186090+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:57:20.186090+0000","last_peered":"2024-09-15T21:57:20.186090+0000","last_clean":"2024-09-15T21:57:20.186090+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:57:20.186090+0000","last_undegraded":"2024-09-15T21:57:20.186090+0000","last_fullsized":"2024-09-15T21:57:20.186090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472240+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:57:20.472240+0000","last_peered":"2024-09-15T21:57:20.472240+0000","last_clean":"2024-09-15T21:57:20.472240+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:57:20.472240+0000","last_undegraded":"2024-09-15T21:57:20.472240+0000","last_fullsized":"2024-09-15T21:57:20.472240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396338+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:57:20.396338+0000","last_peered":"2024-09-15T21:57:20.396338+0000","last_clean":"2024-09-15T21:57:20.396338+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:57:20.396338+0000","last_undegraded":"2024-09-15T21:57:20.396338+0000","last_fullsized":"2024-09-15T21:57:20.396338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280054+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:57:20.280054+0000","last_peered":"2024-09-15T21:57:20.280054+0000","last_clean":"2024-09-15T21:57:20.280054+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:57:20.280054+0000","last_undegraded":"2024-09-15T21:57:20.280054+0000","last_fullsized":"2024-09-15T21:57:20.280054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187475+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:57:20.187475+0000","last_peered":"2024-09-15T21:57:20.187475+0000","last_clean":"2024-09-15T21:57:20.187475+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:57:20.187475+0000","last_undegraded":"2024-09-15T21:57:20.187475+0000","last_fullsized":"2024-09-15T21:57:20.187475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278973+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:57:20.278973+0000","last_peered":"2024-09-15T21:57:20.278973+0000","last_clean":"2024-09-15T21:57:20.278973+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:57:20.278973+0000","last_undegraded":"2024-09-15T21:57:20.278973+0000","last_fullsized":"2024-09-15T21:57:20.278973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463016+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:57:20.463016+0000","last_peered":"2024-09-15T21:57:20.463016+0000","last_clean":"2024-09-15T21:57:20.463016+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:57:20.463016+0000","last_undegraded":"2024-09-15T21:57:20.463016+0000","last_fullsized":"2024-09-15T21:57:20.463016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186044+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:57:20.186044+0000","last_peered":"2024-09-15T21:57:20.186044+0000","last_clean":"2024-09-15T21:57:20.186044+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:57:20.186044+0000","last_undegraded":"2024-09-15T21:57:20.186044+0000","last_fullsized":"2024-09-15T21:57:20.186044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472203+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:57:20.472203+0000","last_peered":"2024-09-15T21:57:20.472203+0000","last_clean":"2024-09-15T21:57:20.472203+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:57:20.472203+0000","last_undegraded":"2024-09-15T21:57:20.472203+0000","last_fullsized":"2024-09-15T21:57:20.472203+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396319+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:57:20.396319+0000","last_peered":"2024-09-15T21:57:20.396319+0000","last_clean":"2024-09-15T21:57:20.396319+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:57:20.396319+0000","last_undegraded":"2024-09-15T21:57:20.396319+0000","last_fullsized":"2024-09-15T21:57:20.396319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276457+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:57:20.276457+0000","last_peered":"2024-09-15T21:57:20.276457+0000","last_clean":"2024-09-15T21:57:20.276457+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:57:20.276457+0000","last_undegraded":"2024-09-15T21:57:20.276457+0000","last_fullsized":"2024-09-15T21:57:20.276457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472556+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:57:20.472556+0000","last_peered":"2024-09-15T21:57:20.472556+0000","last_clean":"2024-09-15T21:57:20.472556+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:57:20.472556+0000","last_undegraded":"2024-09-15T21:57:20.472556+0000","last_fullsized":"2024-09-15T21:57:20.472556+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462857+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:57:20.462857+0000","last_peered":"2024-09-15T21:57:20.462857+0000","last_clean":"2024-09-15T21:57:20.462857+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:57:20.462857+0000","last_undegraded":"2024-09-15T21:57:20.462857+0000","last_fullsized":"2024-09-15T21:57:20.462857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466723+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:57:20.466723+0000","last_peered":"2024-09-15T21:57:20.466723+0000","last_clean":"2024-09-15T21:57:20.466723+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:57:20.466723+0000","last_undegraded":"2024-09-15T21:57:20.466723+0000","last_fullsized":"2024-09-15T21:57:20.466723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466886+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:57:20.466886+0000","last_peered":"2024-09-15T21:57:20.466886+0000","last_clean":"2024-09-15T21:57:20.466886+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:57:20.466886+0000","last_undegraded":"2024-09-15T21:57:20.466886+0000","last_fullsized":"2024-09-15T21:57:20.466886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186173+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:57:20.186173+0000","last_peered":"2024-09-15T21:57:20.186173+0000","last_clean":"2024-09-15T21:57:20.186173+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:57:20.186173+0000","last_undegraded":"2024-09-15T21:57:20.186173+0000","last_fullsized":"2024-09-15T21:57:20.186173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276015+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:57:20.276015+0000","last_peered":"2024-09-15T21:57:20.276015+0000","last_clean":"2024-09-15T21:57:20.276015+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:57:20.276015+0000","last_undegraded":"2024-09-15T21:57:20.276015+0000","last_fullsized":"2024-09-15T21:57:20.276015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461069+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:57:20.461069+0000","last_peered":"2024-09-15T21:57:20.461069+0000","last_clean":"2024-09-15T21:57:20.461069+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:57:20.461069+0000","last_undegraded":"2024-09-15T21:57:20.461069+0000","last_fullsized":"2024-09-15T21:57:20.461069+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.286073+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:57:20.286073+0000","last_peered":"2024-09-15T21:57:20.286073+0000","last_clean":"2024-09-15T21:57:20.286073+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:57:20.286073+0000","last_undegraded":"2024-09-15T21:57:20.286073+0000","last_fullsized":"2024-09-15T21:57:20.286073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463316+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:57:20.463316+0000","last_peered":"2024-09-15T21:57:20.463316+0000","last_clean":"2024-09-15T21:57:20.463316+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:57:20.463316+0000","last_undegraded":"2024-09-15T21:57:20.463316+0000","last_fullsized":"2024-09-15T21:57:20.463316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278186+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:57:20.278186+0000","last_peered":"2024-09-15T21:57:20.278186+0000","last_clean":"2024-09-15T21:57:20.278186+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:57:20.278186+0000","last_undegraded":"2024-09-15T21:57:20.278186+0000","last_fullsized":"2024-09-15T21:57:20.278186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460530+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:57:20.460530+0000","last_peered":"2024-09-15T21:57:20.460530+0000","last_clean":"2024-09-15T21:57:20.460530+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:57:20.460530+0000","last_undegraded":"2024-09-15T21:57:20.460530+0000","last_fullsized":"2024-09-15T21:57:20.460530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285510+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:57:20.285510+0000","last_peered":"2024-09-15T21:57:20.285510+0000","last_clean":"2024-09-15T21:57:20.285510+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:57:20.285510+0000","last_undegraded":"2024-09-15T21:57:20.285510+0000","last_fullsized":"2024-09-15T21:57:20.285510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278179+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:57:20.278179+0000","last_peered":"2024-09-15T21:57:20.278179+0000","last_clean":"2024-09-15T21:57:20.278179+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:57:20.278179+0000","last_undegraded":"2024-09-15T21:57:20.278179+0000","last_fullsized":"2024-09-15T21:57:20.278179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190742+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:57:20.190742+0000","last_peered":"2024-09-15T21:57:20.190742+0000","last_clean":"2024-09-15T21:57:20.190742+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:57:20.190742+0000","last_undegraded":"2024-09-15T21:57:20.190742+0000","last_fullsized":"2024-09-15T21:57:20.190742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467111+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:57:20.467111+0000","last_peered":"2024-09-15T21:57:20.467111+0000","last_clean":"2024-09-15T21:57:20.467111+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:57:20.467111+0000","last_undegraded":"2024-09-15T21:57:20.467111+0000","last_fullsized":"2024-09-15T21:57:20.467111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467430+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:57:20.467430+0000","last_peered":"2024-09-15T21:57:20.467430+0000","last_clean":"2024-09-15T21:57:20.467430+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:57:20.467430+0000","last_undegraded":"2024-09-15T21:57:20.467430+0000","last_fullsized":"2024-09-15T21:57:20.467430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472483+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:57:20.472483+0000","last_peered":"2024-09-15T21:57:20.472483+0000","last_clean":"2024-09-15T21:57:20.472483+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:57:20.472483+0000","last_undegraded":"2024-09-15T21:57:20.472483+0000","last_fullsized":"2024-09-15T21:57:20.472483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":603167,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1172,"num_read_kb":704,"num_write":389,"num_write_kb":1477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1809501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514538,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1081344,"data_stored":536154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776174,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28900,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716252,"statfs":{"total":95995035648,"available":95965442048,"internally_reserved":0,"allocated":1921024,"data_stored":1330627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939701,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36804,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708348,"statfs":{"total":95995035648,"available":95957348352,"internally_reserved":0,"allocated":1490944,"data_stored":923695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168633,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1486848,"data_stored":927587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397567,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36988,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708164,"statfs":{"total":95995035648,"available":95957159936,"internally_reserved":0,"allocated":1679360,"data_stored":1122064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561090,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":530472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28820,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716332,"statfs":{"total":95995035648,"available":95965523968,"internally_reserved":0,"allocated":1839104,"data_stored":1259476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":534939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28784,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716368,"statfs":{"total":95995035648,"available":95965560832,"internally_reserved":0,"allocated":1617920,"data_stored":1054603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":590670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1150976,"data_stored":600090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":532462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:57:25.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:57:25.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 -- 172.21.15.28:0/1566837375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c101340 msgr2=0x7fab7c10d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 --2- 172.21.15.28:0/1566837375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c101340 0x7fab7c10d870 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fab74009900 tx=0x7fab7402a410 comp rx=0 tx=0).stop 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 -- 172.21.15.28:0/1566837375 shutdown_connections 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 --2- 172.21.15.28:0/1566837375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c101340 0x7fab7c10d870 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 --2- 172.21.15.28:0/1566837375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c1009a0 0x7fab7c100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 --2- 172.21.15.28:0/1566837375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c1069b0 0x7fab7c106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 -- 172.21.15.28:0/1566837375 >> 172.21.15.28:0/1566837375 conn(0x7fab7c0fc6c0 msgr2=0x7fab7c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 -- 172.21.15.28:0/1566837375 shutdown_connections 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.207+0000 7fab82f2a640 1 -- 172.21.15.28:0/1566837375 wait complete. 2024-09-15T21:57:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 Processor -- start 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 -- start start 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c1009a0 0x7fab7c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c101340 0x7fab7c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c1069b0 0x7fab7c19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c111ec0 con 0x7fab7c1009a0 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c111d40 con 0x7fab7c101340 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab82f2a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c112040 con 0x7fab7c1069b0 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c1009a0 0x7fab7c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c1009a0 0x7fab7c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59726/0 (socket says 172.21.15.28:59726) 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 -- 172.21.15.28:0/2479448768 learned_addr learned my addr 172.21.15.28:0/2479448768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab7bfff640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c101340 0x7fab7c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab814c7640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c1069b0 0x7fab7c19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 -- 172.21.15.28:0/2479448768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c1069b0 msgr2=0x7fab7c19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c1069b0 0x7fab7c19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 -- 172.21.15.28:0/2479448768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c101340 msgr2=0x7fab7c198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c101340 0x7fab7c198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 -- 172.21.15.28:0/2479448768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c19d2f0 con 0x7fab7c1009a0 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.208+0000 7fab80cc6640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c1009a0 0x7fab7c198300 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e9f0 tx=0x7fab6400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.209+0000 7fab79ffb640 1 -- 172.21.15.28:0/2479448768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6400cd80 con 0x7fab7c1009a0 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.209+0000 7fab79ffb640 1 -- 172.21.15.28:0/2479448768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab64010910 con 0x7fab7c1009a0 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.209+0000 7fab79ffb640 1 -- 172.21.15.28:0/2479448768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab64018990 con 0x7fab7c1009a0 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.209+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7c19d5e0 con 0x7fab7c1009a0 2024-09-15T21:57:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.209+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7c1a4e70 con 0x7fab7c1009a0 2024-09-15T21:57:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.210+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7c100e00 con 0x7fab7c1009a0 2024-09-15T21:57:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.211+0000 7fab79ffb640 1 -- 172.21.15.28:0/2479448768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fab64010ac0 con 0x7fab7c1009a0 2024-09-15T21:57:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.212+0000 7fab79ffb640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab680a3560 0x7fab680a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.212+0000 7fab7bfff640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab680a3560 0x7fab680a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.212+0000 7fab79ffb640 1 -- 172.21.15.28:0/2479448768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fab640d05b0 con 0x7fab7c1009a0 2024-09-15T21:57:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.213+0000 7fab7bfff640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab680a3560 0x7fab680a5a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0fec70 tx=0x7fab6c0075b0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.217+0000 7fab79ffb640 1 -- 172.21.15.28:0/2479448768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab6409c9e0 con 0x7fab7c1009a0 2024-09-15T21:57:25.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.392+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab7c0fd4d0 con 0x7fab680a3560 2024-09-15T21:57:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.400+0000 7fab79ffb640 1 -- 172.21.15.28:0/2479448768 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339443 (secure 0 0 0) 0x7fab7c0fd4d0 con 0x7fab680a3560 2024-09-15T21:57:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:25.407 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:57:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab680a3560 msgr2=0x7fab680a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab680a3560 0x7fab680a5a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0fec70 tx=0x7fab6c0075b0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c1009a0 msgr2=0x7fab7c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c1009a0 0x7fab7c198300 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e9f0 tx=0x7fab6400eeb0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 shutdown_connections 2024-09-15T21:57:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab680a3560 0x7fab680a5a20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c1069b0 0x7fab7c19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab7c101340 0x7fab7c198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 --2- 172.21.15.28:0/2479448768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7c1009a0 0x7fab7c198300 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 >> 172.21.15.28:0/2479448768 conn(0x7fab7c0fc6c0 msgr2=0x7fab7c0fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:25.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 shutdown_connections 2024-09-15T21:57:25.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.404+0000 7fab82f2a640 1 -- 172.21.15.28:0/2479448768 wait complete. 2024-09-15T21:57:25.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T21:57:25.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Unable to set osd_memory_target on smithi174 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T21:57:25.416 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi203 to 176.0M 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Unable to set osd_memory_target on smithi203 to 184563097: error parsing value: Value '184563097' is below minimum 939524096 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Unable to set osd_memory_target on smithi028 to 185689190: error parsing value: Value '185689190' is below minimum 939524096 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.417 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1035486229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:25.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:25.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:25 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.419 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-09-15T21:57:24.202295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193769,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1308,"num_read_kb":823,"num_write":572,"num_write_kb":4417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":952,"ondisk_log_size":952,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365756,"kb_used_data":16296,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576068,"statfs":{"total":1151940427776,"available":1151565893632,"internally_reserved":0,"allocated":16687104,"data_stored":9942839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993652"},"pg_stats":[{"pgid":"2.1f","version":"93'14","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.469302+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:57:20.469302+0000","last_peered":"2024-09-15T21:57:20.469302+0000","last_clean":"2024-09-15T21:57:20.469302+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:57:20.469302+0000","last_undegraded":"2024-09-15T21:57:20.469302+0000","last_fullsized":"2024-09-15T21:57:20.469302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280359+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:57:20.280359+0000","last_peered":"2024-09-15T21:57:20.280359+0000","last_clean":"2024-09-15T21:57:20.280359+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:57:20.280359+0000","last_undegraded":"2024-09-15T21:57:20.280359+0000","last_fullsized":"2024-09-15T21:57:20.280359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395705+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:57:20.395705+0000","last_peered":"2024-09-15T21:57:20.395705+0000","last_clean":"2024-09-15T21:57:20.395705+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:57:20.395705+0000","last_undegraded":"2024-09-15T21:57:20.395705+0000","last_fullsized":"2024-09-15T21:57:20.395705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280756+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:57:20.280756+0000","last_peered":"2024-09-15T21:57:20.280756+0000","last_clean":"2024-09-15T21:57:20.280756+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:57:20.280756+0000","last_undegraded":"2024-09-15T21:57:20.280756+0000","last_fullsized":"2024-09-15T21:57:20.280756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467940+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:57:20.467940+0000","last_peered":"2024-09-15T21:57:20.467940+0000","last_clean":"2024-09-15T21:57:20.467940+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:57:20.467940+0000","last_undegraded":"2024-09-15T21:57:20.467940+0000","last_fullsized":"2024-09-15T21:57:20.467940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190047+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:57:20.190047+0000","last_peered":"2024-09-15T21:57:20.190047+0000","last_clean":"2024-09-15T21:57:20.190047+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:57:20.190047+0000","last_undegraded":"2024-09-15T21:57:20.190047+0000","last_fullsized":"2024-09-15T21:57:20.190047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278626+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:57:20.278626+0000","last_peered":"2024-09-15T21:57:20.278626+0000","last_clean":"2024-09-15T21:57:20.278626+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:57:20.278626+0000","last_undegraded":"2024-09-15T21:57:20.278626+0000","last_fullsized":"2024-09-15T21:57:20.278626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463402+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:57:20.463402+0000","last_peered":"2024-09-15T21:57:20.463402+0000","last_clean":"2024-09-15T21:57:20.463402+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:57:20.463402+0000","last_undegraded":"2024-09-15T21:57:20.463402+0000","last_fullsized":"2024-09-15T21:57:20.463402+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190387+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:57:20.190387+0000","last_peered":"2024-09-15T21:57:20.190387+0000","last_clean":"2024-09-15T21:57:20.190387+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:57:20.190387+0000","last_undegraded":"2024-09-15T21:57:20.190387+0000","last_fullsized":"2024-09-15T21:57:20.190387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472029+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:57:20.472029+0000","last_peered":"2024-09-15T21:57:20.472029+0000","last_clean":"2024-09-15T21:57:20.472029+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:57:20.472029+0000","last_undegraded":"2024-09-15T21:57:20.472029+0000","last_fullsized":"2024-09-15T21:57:20.472029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188165+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:57:20.188165+0000","last_peered":"2024-09-15T21:57:20.188165+0000","last_clean":"2024-09-15T21:57:20.188165+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:57:20.188165+0000","last_undegraded":"2024-09-15T21:57:20.188165+0000","last_fullsized":"2024-09-15T21:57:20.188165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190320+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:57:20.190320+0000","last_peered":"2024-09-15T21:57:20.190320+0000","last_clean":"2024-09-15T21:57:20.190320+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:57:20.190320+0000","last_undegraded":"2024-09-15T21:57:20.190320+0000","last_fullsized":"2024-09-15T21:57:20.190320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460655+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:57:20.460655+0000","last_peered":"2024-09-15T21:57:20.460655+0000","last_clean":"2024-09-15T21:57:20.460655+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:57:20.460655+0000","last_undegraded":"2024-09-15T21:57:20.460655+0000","last_fullsized":"2024-09-15T21:57:20.460655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278262+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:57:20.278262+0000","last_peered":"2024-09-15T21:57:20.278262+0000","last_clean":"2024-09-15T21:57:20.278262+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:57:20.278262+0000","last_undegraded":"2024-09-15T21:57:20.278262+0000","last_fullsized":"2024-09-15T21:57:20.278262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190739+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:57:20.190739+0000","last_peered":"2024-09-15T21:57:20.190739+0000","last_clean":"2024-09-15T21:57:20.190739+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:57:20.190739+0000","last_undegraded":"2024-09-15T21:57:20.190739+0000","last_fullsized":"2024-09-15T21:57:20.190739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279728+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:57:20.279728+0000","last_peered":"2024-09-15T21:57:20.279728+0000","last_clean":"2024-09-15T21:57:20.279728+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:57:20.279728+0000","last_undegraded":"2024-09-15T21:57:20.279728+0000","last_fullsized":"2024-09-15T21:57:20.279728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"92'4","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471831+0000","last_change":"2024-09-15T21:56:58.926427+0000","last_active":"2024-09-15T21:57:20.471831+0000","last_peered":"2024-09-15T21:57:20.471831+0000","last_clean":"2024-09-15T21:57:20.471831+0000","last_became_active":"2024-09-15T21:56:58.922755+0000","last_became_peered":"2024-09-15T21:56:58.922755+0000","last_unstale":"2024-09-15T21:57:20.471831+0000","last_undegraded":"2024-09-15T21:57:20.471831+0000","last_fullsized":"2024-09-15T21:57:20.471831+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:22.033354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:57:20.190660+0000","last_peered":"2024-09-15T21:57:20.190660+0000","last_clean":"2024-09-15T21:57:20.190660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:57:20.190660+0000","last_undegraded":"2024-09-15T21:57:20.190660+0000","last_fullsized":"2024-09-15T21:57:20.190660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471869+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:57:20.471869+0000","last_peered":"2024-09-15T21:57:20.471869+0000","last_clean":"2024-09-15T21:57:20.471869+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:57:20.471869+0000","last_undegraded":"2024-09-15T21:57:20.471869+0000","last_fullsized":"2024-09-15T21:57:20.471869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279173+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:57:20.279173+0000","last_peered":"2024-09-15T21:57:20.279173+0000","last_clean":"2024-09-15T21:57:20.279173+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:57:20.279173+0000","last_undegraded":"2024-09-15T21:57:20.279173+0000","last_fullsized":"2024-09-15T21:57:20.279173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276125+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:57:20.276125+0000","last_peered":"2024-09-15T21:57:20.276125+0000","last_clean":"2024-09-15T21:57:20.276125+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:57:20.276125+0000","last_undegraded":"2024-09-15T21:57:20.276125+0000","last_fullsized":"2024-09-15T21:57:20.276125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278075+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:57:20.278075+0000","last_peered":"2024-09-15T21:57:20.278075+0000","last_clean":"2024-09-15T21:57:20.278075+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:57:20.278075+0000","last_undegraded":"2024-09-15T21:57:20.278075+0000","last_fullsized":"2024-09-15T21:57:20.278075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462853+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:57:20.462853+0000","last_peered":"2024-09-15T21:57:20.462853+0000","last_clean":"2024-09-15T21:57:20.462853+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:57:20.462853+0000","last_undegraded":"2024-09-15T21:57:20.462853+0000","last_fullsized":"2024-09-15T21:57:20.462853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190763+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:57:20.190763+0000","last_peered":"2024-09-15T21:57:20.190763+0000","last_clean":"2024-09-15T21:57:20.190763+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:57:20.190763+0000","last_undegraded":"2024-09-15T21:57:20.190763+0000","last_fullsized":"2024-09-15T21:57:20.190763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467562+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:57:20.467562+0000","last_peered":"2024-09-15T21:57:20.467562+0000","last_clean":"2024-09-15T21:57:20.467562+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:57:20.467562+0000","last_undegraded":"2024-09-15T21:57:20.467562+0000","last_fullsized":"2024-09-15T21:57:20.467562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190932+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:57:20.190932+0000","last_peered":"2024-09-15T21:57:20.190932+0000","last_clean":"2024-09-15T21:57:20.190932+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:57:20.190932+0000","last_undegraded":"2024-09-15T21:57:20.190932+0000","last_fullsized":"2024-09-15T21:57:20.190932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395988+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:57:20.395988+0000","last_peered":"2024-09-15T21:57:20.395988+0000","last_clean":"2024-09-15T21:57:20.395988+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:57:20.395988+0000","last_undegraded":"2024-09-15T21:57:20.395988+0000","last_fullsized":"2024-09-15T21:57:20.395988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280166+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:57:20.280166+0000","last_peered":"2024-09-15T21:57:20.280166+0000","last_clean":"2024-09-15T21:57:20.280166+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:57:20.280166+0000","last_undegraded":"2024-09-15T21:57:20.280166+0000","last_fullsized":"2024-09-15T21:57:20.280166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'53","reported_seq":117,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.416465+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:57:20.416465+0000","last_peered":"2024-09-15T21:57:20.416465+0000","last_clean":"2024-09-15T21:57:20.416465+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:57:20.416465+0000","last_undegraded":"2024-09-15T21:57:20.416465+0000","last_fullsized":"2024-09-15T21:57:20.416465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":94,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.277996+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:57:20.277996+0000","last_peered":"2024-09-15T21:57:20.277996+0000","last_clean":"2024-09-15T21:57:20.277996+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:57:20.277996+0000","last_undegraded":"2024-09-15T21:57:20.277996+0000","last_fullsized":"2024-09-15T21:57:20.277996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186161+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:57:20.186161+0000","last_peered":"2024-09-15T21:57:20.186161+0000","last_clean":"2024-09-15T21:57:20.186161+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:57:20.186161+0000","last_undegraded":"2024-09-15T21:57:20.186161+0000","last_fullsized":"2024-09-15T21:57:20.186161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278630+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:57:20.278630+0000","last_peered":"2024-09-15T21:57:20.278630+0000","last_clean":"2024-09-15T21:57:20.278630+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:57:20.278630+0000","last_undegraded":"2024-09-15T21:57:20.278630+0000","last_fullsized":"2024-09-15T21:57:20.278630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472019+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:57:20.472019+0000","last_peered":"2024-09-15T21:57:20.472019+0000","last_clean":"2024-09-15T21:57:20.472019+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:57:20.472019+0000","last_undegraded":"2024-09-15T21:57:20.472019+0000","last_fullsized":"2024-09-15T21:57:20.472019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285785+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:57:20.285785+0000","last_peered":"2024-09-15T21:57:20.285785+0000","last_clean":"2024-09-15T21:57:20.285785+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:57:20.285785+0000","last_undegraded":"2024-09-15T21:57:20.285785+0000","last_fullsized":"2024-09-15T21:57:20.285785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276329+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:57:20.276329+0000","last_peered":"2024-09-15T21:57:20.276329+0000","last_clean":"2024-09-15T21:57:20.276329+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:57:20.276329+0000","last_undegraded":"2024-09-15T21:57:20.276329+0000","last_fullsized":"2024-09-15T21:57:20.276329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278087+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:57:20.278087+0000","last_peered":"2024-09-15T21:57:20.278087+0000","last_clean":"2024-09-15T21:57:20.278087+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:57:20.278087+0000","last_undegraded":"2024-09-15T21:57:20.278087+0000","last_fullsized":"2024-09-15T21:57:20.278087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285570+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:57:20.285570+0000","last_peered":"2024-09-15T21:57:20.285570+0000","last_clean":"2024-09-15T21:57:20.285570+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:57:20.285570+0000","last_undegraded":"2024-09-15T21:57:20.285570+0000","last_fullsized":"2024-09-15T21:57:20.285570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.711889+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:57:20.711889+0000","last_peered":"2024-09-15T21:57:20.711889+0000","last_clean":"2024-09-15T21:57:20.711889+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:57:20.711889+0000","last_undegraded":"2024-09-15T21:57:20.711889+0000","last_fullsized":"2024-09-15T21:57:20.711889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185976+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:57:20.185976+0000","last_peered":"2024-09-15T21:57:20.185976+0000","last_clean":"2024-09-15T21:57:20.185976+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:57:20.185976+0000","last_undegraded":"2024-09-15T21:57:20.185976+0000","last_fullsized":"2024-09-15T21:57:20.185976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187498+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:57:20.187498+0000","last_peered":"2024-09-15T21:57:20.187498+0000","last_clean":"2024-09-15T21:57:20.187498+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:57:20.187498+0000","last_undegraded":"2024-09-15T21:57:20.187498+0000","last_fullsized":"2024-09-15T21:57:20.187498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":151,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466327+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:57:20.466327+0000","last_peered":"2024-09-15T21:57:20.466327+0000","last_clean":"2024-09-15T21:57:20.466327+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:57:20.466327+0000","last_undegraded":"2024-09-15T21:57:20.466327+0000","last_fullsized":"2024-09-15T21:57:20.466327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186270+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:57:20.186270+0000","last_peered":"2024-09-15T21:57:20.186270+0000","last_clean":"2024-09-15T21:57:20.186270+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:57:20.186270+0000","last_undegraded":"2024-09-15T21:57:20.186270+0000","last_fullsized":"2024-09-15T21:57:20.186270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185590+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:57:20.185590+0000","last_peered":"2024-09-15T21:57:20.185590+0000","last_clean":"2024-09-15T21:57:20.185590+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:57:20.185590+0000","last_undegraded":"2024-09-15T21:57:20.185590+0000","last_fullsized":"2024-09-15T21:57:20.185590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460124+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:57:20.460124+0000","last_peered":"2024-09-15T21:57:20.460124+0000","last_clean":"2024-09-15T21:57:20.460124+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:57:20.460124+0000","last_undegraded":"2024-09-15T21:57:20.460124+0000","last_fullsized":"2024-09-15T21:57:20.460124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188088+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:57:20.188088+0000","last_peered":"2024-09-15T21:57:20.188088+0000","last_clean":"2024-09-15T21:57:20.188088+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:57:20.188088+0000","last_undegraded":"2024-09-15T21:57:20.188088+0000","last_fullsized":"2024-09-15T21:57:20.188088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279569+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:57:20.279569+0000","last_peered":"2024-09-15T21:57:20.279569+0000","last_clean":"2024-09-15T21:57:20.279569+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:57:20.279569+0000","last_undegraded":"2024-09-15T21:57:20.279569+0000","last_fullsized":"2024-09-15T21:57:20.279569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190561+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:57:20.190561+0000","last_peered":"2024-09-15T21:57:20.190561+0000","last_clean":"2024-09-15T21:57:20.190561+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:57:20.190561+0000","last_undegraded":"2024-09-15T21:57:20.190561+0000","last_fullsized":"2024-09-15T21:57:20.190561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460734+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:57:20.460734+0000","last_peered":"2024-09-15T21:57:20.460734+0000","last_clean":"2024-09-15T21:57:20.460734+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:57:20.460734+0000","last_undegraded":"2024-09-15T21:57:20.460734+0000","last_fullsized":"2024-09-15T21:57:20.460734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460868+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:57:20.460868+0000","last_peered":"2024-09-15T21:57:20.460868+0000","last_clean":"2024-09-15T21:57:20.460868+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:57:20.460868+0000","last_undegraded":"2024-09-15T21:57:20.460868+0000","last_fullsized":"2024-09-15T21:57:20.460868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461076+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:57:20.461076+0000","last_peered":"2024-09-15T21:57:20.461076+0000","last_clean":"2024-09-15T21:57:20.461076+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:57:20.461076+0000","last_undegraded":"2024-09-15T21:57:20.461076+0000","last_fullsized":"2024-09-15T21:57:20.461076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.719909+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:57:20.719909+0000","last_peered":"2024-09-15T21:57:20.719909+0000","last_clean":"2024-09-15T21:57:20.719909+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:57:20.719909+0000","last_undegraded":"2024-09-15T21:57:20.719909+0000","last_fullsized":"2024-09-15T21:57:20.719909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462918+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:57:20.462918+0000","last_peered":"2024-09-15T21:57:20.462918+0000","last_clean":"2024-09-15T21:57:20.462918+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:57:20.462918+0000","last_undegraded":"2024-09-15T21:57:20.462918+0000","last_fullsized":"2024-09-15T21:57:20.462918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460824+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:57:20.460824+0000","last_peered":"2024-09-15T21:57:20.460824+0000","last_clean":"2024-09-15T21:57:20.460824+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:57:20.460824+0000","last_undegraded":"2024-09-15T21:57:20.460824+0000","last_fullsized":"2024-09-15T21:57:20.460824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'41","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.413534+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:57:20.413534+0000","last_peered":"2024-09-15T21:57:20.413534+0000","last_clean":"2024-09-15T21:57:20.413534+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:57:20.413534+0000","last_undegraded":"2024-09-15T21:57:20.413534+0000","last_fullsized":"2024-09-15T21:57:20.413534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466957+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:57:20.466957+0000","last_peered":"2024-09-15T21:57:20.466957+0000","last_clean":"2024-09-15T21:57:20.466957+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:57:20.466957+0000","last_undegraded":"2024-09-15T21:57:20.466957+0000","last_fullsized":"2024-09-15T21:57:20.466957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460987+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:57:20.460987+0000","last_peered":"2024-09-15T21:57:20.460987+0000","last_clean":"2024-09-15T21:57:20.460987+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:57:20.460987+0000","last_undegraded":"2024-09-15T21:57:20.460987+0000","last_fullsized":"2024-09-15T21:57:20.460987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186150+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:57:20.186150+0000","last_peered":"2024-09-15T21:57:20.186150+0000","last_clean":"2024-09-15T21:57:20.186150+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:57:20.186150+0000","last_undegraded":"2024-09-15T21:57:20.186150+0000","last_fullsized":"2024-09-15T21:57:20.186150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187377+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:57:20.187377+0000","last_peered":"2024-09-15T21:57:20.187377+0000","last_clean":"2024-09-15T21:57:20.187377+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:57:20.187377+0000","last_undegraded":"2024-09-15T21:57:20.187377+0000","last_fullsized":"2024-09-15T21:57:20.187377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472329+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:57:20.472329+0000","last_peered":"2024-09-15T21:57:20.472329+0000","last_clean":"2024-09-15T21:57:20.472329+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:57:20.472329+0000","last_undegraded":"2024-09-15T21:57:20.472329+0000","last_fullsized":"2024-09-15T21:57:20.472329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278620+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:57:20.278620+0000","last_peered":"2024-09-15T21:57:20.278620+0000","last_clean":"2024-09-15T21:57:20.278620+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:57:20.278620+0000","last_undegraded":"2024-09-15T21:57:20.278620+0000","last_fullsized":"2024-09-15T21:57:20.278620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187448+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:57:20.187448+0000","last_peered":"2024-09-15T21:57:20.187448+0000","last_clean":"2024-09-15T21:57:20.187448+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:57:20.187448+0000","last_undegraded":"2024-09-15T21:57:20.187448+0000","last_fullsized":"2024-09-15T21:57:20.187448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187552+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:57:20.187552+0000","last_peered":"2024-09-15T21:57:20.187552+0000","last_clean":"2024-09-15T21:57:20.187552+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:57:20.187552+0000","last_undegraded":"2024-09-15T21:57:20.187552+0000","last_fullsized":"2024-09-15T21:57:20.187552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278144+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:57:20.278144+0000","last_peered":"2024-09-15T21:57:20.278144+0000","last_clean":"2024-09-15T21:57:20.278144+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:57:20.278144+0000","last_undegraded":"2024-09-15T21:57:20.278144+0000","last_fullsized":"2024-09-15T21:57:20.278144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186058+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:57:20.186058+0000","last_peered":"2024-09-15T21:57:20.186058+0000","last_clean":"2024-09-15T21:57:20.186058+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:57:20.186058+0000","last_undegraded":"2024-09-15T21:57:20.186058+0000","last_fullsized":"2024-09-15T21:57:20.186058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460653+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:57:20.460653+0000","last_peered":"2024-09-15T21:57:20.460653+0000","last_clean":"2024-09-15T21:57:20.460653+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:57:20.460653+0000","last_undegraded":"2024-09-15T21:57:20.460653+0000","last_fullsized":"2024-09-15T21:57:20.460653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":154,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.720943+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:57:20.720943+0000","last_peered":"2024-09-15T21:57:20.720943+0000","last_clean":"2024-09-15T21:57:20.720943+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:57:20.720943+0000","last_undegraded":"2024-09-15T21:57:20.720943+0000","last_fullsized":"2024-09-15T21:57:20.720943+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276236+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:57:20.276236+0000","last_peered":"2024-09-15T21:57:20.276236+0000","last_clean":"2024-09-15T21:57:20.276236+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:57:20.276236+0000","last_undegraded":"2024-09-15T21:57:20.276236+0000","last_fullsized":"2024-09-15T21:57:20.276236+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460781+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:57:20.460781+0000","last_peered":"2024-09-15T21:57:20.460781+0000","last_clean":"2024-09-15T21:57:20.460781+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:57:20.460781+0000","last_undegraded":"2024-09-15T21:57:20.460781+0000","last_fullsized":"2024-09-15T21:57:20.460781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187760+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:57:20.187760+0000","last_peered":"2024-09-15T21:57:20.187760+0000","last_clean":"2024-09-15T21:57:20.187760+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:57:20.187760+0000","last_undegraded":"2024-09-15T21:57:20.187760+0000","last_fullsized":"2024-09-15T21:57:20.187760+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472044+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:57:20.472044+0000","last_peered":"2024-09-15T21:57:20.472044+0000","last_clean":"2024-09-15T21:57:20.472044+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:57:20.472044+0000","last_undegraded":"2024-09-15T21:57:20.472044+0000","last_fullsized":"2024-09-15T21:57:20.472044+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185779+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:57:20.185779+0000","last_peered":"2024-09-15T21:57:20.185779+0000","last_clean":"2024-09-15T21:57:20.185779+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:57:20.185779+0000","last_undegraded":"2024-09-15T21:57:20.185779+0000","last_fullsized":"2024-09-15T21:57:20.185779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395654+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:57:20.395654+0000","last_peered":"2024-09-15T21:57:20.395654+0000","last_clean":"2024-09-15T21:57:20.395654+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:57:20.395654+0000","last_undegraded":"2024-09-15T21:57:20.395654+0000","last_fullsized":"2024-09-15T21:57:20.395654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188062+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:57:20.188062+0000","last_peered":"2024-09-15T21:57:20.188062+0000","last_clean":"2024-09-15T21:57:20.188062+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:57:20.188062+0000","last_undegraded":"2024-09-15T21:57:20.188062+0000","last_fullsized":"2024-09-15T21:57:20.188062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278543+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:57:20.278543+0000","last_peered":"2024-09-15T21:57:20.278543+0000","last_clean":"2024-09-15T21:57:20.278543+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:57:20.278543+0000","last_undegraded":"2024-09-15T21:57:20.278543+0000","last_fullsized":"2024-09-15T21:57:20.278543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'42","reported_seq":301,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712883+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:57:20.712883+0000","last_peered":"2024-09-15T21:57:20.712883+0000","last_clean":"2024-09-15T21:57:20.712883+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:57:20.712883+0000","last_undegraded":"2024-09-15T21:57:20.712883+0000","last_fullsized":"2024-09-15T21:57:20.712883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68588,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":218,"num_read_kb":175,"num_write":44,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275841+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:57:20.275841+0000","last_peered":"2024-09-15T21:57:20.275841+0000","last_clean":"2024-09-15T21:57:20.275841+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:57:20.275841+0000","last_undegraded":"2024-09-15T21:57:20.275841+0000","last_fullsized":"2024-09-15T21:57:20.275841+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463470+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:57:20.463470+0000","last_peered":"2024-09-15T21:57:20.463470+0000","last_clean":"2024-09-15T21:57:20.463470+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:57:20.463470+0000","last_undegraded":"2024-09-15T21:57:20.463470+0000","last_fullsized":"2024-09-15T21:57:20.463470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188450+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:57:20.188450+0000","last_peered":"2024-09-15T21:57:20.188450+0000","last_clean":"2024-09-15T21:57:20.188450+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:57:20.188450+0000","last_undegraded":"2024-09-15T21:57:20.188450+0000","last_fullsized":"2024-09-15T21:57:20.188450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190331+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:57:20.190331+0000","last_peered":"2024-09-15T21:57:20.190331+0000","last_clean":"2024-09-15T21:57:20.190331+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:57:20.190331+0000","last_undegraded":"2024-09-15T21:57:20.190331+0000","last_fullsized":"2024-09-15T21:57:20.190331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467936+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:57:20.467936+0000","last_peered":"2024-09-15T21:57:20.467936+0000","last_clean":"2024-09-15T21:57:20.467936+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:57:20.467936+0000","last_undegraded":"2024-09-15T21:57:20.467936+0000","last_fullsized":"2024-09-15T21:57:20.467936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461005+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:57:20.461005+0000","last_peered":"2024-09-15T21:57:20.461005+0000","last_clean":"2024-09-15T21:57:20.461005+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:57:20.461005+0000","last_undegraded":"2024-09-15T21:57:20.461005+0000","last_fullsized":"2024-09-15T21:57:20.461005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275613+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:57:20.275613+0000","last_peered":"2024-09-15T21:57:20.275613+0000","last_clean":"2024-09-15T21:57:20.275613+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:57:20.275613+0000","last_undegraded":"2024-09-15T21:57:20.275613+0000","last_fullsized":"2024-09-15T21:57:20.275613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463309+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:57:20.463309+0000","last_peered":"2024-09-15T21:57:20.463309+0000","last_clean":"2024-09-15T21:57:20.463309+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:57:20.463309+0000","last_undegraded":"2024-09-15T21:57:20.463309+0000","last_fullsized":"2024-09-15T21:57:20.463309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'8","reported_seq":254,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712100+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:57:20.712100+0000","last_peered":"2024-09-15T21:57:20.712100+0000","last_clean":"2024-09-15T21:57:20.712100+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:57:20.712100+0000","last_undegraded":"2024-09-15T21:57:20.712100+0000","last_fullsized":"2024-09-15T21:57:20.712100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":206,"num_read_kb":206,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278726+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:57:20.278726+0000","last_peered":"2024-09-15T21:57:20.278726+0000","last_clean":"2024-09-15T21:57:20.278726+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:57:20.278726+0000","last_undegraded":"2024-09-15T21:57:20.278726+0000","last_fullsized":"2024-09-15T21:57:20.278726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467332+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:57:20.467332+0000","last_peered":"2024-09-15T21:57:20.467332+0000","last_clean":"2024-09-15T21:57:20.467332+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:57:20.467332+0000","last_undegraded":"2024-09-15T21:57:20.467332+0000","last_fullsized":"2024-09-15T21:57:20.467332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275825+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:57:20.275825+0000","last_peered":"2024-09-15T21:57:20.275825+0000","last_clean":"2024-09-15T21:57:20.275825+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:57:20.275825+0000","last_undegraded":"2024-09-15T21:57:20.275825+0000","last_fullsized":"2024-09-15T21:57:20.275825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187675+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:57:20.187675+0000","last_peered":"2024-09-15T21:57:20.187675+0000","last_clean":"2024-09-15T21:57:20.187675+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:57:20.187675+0000","last_undegraded":"2024-09-15T21:57:20.187675+0000","last_fullsized":"2024-09-15T21:57:20.187675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472373+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:57:20.472373+0000","last_peered":"2024-09-15T21:57:20.472373+0000","last_clean":"2024-09-15T21:57:20.472373+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:57:20.472373+0000","last_undegraded":"2024-09-15T21:57:20.472373+0000","last_fullsized":"2024-09-15T21:57:20.472373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278372+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:57:20.278372+0000","last_peered":"2024-09-15T21:57:20.278372+0000","last_clean":"2024-09-15T21:57:20.278372+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:57:20.278372+0000","last_undegraded":"2024-09-15T21:57:20.278372+0000","last_fullsized":"2024-09-15T21:57:20.278372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185595+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:57:20.185595+0000","last_peered":"2024-09-15T21:57:20.185595+0000","last_clean":"2024-09-15T21:57:20.185595+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:57:20.185595+0000","last_undegraded":"2024-09-15T21:57:20.185595+0000","last_fullsized":"2024-09-15T21:57:20.185595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285572+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:57:20.285572+0000","last_peered":"2024-09-15T21:57:20.285572+0000","last_clean":"2024-09-15T21:57:20.285572+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:57:20.285572+0000","last_undegraded":"2024-09-15T21:57:20.285572+0000","last_fullsized":"2024-09-15T21:57:20.285572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187909+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:57:20.187909+0000","last_peered":"2024-09-15T21:57:20.187909+0000","last_clean":"2024-09-15T21:57:20.187909+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:57:20.187909+0000","last_undegraded":"2024-09-15T21:57:20.187909+0000","last_fullsized":"2024-09-15T21:57:20.187909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396115+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:57:20.396115+0000","last_peered":"2024-09-15T21:57:20.396115+0000","last_clean":"2024-09-15T21:57:20.396115+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:57:20.396115+0000","last_undegraded":"2024-09-15T21:57:20.396115+0000","last_fullsized":"2024-09-15T21:57:20.396115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'9","reported_seq":158,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712626+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:57:20.712626+0000","last_peered":"2024-09-15T21:57:20.712626+0000","last_clean":"2024-09-15T21:57:20.712626+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:57:20.712626+0000","last_undegraded":"2024-09-15T21:57:20.712626+0000","last_fullsized":"2024-09-15T21:57:20.712626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1857,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":109,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278171+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:57:20.278171+0000","last_peered":"2024-09-15T21:57:20.278171+0000","last_clean":"2024-09-15T21:57:20.278171+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:57:20.278171+0000","last_undegraded":"2024-09-15T21:57:20.278171+0000","last_fullsized":"2024-09-15T21:57:20.278171+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396197+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:57:20.396197+0000","last_peered":"2024-09-15T21:57:20.396197+0000","last_clean":"2024-09-15T21:57:20.396197+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:57:20.396197+0000","last_undegraded":"2024-09-15T21:57:20.396197+0000","last_fullsized":"2024-09-15T21:57:20.396197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471807+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:57:20.471807+0000","last_peered":"2024-09-15T21:57:20.471807+0000","last_clean":"2024-09-15T21:57:20.471807+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:57:20.471807+0000","last_undegraded":"2024-09-15T21:57:20.471807+0000","last_fullsized":"2024-09-15T21:57:20.471807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396260+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:57:20.396260+0000","last_peered":"2024-09-15T21:57:20.396260+0000","last_clean":"2024-09-15T21:57:20.396260+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:57:20.396260+0000","last_undegraded":"2024-09-15T21:57:20.396260+0000","last_fullsized":"2024-09-15T21:57:20.396260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395966+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:57:20.395966+0000","last_peered":"2024-09-15T21:57:20.395966+0000","last_clean":"2024-09-15T21:57:20.395966+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:57:20.395966+0000","last_undegraded":"2024-09-15T21:57:20.395966+0000","last_fullsized":"2024-09-15T21:57:20.395966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466952+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:57:20.466952+0000","last_peered":"2024-09-15T21:57:20.466952+0000","last_clean":"2024-09-15T21:57:20.466952+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:57:20.466952+0000","last_undegraded":"2024-09-15T21:57:20.466952+0000","last_fullsized":"2024-09-15T21:57:20.466952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186090+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:57:20.186090+0000","last_peered":"2024-09-15T21:57:20.186090+0000","last_clean":"2024-09-15T21:57:20.186090+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:57:20.186090+0000","last_undegraded":"2024-09-15T21:57:20.186090+0000","last_fullsized":"2024-09-15T21:57:20.186090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472240+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:57:20.472240+0000","last_peered":"2024-09-15T21:57:20.472240+0000","last_clean":"2024-09-15T21:57:20.472240+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:57:20.472240+0000","last_undegraded":"2024-09-15T21:57:20.472240+0000","last_fullsized":"2024-09-15T21:57:20.472240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396338+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:57:20.396338+0000","last_peered":"2024-09-15T21:57:20.396338+0000","last_clean":"2024-09-15T21:57:20.396338+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:57:20.396338+0000","last_undegraded":"2024-09-15T21:57:20.396338+0000","last_fullsized":"2024-09-15T21:57:20.396338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280054+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:57:20.280054+0000","last_peered":"2024-09-15T21:57:20.280054+0000","last_clean":"2024-09-15T21:57:20.280054+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:57:20.280054+0000","last_undegraded":"2024-09-15T21:57:20.280054+0000","last_fullsized":"2024-09-15T21:57:20.280054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187475+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:57:20.187475+0000","last_peered":"2024-09-15T21:57:20.187475+0000","last_clean":"2024-09-15T21:57:20.187475+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:57:20.187475+0000","last_undegraded":"2024-09-15T21:57:20.187475+0000","last_fullsized":"2024-09-15T21:57:20.187475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278973+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:57:20.278973+0000","last_peered":"2024-09-15T21:57:20.278973+0000","last_clean":"2024-09-15T21:57:20.278973+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:57:20.278973+0000","last_undegraded":"2024-09-15T21:57:20.278973+0000","last_fullsized":"2024-09-15T21:57:20.278973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463016+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:57:20.463016+0000","last_peered":"2024-09-15T21:57:20.463016+0000","last_clean":"2024-09-15T21:57:20.463016+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:57:20.463016+0000","last_undegraded":"2024-09-15T21:57:20.463016+0000","last_fullsized":"2024-09-15T21:57:20.463016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186044+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:57:20.186044+0000","last_peered":"2024-09-15T21:57:20.186044+0000","last_clean":"2024-09-15T21:57:20.186044+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:57:20.186044+0000","last_undegraded":"2024-09-15T21:57:20.186044+0000","last_fullsized":"2024-09-15T21:57:20.186044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472203+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:57:20.472203+0000","last_peered":"2024-09-15T21:57:20.472203+0000","last_clean":"2024-09-15T21:57:20.472203+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:57:20.472203+0000","last_undegraded":"2024-09-15T21:57:20.472203+0000","last_fullsized":"2024-09-15T21:57:20.472203+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396319+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:57:20.396319+0000","last_peered":"2024-09-15T21:57:20.396319+0000","last_clean":"2024-09-15T21:57:20.396319+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:57:20.396319+0000","last_undegraded":"2024-09-15T21:57:20.396319+0000","last_fullsized":"2024-09-15T21:57:20.396319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276457+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:57:20.276457+0000","last_peered":"2024-09-15T21:57:20.276457+0000","last_clean":"2024-09-15T21:57:20.276457+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:57:20.276457+0000","last_undegraded":"2024-09-15T21:57:20.276457+0000","last_fullsized":"2024-09-15T21:57:20.276457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472556+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:57:20.472556+0000","last_peered":"2024-09-15T21:57:20.472556+0000","last_clean":"2024-09-15T21:57:20.472556+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:57:20.472556+0000","last_undegraded":"2024-09-15T21:57:20.472556+0000","last_fullsized":"2024-09-15T21:57:20.472556+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462857+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:57:20.462857+0000","last_peered":"2024-09-15T21:57:20.462857+0000","last_clean":"2024-09-15T21:57:20.462857+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:57:20.462857+0000","last_undegraded":"2024-09-15T21:57:20.462857+0000","last_fullsized":"2024-09-15T21:57:20.462857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466723+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:57:20.466723+0000","last_peered":"2024-09-15T21:57:20.466723+0000","last_clean":"2024-09-15T21:57:20.466723+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:57:20.466723+0000","last_undegraded":"2024-09-15T21:57:20.466723+0000","last_fullsized":"2024-09-15T21:57:20.466723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466886+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:57:20.466886+0000","last_peered":"2024-09-15T21:57:20.466886+0000","last_clean":"2024-09-15T21:57:20.466886+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:57:20.466886+0000","last_undegraded":"2024-09-15T21:57:20.466886+0000","last_fullsized":"2024-09-15T21:57:20.466886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186173+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:57:20.186173+0000","last_peered":"2024-09-15T21:57:20.186173+0000","last_clean":"2024-09-15T21:57:20.186173+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:57:20.186173+0000","last_undegraded":"2024-09-15T21:57:20.186173+0000","last_fullsized":"2024-09-15T21:57:20.186173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276015+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:57:20.276015+0000","last_peered":"2024-09-15T21:57:20.276015+0000","last_clean":"2024-09-15T21:57:20.276015+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:57:20.276015+0000","last_undegraded":"2024-09-15T21:57:20.276015+0000","last_fullsized":"2024-09-15T21:57:20.276015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461069+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:57:20.461069+0000","last_peered":"2024-09-15T21:57:20.461069+0000","last_clean":"2024-09-15T21:57:20.461069+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:57:20.461069+0000","last_undegraded":"2024-09-15T21:57:20.461069+0000","last_fullsized":"2024-09-15T21:57:20.461069+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.286073+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:57:20.286073+0000","last_peered":"2024-09-15T21:57:20.286073+0000","last_clean":"2024-09-15T21:57:20.286073+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:57:20.286073+0000","last_undegraded":"2024-09-15T21:57:20.286073+0000","last_fullsized":"2024-09-15T21:57:20.286073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463316+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:57:20.463316+0000","last_peered":"2024-09-15T21:57:20.463316+0000","last_clean":"2024-09-15T21:57:20.463316+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:57:20.463316+0000","last_undegraded":"2024-09-15T21:57:20.463316+0000","last_fullsized":"2024-09-15T21:57:20.463316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278186+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:57:20.278186+0000","last_peered":"2024-09-15T21:57:20.278186+0000","last_clean":"2024-09-15T21:57:20.278186+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:57:20.278186+0000","last_undegraded":"2024-09-15T21:57:20.278186+0000","last_fullsized":"2024-09-15T21:57:20.278186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460530+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:57:20.460530+0000","last_peered":"2024-09-15T21:57:20.460530+0000","last_clean":"2024-09-15T21:57:20.460530+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:57:20.460530+0000","last_undegraded":"2024-09-15T21:57:20.460530+0000","last_fullsized":"2024-09-15T21:57:20.460530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285510+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:57:20.285510+0000","last_peered":"2024-09-15T21:57:20.285510+0000","last_clean":"2024-09-15T21:57:20.285510+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:57:20.285510+0000","last_undegraded":"2024-09-15T21:57:20.285510+0000","last_fullsized":"2024-09-15T21:57:20.285510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278179+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:57:20.278179+0000","last_peered":"2024-09-15T21:57:20.278179+0000","last_clean":"2024-09-15T21:57:20.278179+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:57:20.278179+0000","last_undegraded":"2024-09-15T21:57:20.278179+0000","last_fullsized":"2024-09-15T21:57:20.278179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190742+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:57:20.190742+0000","last_peered":"2024-09-15T21:57:20.190742+0000","last_clean":"2024-09-15T21:57:20.190742+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:57:20.190742+0000","last_undegraded":"2024-09-15T21:57:20.190742+0000","last_fullsized":"2024-09-15T21:57:20.190742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467111+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:57:20.467111+0000","last_peered":"2024-09-15T21:57:20.467111+0000","last_clean":"2024-09-15T21:57:20.467111+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:57:20.467111+0000","last_undegraded":"2024-09-15T21:57:20.467111+0000","last_fullsized":"2024-09-15T21:57:20.467111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467430+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:57:20.467430+0000","last_peered":"2024-09-15T21:57:20.467430+0000","last_clean":"2024-09-15T21:57:20.467430+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:57:20.467430+0000","last_undegraded":"2024-09-15T21:57:20.467430+0000","last_fullsized":"2024-09-15T21:57:20.467430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472483+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:57:20.472483+0000","last_peered":"2024-09-15T21:57:20.472483+0000","last_clean":"2024-09-15T21:57:20.472483+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:57:20.472483+0000","last_undegraded":"2024-09-15T21:57:20.472483+0000","last_fullsized":"2024-09-15T21:57:20.472483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":603167,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1172,"num_read_kb":704,"num_write":389,"num_write_kb":1477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1809501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514538,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1081344,"data_stored":536154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776174,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28900,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716252,"statfs":{"total":95995035648,"available":95965442048,"internally_reserved":0,"allocated":1921024,"data_stored":1330627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939701,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36804,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708348,"statfs":{"total":95995035648,"available":95957348352,"internally_reserved":0,"allocated":1490944,"data_stored":923695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168633,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1486848,"data_stored":927587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397567,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36988,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708164,"statfs":{"total":95995035648,"available":95957159936,"internally_reserved":0,"allocated":1679360,"data_stored":1122064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561090,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":530472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28820,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716332,"statfs":{"total":95995035648,"available":95965523968,"internally_reserved":0,"allocated":1839104,"data_stored":1259476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":534939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28784,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716368,"statfs":{"total":95995035648,"available":95965560832,"internally_reserved":0,"allocated":1617920,"data_stored":1054603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":590670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1150976,"data_stored":600090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":532462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:57:25.425 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-09-15T21:57:25.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:57:25.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T21:57:25.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Unable to set osd_memory_target on smithi174 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T21:57:25.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi203 to 176.0M 2024-09-15T21:57:25.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Unable to set osd_memory_target on smithi203 to 184563097: error parsing value: Value '184563097' is below minimum 939524096 2024-09-15T21:57:25.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Unable to set osd_memory_target on smithi028 to 185689190: error parsing value: Value '185689190' is below minimum 939524096 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1035486229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:25.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:25 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T21:57:25.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Unable to set osd_memory_target on smithi174 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T21:57:25.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi203 to 176.0M 2024-09-15T21:57:25.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Unable to set osd_memory_target on smithi203 to 184563097: error parsing value: Value '184563097' is below minimum 939524096 2024-09-15T21:57:25.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Unable to set osd_memory_target on smithi028 to 185689190: error parsing value: Value '185689190' is below minimum 939524096 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi028:/etc/ceph/ceph.conf 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.511 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: Updating smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1035486229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.512 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:25.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:25.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:25.513 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:25 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:25.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 -- 172.21.15.28:0/3585453526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c1035b0 msgr2=0x7f932c10ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 --2- 172.21.15.28:0/3585453526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c1035b0 0x7f932c10ba90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9320009900 tx=0x7f932002a480 comp rx=0 tx=0).stop 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 -- 172.21.15.28:0/3585453526 shutdown_connections 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 --2- 172.21.15.28:0/3585453526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f932c10c0e0 0x7f932c10e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 --2- 172.21.15.28:0/3585453526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c1035b0 0x7f932c10ba90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 --2- 172.21.15.28:0/3585453526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f932c102c90 0x7f932c103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 -- 172.21.15.28:0/3585453526 >> 172.21.15.28:0/3585453526 conn(0x7f932c0fc6a0 msgr2=0x7f932c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 -- 172.21.15.28:0/3585453526 shutdown_connections 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 -- 172.21.15.28:0/3585453526 wait complete. 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 Processor -- start 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.561+0000 7f9332835640 1 -- start start 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9332835640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c102c90 0x7f932c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9332835640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f932c1035b0 0x7f932c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9332835640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f932c10c0e0 0x7f932c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9332835640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f932c111ec0 con 0x7f932c102c90 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9332835640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f932c111d40 con 0x7f932c10c0e0 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9332835640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f932c112040 con 0x7f932c1035b0 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c102c90 0x7f932c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c102c90 0x7f932c1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59752/0 (socket says 172.21.15.28:59752) 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 -- 172.21.15.28:0/3567580401 learned_addr learned my addr 172.21.15.28:0/3567580401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932b7fe640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f932c1035b0 0x7f932c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9330dd2640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f932c10c0e0 0x7f932c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 -- 172.21.15.28:0/3567580401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f932c1035b0 msgr2=0x7f932c198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f932c1035b0 0x7f932c198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 -- 172.21.15.28:0/3567580401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f932c10c0e0 msgr2=0x7f932c19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f932c10c0e0 0x7f932c19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 -- 172.21.15.28:0/3567580401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c19d3e0 con 0x7f932c102c90 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932b7fe640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f932c1035b0 0x7f932c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f932bfff640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c102c90 0x7f932c1983f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e9f0 tx=0x7f931c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f93297fa640 1 -- 172.21.15.28:0/3567580401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c00cd80 con 0x7f932c102c90 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f93297fa640 1 -- 172.21.15.28:0/3567580401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f931c010910 con 0x7f932c102c90 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.562+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932c19d6d0 con 0x7f932c102c90 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.563+0000 7f93297fa640 1 -- 172.21.15.28:0/3567580401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c018990 con 0x7f932c102c90 2024-09-15T21:57:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.563+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932c1a4f60 con 0x7f932c102c90 2024-09-15T21:57:25.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.564+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f932c103070 con 0x7f932c102c90 2024-09-15T21:57:25.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.565+0000 7f93297fa640 1 -- 172.21.15.28:0/3567580401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f931c010ac0 con 0x7f932c102c90 2024-09-15T21:57:25.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.565+0000 7f93297fa640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93140a34e0 0x7f93140a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:25.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.565+0000 7f932b7fe640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93140a34e0 0x7f93140a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.565+0000 7f93297fa640 1 -- 172.21.15.28:0/3567580401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f931c0d0780 con 0x7f932c102c90 2024-09-15T21:57:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.566+0000 7f932b7fe640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93140a34e0 0x7f93140a59a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f932c0fecc0 tx=0x7f9320005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.569+0000 7f93297fa640 1 -- 172.21.15.28:0/3567580401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f931c09cbb0 con 0x7f932c102c90 2024-09-15T21:57:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.742+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f932c0fe490 con 0x7f93140a34e0 2024-09-15T21:57:25.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.750+0000 7f93297fa640 1 -- 172.21.15.28:0/3567580401 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339443 (secure 0 0 0) 0x7f932c0fe490 con 0x7f93140a34e0 2024-09-15T21:57:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:57:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93140a34e0 msgr2=0x7f93140a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93140a34e0 0x7f93140a59a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f932c0fecc0 tx=0x7f9320005a20 comp rx=0 tx=0).stop 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c102c90 msgr2=0x7f932c1983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c102c90 0x7f932c1983f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e9f0 tx=0x7f931c00eeb0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 shutdown_connections 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93140a34e0 0x7f93140a59a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f932c10c0e0 0x7f932c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f932c1035b0 0x7f932c198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 --2- 172.21.15.28:0/3567580401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c102c90 0x7f932c1983f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 >> 172.21.15.28:0/3567580401 conn(0x7f932c0fc6a0 msgr2=0x7f932c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 shutdown_connections 2024-09-15T21:57:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:25.754+0000 7f9332835640 1 -- 172.21.15.28:0/3567580401 wait complete. 2024-09-15T21:57:25.768 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-09-15T21:57:24.202295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193769,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1308,"num_read_kb":823,"num_write":572,"num_write_kb":4417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":952,"ondisk_log_size":952,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365756,"kb_used_data":16296,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576068,"statfs":{"total":1151940427776,"available":1151565893632,"internally_reserved":0,"allocated":16687104,"data_stored":9942839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.993652"},"pg_stats":[{"pgid":"2.1f","version":"93'14","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.469302+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:57:20.469302+0000","last_peered":"2024-09-15T21:57:20.469302+0000","last_clean":"2024-09-15T21:57:20.469302+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:57:20.469302+0000","last_undegraded":"2024-09-15T21:57:20.469302+0000","last_fullsized":"2024-09-15T21:57:20.469302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280359+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:57:20.280359+0000","last_peered":"2024-09-15T21:57:20.280359+0000","last_clean":"2024-09-15T21:57:20.280359+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:57:20.280359+0000","last_undegraded":"2024-09-15T21:57:20.280359+0000","last_fullsized":"2024-09-15T21:57:20.280359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395705+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:57:20.395705+0000","last_peered":"2024-09-15T21:57:20.395705+0000","last_clean":"2024-09-15T21:57:20.395705+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:57:20.395705+0000","last_undegraded":"2024-09-15T21:57:20.395705+0000","last_fullsized":"2024-09-15T21:57:20.395705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280756+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:57:20.280756+0000","last_peered":"2024-09-15T21:57:20.280756+0000","last_clean":"2024-09-15T21:57:20.280756+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:57:20.280756+0000","last_undegraded":"2024-09-15T21:57:20.280756+0000","last_fullsized":"2024-09-15T21:57:20.280756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467940+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:57:20.467940+0000","last_peered":"2024-09-15T21:57:20.467940+0000","last_clean":"2024-09-15T21:57:20.467940+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:57:20.467940+0000","last_undegraded":"2024-09-15T21:57:20.467940+0000","last_fullsized":"2024-09-15T21:57:20.467940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190047+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:57:20.190047+0000","last_peered":"2024-09-15T21:57:20.190047+0000","last_clean":"2024-09-15T21:57:20.190047+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:57:20.190047+0000","last_undegraded":"2024-09-15T21:57:20.190047+0000","last_fullsized":"2024-09-15T21:57:20.190047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278626+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:57:20.278626+0000","last_peered":"2024-09-15T21:57:20.278626+0000","last_clean":"2024-09-15T21:57:20.278626+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:57:20.278626+0000","last_undegraded":"2024-09-15T21:57:20.278626+0000","last_fullsized":"2024-09-15T21:57:20.278626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463402+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:57:20.463402+0000","last_peered":"2024-09-15T21:57:20.463402+0000","last_clean":"2024-09-15T21:57:20.463402+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:57:20.463402+0000","last_undegraded":"2024-09-15T21:57:20.463402+0000","last_fullsized":"2024-09-15T21:57:20.463402+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190387+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:57:20.190387+0000","last_peered":"2024-09-15T21:57:20.190387+0000","last_clean":"2024-09-15T21:57:20.190387+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:57:20.190387+0000","last_undegraded":"2024-09-15T21:57:20.190387+0000","last_fullsized":"2024-09-15T21:57:20.190387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472029+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:57:20.472029+0000","last_peered":"2024-09-15T21:57:20.472029+0000","last_clean":"2024-09-15T21:57:20.472029+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:57:20.472029+0000","last_undegraded":"2024-09-15T21:57:20.472029+0000","last_fullsized":"2024-09-15T21:57:20.472029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188165+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:57:20.188165+0000","last_peered":"2024-09-15T21:57:20.188165+0000","last_clean":"2024-09-15T21:57:20.188165+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:57:20.188165+0000","last_undegraded":"2024-09-15T21:57:20.188165+0000","last_fullsized":"2024-09-15T21:57:20.188165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190320+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:57:20.190320+0000","last_peered":"2024-09-15T21:57:20.190320+0000","last_clean":"2024-09-15T21:57:20.190320+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:57:20.190320+0000","last_undegraded":"2024-09-15T21:57:20.190320+0000","last_fullsized":"2024-09-15T21:57:20.190320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460655+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:57:20.460655+0000","last_peered":"2024-09-15T21:57:20.460655+0000","last_clean":"2024-09-15T21:57:20.460655+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:57:20.460655+0000","last_undegraded":"2024-09-15T21:57:20.460655+0000","last_fullsized":"2024-09-15T21:57:20.460655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278262+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:57:20.278262+0000","last_peered":"2024-09-15T21:57:20.278262+0000","last_clean":"2024-09-15T21:57:20.278262+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:57:20.278262+0000","last_undegraded":"2024-09-15T21:57:20.278262+0000","last_fullsized":"2024-09-15T21:57:20.278262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190739+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:57:20.190739+0000","last_peered":"2024-09-15T21:57:20.190739+0000","last_clean":"2024-09-15T21:57:20.190739+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:57:20.190739+0000","last_undegraded":"2024-09-15T21:57:20.190739+0000","last_fullsized":"2024-09-15T21:57:20.190739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279728+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:57:20.279728+0000","last_peered":"2024-09-15T21:57:20.279728+0000","last_clean":"2024-09-15T21:57:20.279728+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:57:20.279728+0000","last_undegraded":"2024-09-15T21:57:20.279728+0000","last_fullsized":"2024-09-15T21:57:20.279728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"92'4","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471831+0000","last_change":"2024-09-15T21:56:58.926427+0000","last_active":"2024-09-15T21:57:20.471831+0000","last_peered":"2024-09-15T21:57:20.471831+0000","last_clean":"2024-09-15T21:57:20.471831+0000","last_became_active":"2024-09-15T21:56:58.922755+0000","last_became_peered":"2024-09-15T21:56:58.922755+0000","last_unstale":"2024-09-15T21:57:20.471831+0000","last_undegraded":"2024-09-15T21:57:20.471831+0000","last_fullsized":"2024-09-15T21:57:20.471831+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:22.033354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:57:20.190660+0000","last_peered":"2024-09-15T21:57:20.190660+0000","last_clean":"2024-09-15T21:57:20.190660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:57:20.190660+0000","last_undegraded":"2024-09-15T21:57:20.190660+0000","last_fullsized":"2024-09-15T21:57:20.190660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471869+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:57:20.471869+0000","last_peered":"2024-09-15T21:57:20.471869+0000","last_clean":"2024-09-15T21:57:20.471869+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:57:20.471869+0000","last_undegraded":"2024-09-15T21:57:20.471869+0000","last_fullsized":"2024-09-15T21:57:20.471869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279173+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:57:20.279173+0000","last_peered":"2024-09-15T21:57:20.279173+0000","last_clean":"2024-09-15T21:57:20.279173+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:57:20.279173+0000","last_undegraded":"2024-09-15T21:57:20.279173+0000","last_fullsized":"2024-09-15T21:57:20.279173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276125+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:57:20.276125+0000","last_peered":"2024-09-15T21:57:20.276125+0000","last_clean":"2024-09-15T21:57:20.276125+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:57:20.276125+0000","last_undegraded":"2024-09-15T21:57:20.276125+0000","last_fullsized":"2024-09-15T21:57:20.276125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278075+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:57:20.278075+0000","last_peered":"2024-09-15T21:57:20.278075+0000","last_clean":"2024-09-15T21:57:20.278075+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:57:20.278075+0000","last_undegraded":"2024-09-15T21:57:20.278075+0000","last_fullsized":"2024-09-15T21:57:20.278075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462853+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:57:20.462853+0000","last_peered":"2024-09-15T21:57:20.462853+0000","last_clean":"2024-09-15T21:57:20.462853+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:57:20.462853+0000","last_undegraded":"2024-09-15T21:57:20.462853+0000","last_fullsized":"2024-09-15T21:57:20.462853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190763+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:57:20.190763+0000","last_peered":"2024-09-15T21:57:20.190763+0000","last_clean":"2024-09-15T21:57:20.190763+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:57:20.190763+0000","last_undegraded":"2024-09-15T21:57:20.190763+0000","last_fullsized":"2024-09-15T21:57:20.190763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467562+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:57:20.467562+0000","last_peered":"2024-09-15T21:57:20.467562+0000","last_clean":"2024-09-15T21:57:20.467562+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:57:20.467562+0000","last_undegraded":"2024-09-15T21:57:20.467562+0000","last_fullsized":"2024-09-15T21:57:20.467562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190932+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:57:20.190932+0000","last_peered":"2024-09-15T21:57:20.190932+0000","last_clean":"2024-09-15T21:57:20.190932+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:57:20.190932+0000","last_undegraded":"2024-09-15T21:57:20.190932+0000","last_fullsized":"2024-09-15T21:57:20.190932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395988+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:57:20.395988+0000","last_peered":"2024-09-15T21:57:20.395988+0000","last_clean":"2024-09-15T21:57:20.395988+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:57:20.395988+0000","last_undegraded":"2024-09-15T21:57:20.395988+0000","last_fullsized":"2024-09-15T21:57:20.395988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280166+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:57:20.280166+0000","last_peered":"2024-09-15T21:57:20.280166+0000","last_clean":"2024-09-15T21:57:20.280166+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:57:20.280166+0000","last_undegraded":"2024-09-15T21:57:20.280166+0000","last_fullsized":"2024-09-15T21:57:20.280166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'53","reported_seq":117,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.416465+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:57:20.416465+0000","last_peered":"2024-09-15T21:57:20.416465+0000","last_clean":"2024-09-15T21:57:20.416465+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:57:20.416465+0000","last_undegraded":"2024-09-15T21:57:20.416465+0000","last_fullsized":"2024-09-15T21:57:20.416465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":94,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.277996+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:57:20.277996+0000","last_peered":"2024-09-15T21:57:20.277996+0000","last_clean":"2024-09-15T21:57:20.277996+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:57:20.277996+0000","last_undegraded":"2024-09-15T21:57:20.277996+0000","last_fullsized":"2024-09-15T21:57:20.277996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186161+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:57:20.186161+0000","last_peered":"2024-09-15T21:57:20.186161+0000","last_clean":"2024-09-15T21:57:20.186161+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:57:20.186161+0000","last_undegraded":"2024-09-15T21:57:20.186161+0000","last_fullsized":"2024-09-15T21:57:20.186161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278630+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:57:20.278630+0000","last_peered":"2024-09-15T21:57:20.278630+0000","last_clean":"2024-09-15T21:57:20.278630+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:57:20.278630+0000","last_undegraded":"2024-09-15T21:57:20.278630+0000","last_fullsized":"2024-09-15T21:57:20.278630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472019+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:57:20.472019+0000","last_peered":"2024-09-15T21:57:20.472019+0000","last_clean":"2024-09-15T21:57:20.472019+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:57:20.472019+0000","last_undegraded":"2024-09-15T21:57:20.472019+0000","last_fullsized":"2024-09-15T21:57:20.472019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285785+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:57:20.285785+0000","last_peered":"2024-09-15T21:57:20.285785+0000","last_clean":"2024-09-15T21:57:20.285785+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:57:20.285785+0000","last_undegraded":"2024-09-15T21:57:20.285785+0000","last_fullsized":"2024-09-15T21:57:20.285785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276329+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:57:20.276329+0000","last_peered":"2024-09-15T21:57:20.276329+0000","last_clean":"2024-09-15T21:57:20.276329+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:57:20.276329+0000","last_undegraded":"2024-09-15T21:57:20.276329+0000","last_fullsized":"2024-09-15T21:57:20.276329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278087+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:57:20.278087+0000","last_peered":"2024-09-15T21:57:20.278087+0000","last_clean":"2024-09-15T21:57:20.278087+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:57:20.278087+0000","last_undegraded":"2024-09-15T21:57:20.278087+0000","last_fullsized":"2024-09-15T21:57:20.278087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285570+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:57:20.285570+0000","last_peered":"2024-09-15T21:57:20.285570+0000","last_clean":"2024-09-15T21:57:20.285570+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:57:20.285570+0000","last_undegraded":"2024-09-15T21:57:20.285570+0000","last_fullsized":"2024-09-15T21:57:20.285570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.711889+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:57:20.711889+0000","last_peered":"2024-09-15T21:57:20.711889+0000","last_clean":"2024-09-15T21:57:20.711889+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:57:20.711889+0000","last_undegraded":"2024-09-15T21:57:20.711889+0000","last_fullsized":"2024-09-15T21:57:20.711889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185976+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:57:20.185976+0000","last_peered":"2024-09-15T21:57:20.185976+0000","last_clean":"2024-09-15T21:57:20.185976+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:57:20.185976+0000","last_undegraded":"2024-09-15T21:57:20.185976+0000","last_fullsized":"2024-09-15T21:57:20.185976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187498+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:57:20.187498+0000","last_peered":"2024-09-15T21:57:20.187498+0000","last_clean":"2024-09-15T21:57:20.187498+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:57:20.187498+0000","last_undegraded":"2024-09-15T21:57:20.187498+0000","last_fullsized":"2024-09-15T21:57:20.187498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":151,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466327+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:57:20.466327+0000","last_peered":"2024-09-15T21:57:20.466327+0000","last_clean":"2024-09-15T21:57:20.466327+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:57:20.466327+0000","last_undegraded":"2024-09-15T21:57:20.466327+0000","last_fullsized":"2024-09-15T21:57:20.466327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186270+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:57:20.186270+0000","last_peered":"2024-09-15T21:57:20.186270+0000","last_clean":"2024-09-15T21:57:20.186270+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:57:20.186270+0000","last_undegraded":"2024-09-15T21:57:20.186270+0000","last_fullsized":"2024-09-15T21:57:20.186270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185590+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:57:20.185590+0000","last_peered":"2024-09-15T21:57:20.185590+0000","last_clean":"2024-09-15T21:57:20.185590+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:57:20.185590+0000","last_undegraded":"2024-09-15T21:57:20.185590+0000","last_fullsized":"2024-09-15T21:57:20.185590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460124+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:57:20.460124+0000","last_peered":"2024-09-15T21:57:20.460124+0000","last_clean":"2024-09-15T21:57:20.460124+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:57:20.460124+0000","last_undegraded":"2024-09-15T21:57:20.460124+0000","last_fullsized":"2024-09-15T21:57:20.460124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188088+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:57:20.188088+0000","last_peered":"2024-09-15T21:57:20.188088+0000","last_clean":"2024-09-15T21:57:20.188088+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:57:20.188088+0000","last_undegraded":"2024-09-15T21:57:20.188088+0000","last_fullsized":"2024-09-15T21:57:20.188088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279569+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:57:20.279569+0000","last_peered":"2024-09-15T21:57:20.279569+0000","last_clean":"2024-09-15T21:57:20.279569+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:57:20.279569+0000","last_undegraded":"2024-09-15T21:57:20.279569+0000","last_fullsized":"2024-09-15T21:57:20.279569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190561+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:57:20.190561+0000","last_peered":"2024-09-15T21:57:20.190561+0000","last_clean":"2024-09-15T21:57:20.190561+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:57:20.190561+0000","last_undegraded":"2024-09-15T21:57:20.190561+0000","last_fullsized":"2024-09-15T21:57:20.190561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460734+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:57:20.460734+0000","last_peered":"2024-09-15T21:57:20.460734+0000","last_clean":"2024-09-15T21:57:20.460734+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:57:20.460734+0000","last_undegraded":"2024-09-15T21:57:20.460734+0000","last_fullsized":"2024-09-15T21:57:20.460734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460868+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:57:20.460868+0000","last_peered":"2024-09-15T21:57:20.460868+0000","last_clean":"2024-09-15T21:57:20.460868+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:57:20.460868+0000","last_undegraded":"2024-09-15T21:57:20.460868+0000","last_fullsized":"2024-09-15T21:57:20.460868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461076+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:57:20.461076+0000","last_peered":"2024-09-15T21:57:20.461076+0000","last_clean":"2024-09-15T21:57:20.461076+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:57:20.461076+0000","last_undegraded":"2024-09-15T21:57:20.461076+0000","last_fullsized":"2024-09-15T21:57:20.461076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":157,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.719909+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:57:20.719909+0000","last_peered":"2024-09-15T21:57:20.719909+0000","last_clean":"2024-09-15T21:57:20.719909+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:57:20.719909+0000","last_undegraded":"2024-09-15T21:57:20.719909+0000","last_fullsized":"2024-09-15T21:57:20.719909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462918+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:57:20.462918+0000","last_peered":"2024-09-15T21:57:20.462918+0000","last_clean":"2024-09-15T21:57:20.462918+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:57:20.462918+0000","last_undegraded":"2024-09-15T21:57:20.462918+0000","last_fullsized":"2024-09-15T21:57:20.462918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460824+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:57:20.460824+0000","last_peered":"2024-09-15T21:57:20.460824+0000","last_clean":"2024-09-15T21:57:20.460824+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:57:20.460824+0000","last_undegraded":"2024-09-15T21:57:20.460824+0000","last_fullsized":"2024-09-15T21:57:20.460824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'41","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.413534+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:57:20.413534+0000","last_peered":"2024-09-15T21:57:20.413534+0000","last_clean":"2024-09-15T21:57:20.413534+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:57:20.413534+0000","last_undegraded":"2024-09-15T21:57:20.413534+0000","last_fullsized":"2024-09-15T21:57:20.413534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466957+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:57:20.466957+0000","last_peered":"2024-09-15T21:57:20.466957+0000","last_clean":"2024-09-15T21:57:20.466957+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:57:20.466957+0000","last_undegraded":"2024-09-15T21:57:20.466957+0000","last_fullsized":"2024-09-15T21:57:20.466957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460987+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:57:20.460987+0000","last_peered":"2024-09-15T21:57:20.460987+0000","last_clean":"2024-09-15T21:57:20.460987+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:57:20.460987+0000","last_undegraded":"2024-09-15T21:57:20.460987+0000","last_fullsized":"2024-09-15T21:57:20.460987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186150+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:57:20.186150+0000","last_peered":"2024-09-15T21:57:20.186150+0000","last_clean":"2024-09-15T21:57:20.186150+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:57:20.186150+0000","last_undegraded":"2024-09-15T21:57:20.186150+0000","last_fullsized":"2024-09-15T21:57:20.186150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187377+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:57:20.187377+0000","last_peered":"2024-09-15T21:57:20.187377+0000","last_clean":"2024-09-15T21:57:20.187377+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:57:20.187377+0000","last_undegraded":"2024-09-15T21:57:20.187377+0000","last_fullsized":"2024-09-15T21:57:20.187377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472329+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:57:20.472329+0000","last_peered":"2024-09-15T21:57:20.472329+0000","last_clean":"2024-09-15T21:57:20.472329+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:57:20.472329+0000","last_undegraded":"2024-09-15T21:57:20.472329+0000","last_fullsized":"2024-09-15T21:57:20.472329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278620+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:57:20.278620+0000","last_peered":"2024-09-15T21:57:20.278620+0000","last_clean":"2024-09-15T21:57:20.278620+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:57:20.278620+0000","last_undegraded":"2024-09-15T21:57:20.278620+0000","last_fullsized":"2024-09-15T21:57:20.278620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187448+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:57:20.187448+0000","last_peered":"2024-09-15T21:57:20.187448+0000","last_clean":"2024-09-15T21:57:20.187448+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:57:20.187448+0000","last_undegraded":"2024-09-15T21:57:20.187448+0000","last_fullsized":"2024-09-15T21:57:20.187448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187552+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:57:20.187552+0000","last_peered":"2024-09-15T21:57:20.187552+0000","last_clean":"2024-09-15T21:57:20.187552+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:57:20.187552+0000","last_undegraded":"2024-09-15T21:57:20.187552+0000","last_fullsized":"2024-09-15T21:57:20.187552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278144+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:57:20.278144+0000","last_peered":"2024-09-15T21:57:20.278144+0000","last_clean":"2024-09-15T21:57:20.278144+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:57:20.278144+0000","last_undegraded":"2024-09-15T21:57:20.278144+0000","last_fullsized":"2024-09-15T21:57:20.278144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186058+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:57:20.186058+0000","last_peered":"2024-09-15T21:57:20.186058+0000","last_clean":"2024-09-15T21:57:20.186058+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:57:20.186058+0000","last_undegraded":"2024-09-15T21:57:20.186058+0000","last_fullsized":"2024-09-15T21:57:20.186058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460653+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:57:20.460653+0000","last_peered":"2024-09-15T21:57:20.460653+0000","last_clean":"2024-09-15T21:57:20.460653+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:57:20.460653+0000","last_undegraded":"2024-09-15T21:57:20.460653+0000","last_fullsized":"2024-09-15T21:57:20.460653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":154,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.720943+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:57:20.720943+0000","last_peered":"2024-09-15T21:57:20.720943+0000","last_clean":"2024-09-15T21:57:20.720943+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:57:20.720943+0000","last_undegraded":"2024-09-15T21:57:20.720943+0000","last_fullsized":"2024-09-15T21:57:20.720943+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276236+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:57:20.276236+0000","last_peered":"2024-09-15T21:57:20.276236+0000","last_clean":"2024-09-15T21:57:20.276236+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:57:20.276236+0000","last_undegraded":"2024-09-15T21:57:20.276236+0000","last_fullsized":"2024-09-15T21:57:20.276236+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460781+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:57:20.460781+0000","last_peered":"2024-09-15T21:57:20.460781+0000","last_clean":"2024-09-15T21:57:20.460781+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:57:20.460781+0000","last_undegraded":"2024-09-15T21:57:20.460781+0000","last_fullsized":"2024-09-15T21:57:20.460781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187760+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:57:20.187760+0000","last_peered":"2024-09-15T21:57:20.187760+0000","last_clean":"2024-09-15T21:57:20.187760+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:57:20.187760+0000","last_undegraded":"2024-09-15T21:57:20.187760+0000","last_fullsized":"2024-09-15T21:57:20.187760+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472044+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:57:20.472044+0000","last_peered":"2024-09-15T21:57:20.472044+0000","last_clean":"2024-09-15T21:57:20.472044+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:57:20.472044+0000","last_undegraded":"2024-09-15T21:57:20.472044+0000","last_fullsized":"2024-09-15T21:57:20.472044+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185779+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:57:20.185779+0000","last_peered":"2024-09-15T21:57:20.185779+0000","last_clean":"2024-09-15T21:57:20.185779+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:57:20.185779+0000","last_undegraded":"2024-09-15T21:57:20.185779+0000","last_fullsized":"2024-09-15T21:57:20.185779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395654+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:57:20.395654+0000","last_peered":"2024-09-15T21:57:20.395654+0000","last_clean":"2024-09-15T21:57:20.395654+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:57:20.395654+0000","last_undegraded":"2024-09-15T21:57:20.395654+0000","last_fullsized":"2024-09-15T21:57:20.395654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188062+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:57:20.188062+0000","last_peered":"2024-09-15T21:57:20.188062+0000","last_clean":"2024-09-15T21:57:20.188062+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:57:20.188062+0000","last_undegraded":"2024-09-15T21:57:20.188062+0000","last_fullsized":"2024-09-15T21:57:20.188062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278543+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:57:20.278543+0000","last_peered":"2024-09-15T21:57:20.278543+0000","last_clean":"2024-09-15T21:57:20.278543+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:57:20.278543+0000","last_undegraded":"2024-09-15T21:57:20.278543+0000","last_fullsized":"2024-09-15T21:57:20.278543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'42","reported_seq":301,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712883+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:57:20.712883+0000","last_peered":"2024-09-15T21:57:20.712883+0000","last_clean":"2024-09-15T21:57:20.712883+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:57:20.712883+0000","last_undegraded":"2024-09-15T21:57:20.712883+0000","last_fullsized":"2024-09-15T21:57:20.712883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68588,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":218,"num_read_kb":175,"num_write":44,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275841+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:57:20.275841+0000","last_peered":"2024-09-15T21:57:20.275841+0000","last_clean":"2024-09-15T21:57:20.275841+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:57:20.275841+0000","last_undegraded":"2024-09-15T21:57:20.275841+0000","last_fullsized":"2024-09-15T21:57:20.275841+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463470+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:57:20.463470+0000","last_peered":"2024-09-15T21:57:20.463470+0000","last_clean":"2024-09-15T21:57:20.463470+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:57:20.463470+0000","last_undegraded":"2024-09-15T21:57:20.463470+0000","last_fullsized":"2024-09-15T21:57:20.463470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188450+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:57:20.188450+0000","last_peered":"2024-09-15T21:57:20.188450+0000","last_clean":"2024-09-15T21:57:20.188450+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:57:20.188450+0000","last_undegraded":"2024-09-15T21:57:20.188450+0000","last_fullsized":"2024-09-15T21:57:20.188450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190331+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:57:20.190331+0000","last_peered":"2024-09-15T21:57:20.190331+0000","last_clean":"2024-09-15T21:57:20.190331+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:57:20.190331+0000","last_undegraded":"2024-09-15T21:57:20.190331+0000","last_fullsized":"2024-09-15T21:57:20.190331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467936+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:57:20.467936+0000","last_peered":"2024-09-15T21:57:20.467936+0000","last_clean":"2024-09-15T21:57:20.467936+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:57:20.467936+0000","last_undegraded":"2024-09-15T21:57:20.467936+0000","last_fullsized":"2024-09-15T21:57:20.467936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461005+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:57:20.461005+0000","last_peered":"2024-09-15T21:57:20.461005+0000","last_clean":"2024-09-15T21:57:20.461005+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:57:20.461005+0000","last_undegraded":"2024-09-15T21:57:20.461005+0000","last_fullsized":"2024-09-15T21:57:20.461005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275613+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:57:20.275613+0000","last_peered":"2024-09-15T21:57:20.275613+0000","last_clean":"2024-09-15T21:57:20.275613+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:57:20.275613+0000","last_undegraded":"2024-09-15T21:57:20.275613+0000","last_fullsized":"2024-09-15T21:57:20.275613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463309+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:57:20.463309+0000","last_peered":"2024-09-15T21:57:20.463309+0000","last_clean":"2024-09-15T21:57:20.463309+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:57:20.463309+0000","last_undegraded":"2024-09-15T21:57:20.463309+0000","last_fullsized":"2024-09-15T21:57:20.463309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'8","reported_seq":254,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712100+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:57:20.712100+0000","last_peered":"2024-09-15T21:57:20.712100+0000","last_clean":"2024-09-15T21:57:20.712100+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:57:20.712100+0000","last_undegraded":"2024-09-15T21:57:20.712100+0000","last_fullsized":"2024-09-15T21:57:20.712100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":206,"num_read_kb":206,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278726+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:57:20.278726+0000","last_peered":"2024-09-15T21:57:20.278726+0000","last_clean":"2024-09-15T21:57:20.278726+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:57:20.278726+0000","last_undegraded":"2024-09-15T21:57:20.278726+0000","last_fullsized":"2024-09-15T21:57:20.278726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467332+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:57:20.467332+0000","last_peered":"2024-09-15T21:57:20.467332+0000","last_clean":"2024-09-15T21:57:20.467332+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:57:20.467332+0000","last_undegraded":"2024-09-15T21:57:20.467332+0000","last_fullsized":"2024-09-15T21:57:20.467332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275825+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:57:20.275825+0000","last_peered":"2024-09-15T21:57:20.275825+0000","last_clean":"2024-09-15T21:57:20.275825+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:57:20.275825+0000","last_undegraded":"2024-09-15T21:57:20.275825+0000","last_fullsized":"2024-09-15T21:57:20.275825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187675+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:57:20.187675+0000","last_peered":"2024-09-15T21:57:20.187675+0000","last_clean":"2024-09-15T21:57:20.187675+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:57:20.187675+0000","last_undegraded":"2024-09-15T21:57:20.187675+0000","last_fullsized":"2024-09-15T21:57:20.187675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472373+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:57:20.472373+0000","last_peered":"2024-09-15T21:57:20.472373+0000","last_clean":"2024-09-15T21:57:20.472373+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:57:20.472373+0000","last_undegraded":"2024-09-15T21:57:20.472373+0000","last_fullsized":"2024-09-15T21:57:20.472373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278372+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:57:20.278372+0000","last_peered":"2024-09-15T21:57:20.278372+0000","last_clean":"2024-09-15T21:57:20.278372+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:57:20.278372+0000","last_undegraded":"2024-09-15T21:57:20.278372+0000","last_fullsized":"2024-09-15T21:57:20.278372+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185595+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:57:20.185595+0000","last_peered":"2024-09-15T21:57:20.185595+0000","last_clean":"2024-09-15T21:57:20.185595+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:57:20.185595+0000","last_undegraded":"2024-09-15T21:57:20.185595+0000","last_fullsized":"2024-09-15T21:57:20.185595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285572+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:57:20.285572+0000","last_peered":"2024-09-15T21:57:20.285572+0000","last_clean":"2024-09-15T21:57:20.285572+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:57:20.285572+0000","last_undegraded":"2024-09-15T21:57:20.285572+0000","last_fullsized":"2024-09-15T21:57:20.285572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187909+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:57:20.187909+0000","last_peered":"2024-09-15T21:57:20.187909+0000","last_clean":"2024-09-15T21:57:20.187909+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:57:20.187909+0000","last_undegraded":"2024-09-15T21:57:20.187909+0000","last_fullsized":"2024-09-15T21:57:20.187909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396115+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:57:20.396115+0000","last_peered":"2024-09-15T21:57:20.396115+0000","last_clean":"2024-09-15T21:57:20.396115+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:57:20.396115+0000","last_undegraded":"2024-09-15T21:57:20.396115+0000","last_fullsized":"2024-09-15T21:57:20.396115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'9","reported_seq":158,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.712626+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:57:20.712626+0000","last_peered":"2024-09-15T21:57:20.712626+0000","last_clean":"2024-09-15T21:57:20.712626+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:57:20.712626+0000","last_undegraded":"2024-09-15T21:57:20.712626+0000","last_fullsized":"2024-09-15T21:57:20.712626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1857,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":109,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278171+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:57:20.278171+0000","last_peered":"2024-09-15T21:57:20.278171+0000","last_clean":"2024-09-15T21:57:20.278171+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:57:20.278171+0000","last_undegraded":"2024-09-15T21:57:20.278171+0000","last_fullsized":"2024-09-15T21:57:20.278171+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396197+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:57:20.396197+0000","last_peered":"2024-09-15T21:57:20.396197+0000","last_clean":"2024-09-15T21:57:20.396197+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:57:20.396197+0000","last_undegraded":"2024-09-15T21:57:20.396197+0000","last_fullsized":"2024-09-15T21:57:20.396197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471807+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:57:20.471807+0000","last_peered":"2024-09-15T21:57:20.471807+0000","last_clean":"2024-09-15T21:57:20.471807+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:57:20.471807+0000","last_undegraded":"2024-09-15T21:57:20.471807+0000","last_fullsized":"2024-09-15T21:57:20.471807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396260+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:57:20.396260+0000","last_peered":"2024-09-15T21:57:20.396260+0000","last_clean":"2024-09-15T21:57:20.396260+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:57:20.396260+0000","last_undegraded":"2024-09-15T21:57:20.396260+0000","last_fullsized":"2024-09-15T21:57:20.396260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395966+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:57:20.395966+0000","last_peered":"2024-09-15T21:57:20.395966+0000","last_clean":"2024-09-15T21:57:20.395966+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:57:20.395966+0000","last_undegraded":"2024-09-15T21:57:20.395966+0000","last_fullsized":"2024-09-15T21:57:20.395966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466952+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:57:20.466952+0000","last_peered":"2024-09-15T21:57:20.466952+0000","last_clean":"2024-09-15T21:57:20.466952+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:57:20.466952+0000","last_undegraded":"2024-09-15T21:57:20.466952+0000","last_fullsized":"2024-09-15T21:57:20.466952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186090+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:57:20.186090+0000","last_peered":"2024-09-15T21:57:20.186090+0000","last_clean":"2024-09-15T21:57:20.186090+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:57:20.186090+0000","last_undegraded":"2024-09-15T21:57:20.186090+0000","last_fullsized":"2024-09-15T21:57:20.186090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472240+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:57:20.472240+0000","last_peered":"2024-09-15T21:57:20.472240+0000","last_clean":"2024-09-15T21:57:20.472240+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:57:20.472240+0000","last_undegraded":"2024-09-15T21:57:20.472240+0000","last_fullsized":"2024-09-15T21:57:20.472240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396338+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:57:20.396338+0000","last_peered":"2024-09-15T21:57:20.396338+0000","last_clean":"2024-09-15T21:57:20.396338+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:57:20.396338+0000","last_undegraded":"2024-09-15T21:57:20.396338+0000","last_fullsized":"2024-09-15T21:57:20.396338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280054+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:57:20.280054+0000","last_peered":"2024-09-15T21:57:20.280054+0000","last_clean":"2024-09-15T21:57:20.280054+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:57:20.280054+0000","last_undegraded":"2024-09-15T21:57:20.280054+0000","last_fullsized":"2024-09-15T21:57:20.280054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187475+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:57:20.187475+0000","last_peered":"2024-09-15T21:57:20.187475+0000","last_clean":"2024-09-15T21:57:20.187475+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:57:20.187475+0000","last_undegraded":"2024-09-15T21:57:20.187475+0000","last_fullsized":"2024-09-15T21:57:20.187475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278973+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:57:20.278973+0000","last_peered":"2024-09-15T21:57:20.278973+0000","last_clean":"2024-09-15T21:57:20.278973+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:57:20.278973+0000","last_undegraded":"2024-09-15T21:57:20.278973+0000","last_fullsized":"2024-09-15T21:57:20.278973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463016+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:57:20.463016+0000","last_peered":"2024-09-15T21:57:20.463016+0000","last_clean":"2024-09-15T21:57:20.463016+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:57:20.463016+0000","last_undegraded":"2024-09-15T21:57:20.463016+0000","last_fullsized":"2024-09-15T21:57:20.463016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186044+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:57:20.186044+0000","last_peered":"2024-09-15T21:57:20.186044+0000","last_clean":"2024-09-15T21:57:20.186044+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:57:20.186044+0000","last_undegraded":"2024-09-15T21:57:20.186044+0000","last_fullsized":"2024-09-15T21:57:20.186044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472203+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:57:20.472203+0000","last_peered":"2024-09-15T21:57:20.472203+0000","last_clean":"2024-09-15T21:57:20.472203+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:57:20.472203+0000","last_undegraded":"2024-09-15T21:57:20.472203+0000","last_fullsized":"2024-09-15T21:57:20.472203+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396319+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:57:20.396319+0000","last_peered":"2024-09-15T21:57:20.396319+0000","last_clean":"2024-09-15T21:57:20.396319+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:57:20.396319+0000","last_undegraded":"2024-09-15T21:57:20.396319+0000","last_fullsized":"2024-09-15T21:57:20.396319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276457+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:57:20.276457+0000","last_peered":"2024-09-15T21:57:20.276457+0000","last_clean":"2024-09-15T21:57:20.276457+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:57:20.276457+0000","last_undegraded":"2024-09-15T21:57:20.276457+0000","last_fullsized":"2024-09-15T21:57:20.276457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472556+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:57:20.472556+0000","last_peered":"2024-09-15T21:57:20.472556+0000","last_clean":"2024-09-15T21:57:20.472556+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:57:20.472556+0000","last_undegraded":"2024-09-15T21:57:20.472556+0000","last_fullsized":"2024-09-15T21:57:20.472556+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462857+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:57:20.462857+0000","last_peered":"2024-09-15T21:57:20.462857+0000","last_clean":"2024-09-15T21:57:20.462857+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:57:20.462857+0000","last_undegraded":"2024-09-15T21:57:20.462857+0000","last_fullsized":"2024-09-15T21:57:20.462857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466723+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:57:20.466723+0000","last_peered":"2024-09-15T21:57:20.466723+0000","last_clean":"2024-09-15T21:57:20.466723+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:57:20.466723+0000","last_undegraded":"2024-09-15T21:57:20.466723+0000","last_fullsized":"2024-09-15T21:57:20.466723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466886+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:57:20.466886+0000","last_peered":"2024-09-15T21:57:20.466886+0000","last_clean":"2024-09-15T21:57:20.466886+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:57:20.466886+0000","last_undegraded":"2024-09-15T21:57:20.466886+0000","last_fullsized":"2024-09-15T21:57:20.466886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186173+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:57:20.186173+0000","last_peered":"2024-09-15T21:57:20.186173+0000","last_clean":"2024-09-15T21:57:20.186173+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:57:20.186173+0000","last_undegraded":"2024-09-15T21:57:20.186173+0000","last_fullsized":"2024-09-15T21:57:20.186173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276015+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:57:20.276015+0000","last_peered":"2024-09-15T21:57:20.276015+0000","last_clean":"2024-09-15T21:57:20.276015+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:57:20.276015+0000","last_undegraded":"2024-09-15T21:57:20.276015+0000","last_fullsized":"2024-09-15T21:57:20.276015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461069+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:57:20.461069+0000","last_peered":"2024-09-15T21:57:20.461069+0000","last_clean":"2024-09-15T21:57:20.461069+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:57:20.461069+0000","last_undegraded":"2024-09-15T21:57:20.461069+0000","last_fullsized":"2024-09-15T21:57:20.461069+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.286073+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:57:20.286073+0000","last_peered":"2024-09-15T21:57:20.286073+0000","last_clean":"2024-09-15T21:57:20.286073+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:57:20.286073+0000","last_undegraded":"2024-09-15T21:57:20.286073+0000","last_fullsized":"2024-09-15T21:57:20.286073+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463316+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:57:20.463316+0000","last_peered":"2024-09-15T21:57:20.463316+0000","last_clean":"2024-09-15T21:57:20.463316+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:57:20.463316+0000","last_undegraded":"2024-09-15T21:57:20.463316+0000","last_fullsized":"2024-09-15T21:57:20.463316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278186+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:57:20.278186+0000","last_peered":"2024-09-15T21:57:20.278186+0000","last_clean":"2024-09-15T21:57:20.278186+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:57:20.278186+0000","last_undegraded":"2024-09-15T21:57:20.278186+0000","last_fullsized":"2024-09-15T21:57:20.278186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460530+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:57:20.460530+0000","last_peered":"2024-09-15T21:57:20.460530+0000","last_clean":"2024-09-15T21:57:20.460530+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:57:20.460530+0000","last_undegraded":"2024-09-15T21:57:20.460530+0000","last_fullsized":"2024-09-15T21:57:20.460530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285510+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:57:20.285510+0000","last_peered":"2024-09-15T21:57:20.285510+0000","last_clean":"2024-09-15T21:57:20.285510+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:57:20.285510+0000","last_undegraded":"2024-09-15T21:57:20.285510+0000","last_fullsized":"2024-09-15T21:57:20.285510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278179+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:57:20.278179+0000","last_peered":"2024-09-15T21:57:20.278179+0000","last_clean":"2024-09-15T21:57:20.278179+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:57:20.278179+0000","last_undegraded":"2024-09-15T21:57:20.278179+0000","last_fullsized":"2024-09-15T21:57:20.278179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190742+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:57:20.190742+0000","last_peered":"2024-09-15T21:57:20.190742+0000","last_clean":"2024-09-15T21:57:20.190742+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:57:20.190742+0000","last_undegraded":"2024-09-15T21:57:20.190742+0000","last_fullsized":"2024-09-15T21:57:20.190742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467111+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:57:20.467111+0000","last_peered":"2024-09-15T21:57:20.467111+0000","last_clean":"2024-09-15T21:57:20.467111+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:57:20.467111+0000","last_undegraded":"2024-09-15T21:57:20.467111+0000","last_fullsized":"2024-09-15T21:57:20.467111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467430+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:57:20.467430+0000","last_peered":"2024-09-15T21:57:20.467430+0000","last_clean":"2024-09-15T21:57:20.467430+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:57:20.467430+0000","last_undegraded":"2024-09-15T21:57:20.467430+0000","last_fullsized":"2024-09-15T21:57:20.467430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472483+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:57:20.472483+0000","last_peered":"2024-09-15T21:57:20.472483+0000","last_clean":"2024-09-15T21:57:20.472483+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:57:20.472483+0000","last_undegraded":"2024-09-15T21:57:20.472483+0000","last_fullsized":"2024-09-15T21:57:20.472483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":603167,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1172,"num_read_kb":704,"num_write":389,"num_write_kb":1477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1809501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514538,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1081344,"data_stored":536154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776174,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28900,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716252,"statfs":{"total":95995035648,"available":95965442048,"internally_reserved":0,"allocated":1921024,"data_stored":1330627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939701,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36804,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708348,"statfs":{"total":95995035648,"available":95957348352,"internally_reserved":0,"allocated":1490944,"data_stored":923695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168633,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1486848,"data_stored":927587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397567,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36988,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708164,"statfs":{"total":95995035648,"available":95957159936,"internally_reserved":0,"allocated":1679360,"data_stored":1122064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561090,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":530472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28820,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716332,"statfs":{"total":95995035648,"available":95965523968,"internally_reserved":0,"allocated":1839104,"data_stored":1259476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":534939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28784,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716368,"statfs":{"total":95995035648,"available":95965560832,"internally_reserved":0,"allocated":1617920,"data_stored":1054603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":590670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1150976,"data_stored":600090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":532462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:57:25.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:57:26.081 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:26 smithi028 ceph-mon[30969]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:57:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:26 smithi028 ceph-mon[30969]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-09-15T21:57:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:26 smithi203 ceph-mon[35546]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:57:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:26 smithi203 ceph-mon[35546]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-09-15T21:57:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:26 smithi174 ceph-mon[35244]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T21:57:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:26 smithi174 ceph-mon[35244]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-09-15T21:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 -- 172.21.15.28:0/3251274409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 msgr2=0x7fcfdc10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 --2- 172.21.15.28:0/3251274409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc10bbf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009900 tx=0x7fcfc402a480 comp rx=0 tx=0).stop 2024-09-15T21:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 -- 172.21.15.28:0/3251274409 shutdown_connections 2024-09-15T21:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 --2- 172.21.15.28:0/3251274409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfdc10c180 0x7fcfdc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 --2- 172.21.15.28:0/3251274409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc10bbf0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 --2- 172.21.15.28:0/3251274409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfdc102de0 0x7fcfdc1031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 -- 172.21.15.28:0/3251274409 >> 172.21.15.28:0/3251274409 conn(0x7fcfdc0fc7a0 msgr2=0x7fcfdc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 -- 172.21.15.28:0/3251274409 shutdown_connections 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 -- 172.21.15.28:0/3251274409 wait complete. 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.574+0000 7fcfe172e640 1 Processor -- start 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfe172e640 1 -- start start 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfe172e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfdc102de0 0x7fcfdc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfe172e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfe172e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfdc10c180 0x7fcfdc19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfe172e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc111f70 con 0x7fcfdc103700 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfe172e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc111df0 con 0x7fcfdc102de0 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfe172e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1120f0 con 0x7fcfdc10c180 2024-09-15T21:57:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59782/0 (socket says 172.21.15.28:59782) 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 -- 172.21.15.28:0/2343706880 learned_addr learned my addr 172.21.15.28:0/2343706880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfdb7fe640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfdc10c180 0x7fcfdc19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfdaffd640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfdc102de0 0x7fcfdc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 -- 172.21.15.28:0/2343706880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfdc10c180 msgr2=0x7fcfdc19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfdc10c180 0x7fcfdc19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 -- 172.21.15.28:0/2343706880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfdc102de0 msgr2=0x7fcfdc198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfdc102de0 0x7fcfdc198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.575+0000 7fcfd3fff640 1 -- 172.21.15.28:0/2343706880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc19d420 con 0x7fcfdc103700 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.576+0000 7fcfd3fff640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc1989b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4006010 tx=0x7fcfc40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.576+0000 7fcfd8ff9640 1 -- 172.21.15.28:0/2343706880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc403f030 con 0x7fcfdc103700 2024-09-15T21:57:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.576+0000 7fcfd8ff9640 1 -- 172.21.15.28:0/2343706880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc4003ce0 con 0x7fcfdc103700 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.576+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc19d6b0 con 0x7fcfdc103700 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.576+0000 7fcfd8ff9640 1 -- 172.21.15.28:0/2343706880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4035070 con 0x7fcfdc103700 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.576+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc1a4ff0 con 0x7fcfdc103700 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.577+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc06afd0 con 0x7fcfdc103700 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.578+0000 7fcfd8ff9640 1 -- 172.21.15.28:0/2343706880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fcfc4042030 con 0x7fcfdc103700 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.579+0000 7fcfd8ff9640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfc80a34d0 0x7fcfc80a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.579+0000 7fcfdaffd640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfc80a34d0 0x7fcfc80a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.579+0000 7fcfd8ff9640 1 -- 172.21.15.28:0/2343706880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fcfc40f0920 con 0x7fcfdc103700 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.579+0000 7fcfdaffd640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfc80a34d0 0x7fcfc80a5990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005fd0 tx=0x7fcfcc005e20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.583+0000 7fcfd8ff9640 1 -- 172.21.15.28:0/2343706880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcfc40bcee0 con 0x7fcfdc103700 2024-09-15T21:57:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.796+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfdc107200 con 0x7fcfdc103700 2024-09-15T21:57:26.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.796+0000 7fcfd8ff9640 1 -- 172.21.15.28:0/2343706880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fcfc40bc7e0 con 0x7fcfdc103700 2024-09-15T21:57:26.799 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:26.800 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:57:26.800 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfc80a34d0 msgr2=0x7fcfc80a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfc80a34d0 0x7fcfc80a5990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005fd0 tx=0x7fcfcc005e20 comp rx=0 tx=0).stop 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 msgr2=0x7fcfdc1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc1989b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4006010 tx=0x7fcfc40042f0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 shutdown_connections 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfc80a34d0 0x7fcfc80a5990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfdc10c180 0x7fcfdc19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfdc103700 0x7fcfdc1989b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 --2- 172.21.15.28:0/2343706880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfdc102de0 0x7fcfdc198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 >> 172.21.15.28:0/2343706880 conn(0x7fcfdc0fc7a0 msgr2=0x7fcfdc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 shutdown_connections 2024-09-15T21:57:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:26.799+0000 7fcfe172e640 1 -- 172.21.15.28:0/2343706880 wait complete. 2024-09-15T21:57:27.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:27 smithi028 ceph-mon[30969]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:27 smithi028 ceph-mon[30969]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:27 smithi028 ceph-mon[30969]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:27 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2343706880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:27.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:57:27.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:27 smithi203 ceph-mon[35546]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:27 smithi203 ceph-mon[35546]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:27 smithi203 ceph-mon[35546]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:27 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2343706880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:27.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:27 smithi174 ceph-mon[35244]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:27 smithi174 ceph-mon[35244]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:27 smithi174 ceph-mon[35244]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:27.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:27 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2343706880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:28.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:28 smithi028 ceph-mon[30969]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.0 KiB/s wr, 7 op/s 2024-09-15T21:57:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.165+0000 7fb3c2fef640 1 -- 172.21.15.28:0/3100064984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc1027a0 msgr2=0x7fb3bc1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.165+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/3100064984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc1027a0 0x7fb3bc1067f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009900 tx=0x7fb3b002a410 comp rx=0 tx=0).stop 2024-09-15T21:57:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.165+0000 7fb3c2fef640 1 -- 172.21.15.28:0/3100064984 shutdown_connections 2024-09-15T21:57:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.165+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/3100064984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3bc106f20 0x7fb3bc10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.165+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/3100064984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc1027a0 0x7fb3bc1067f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.165+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/3100064984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3bc101df0 0x7fb3bc1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.165+0000 7fb3c2fef640 1 -- 172.21.15.28:0/3100064984 >> 172.21.15.28:0/3100064984 conn(0x7fb3bc0fd8b0 msgr2=0x7fb3bc0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 -- 172.21.15.28:0/3100064984 shutdown_connections 2024-09-15T21:57:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 -- 172.21.15.28:0/3100064984 wait complete. 2024-09-15T21:57:28.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 Processor -- start 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 -- start start 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc101df0 0x7fb3bc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3bc1027a0 0x7fb3bc198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3bc106f20 0x7fb3bc19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc111f40 con 0x7fb3bc101df0 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc111dc0 con 0x7fb3bc1027a0 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c2fef640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc1120c0 con 0x7fb3bc106f20 2024-09-15T21:57:28.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.166+0000 7fb3c0d8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc101df0 0x7fb3bc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc101df0 0x7fb3bc198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59790/0 (socket says 172.21.15.28:59790) 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 -- 172.21.15.28:0/1560749663 learned_addr learned my addr 172.21.15.28:0/1560749663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 -- 172.21.15.28:0/1560749663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3bc106f20 msgr2=0x7fb3bc19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c158c640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3bc106f20 0x7fb3bc19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3bbfff640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3bc1027a0 0x7fb3bc198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3bc106f20 0x7fb3bc19cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 -- 172.21.15.28:0/1560749663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3bc1027a0 msgr2=0x7fb3bc198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3bc1027a0 0x7fb3bc198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 -- 172.21.15.28:0/1560749663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3bc19d3f0 con 0x7fb3bc101df0 2024-09-15T21:57:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c0d8b640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc101df0 0x7fb3bc198440 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00c920 tx=0x7fb3ac00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3b9ffb640 1 -- 172.21.15.28:0/1560749663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac007c70 con 0x7fb3bc101df0 2024-09-15T21:57:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3b9ffb640 1 -- 172.21.15.28:0/1560749663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ac007e10 con 0x7fb3bc101df0 2024-09-15T21:57:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3bc19d6e0 con 0x7fb3bc101df0 2024-09-15T21:57:28.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3b9ffb640 1 -- 172.21.15.28:0/1560749663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac018890 con 0x7fb3bc101df0 2024-09-15T21:57:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.167+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3bc1a4fc0 con 0x7fb3bc101df0 2024-09-15T21:57:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.168+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3bc1021d0 con 0x7fb3bc101df0 2024-09-15T21:57:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.170+0000 7fb3b9ffb640 1 -- 172.21.15.28:0/1560749663 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fb3ac018a70 con 0x7fb3bc101df0 2024-09-15T21:57:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.170+0000 7fb3b9ffb640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb3a40a34d0 0x7fb3a40a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.170+0000 7fb3bbfff640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb3a40a34d0 0x7fb3a40a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.170+0000 7fb3b9ffb640 1 -- 172.21.15.28:0/1560749663 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb3ac0d0530 con 0x7fb3bc101df0 2024-09-15T21:57:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.171+0000 7fb3bbfff640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb3a40a34d0 0x7fb3a40a5990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00096c0 tx=0x7fb3b0000bc0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.174+0000 7fb3b9ffb640 1 -- 172.21.15.28:0/1560749663 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb3ac014030 con 0x7fb3bc101df0 2024-09-15T21:57:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.350+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3bc1a51f0 con 0x7fb3bc101df0 2024-09-15T21:57:28.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.352+0000 7fb3b9ffb640 1 -- 172.21.15.28:0/1560749663 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+25138 (secure 0 0 0) 0x7fb3ac00de20 con 0x7fb3bc101df0 2024-09-15T21:57:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:28.355 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:57:20.180529+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/972903625":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/3676950981":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/4057362541":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/955882315":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/1670367809":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6801/1670367809":"2024-09-16T21:57:20.180486+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:57:28.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb3a40a34d0 msgr2=0x7fb3a40a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:28.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb3a40a34d0 0x7fb3a40a5990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00096c0 tx=0x7fb3b0000bc0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc101df0 msgr2=0x7fb3bc198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:28.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc101df0 0x7fb3bc198440 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00c920 tx=0x7fb3ac00cde0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 shutdown_connections 2024-09-15T21:57:28.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb3a40a34d0 0x7fb3a40a5990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3bc106f20 0x7fb3bc19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3bc1027a0 0x7fb3bc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 --2- 172.21.15.28:0/1560749663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3bc101df0 0x7fb3bc198440 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:28.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 >> 172.21.15.28:0/1560749663 conn(0x7fb3bc0fd8b0 msgr2=0x7fb3bc0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:28.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 shutdown_connections 2024-09-15T21:57:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:28.355+0000 7fb3c2fef640 1 -- 172.21.15.28:0/1560749663 wait complete. 2024-09-15T21:57:28.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:28 smithi203 ceph-mon[35546]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.0 KiB/s wr, 7 op/s 2024-09-15T21:57:28.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:28 smithi174 ceph-mon[35244]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.0 KiB/s wr, 7 op/s 2024-09-15T21:57:29.041 DEBUG:tasks.fwd_scrub:Yielding 2024-09-15T21:57:29.041 INFO:teuthology.run_tasks:Running task exec... 2024-09-15T21:57:29.052 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T21:57:29.052 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:57:29.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-09-15T21:57:29.052 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-09-15T21:57:29.053 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T21:57:29.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 -- 172.21.15.28:0/967809647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc1069c0 msgr2=0x7f17fc106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 --2- 172.21.15.28:0/967809647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc1069c0 0x7f17fc106da0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009930 tx=0x7f17ec02a420 comp rx=0 tx=0).stop 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 -- 172.21.15.28:0/967809647 shutdown_connections 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 --2- 172.21.15.28:0/967809647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17fc101350 0x7f17fc10d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 --2- 172.21.15.28:0/967809647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17fc1009b0 0x7f17fc100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 --2- 172.21.15.28:0/967809647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc1069c0 0x7f17fc106da0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 -- 172.21.15.28:0/967809647 >> 172.21.15.28:0/967809647 conn(0x7f17fc077990 msgr2=0x7f17fc0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 -- 172.21.15.28:0/967809647 shutdown_connections 2024-09-15T21:57:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 -- 172.21.15.28:0/967809647 wait complete. 2024-09-15T21:57:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.159+0000 7f1802ee6640 1 Processor -- start 2024-09-15T21:57:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1802ee6640 1 -- start start 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1802ee6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17fc1009b0 0x7f17fc198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1802ee6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc101350 0x7f17fc198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1802ee6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17fc1069c0 0x7f17fc19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1802ee6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc110e00 con 0x7f17fc101350 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1802ee6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc110c80 con 0x7f17fc1009b0 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1802ee6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc110f80 con 0x7f17fc1069c0 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc101350 0x7f17fc198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc101350 0x7f17fc198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59816/0 (socket says 172.21.15.28:59816) 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 -- 172.21.15.28:0/3460164046 learned_addr learned my addr 172.21.15.28:0/3460164046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1801483640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17fc1069c0 0x7f17fc19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1800c82640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17fc1009b0 0x7f17fc198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 -- 172.21.15.28:0/3460164046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17fc1069c0 msgr2=0x7f17fc19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17fc1069c0 0x7f17fc19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 -- 172.21.15.28:0/3460164046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17fc1009b0 msgr2=0x7f17fc198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17fc1009b0 0x7f17fc198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 -- 172.21.15.28:0/3460164046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc19d330 con 0x7f17fc101350 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f1800c82640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17fc1009b0 0x7f17fc198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.160+0000 7f17fbfff640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc101350 0x7f17fc198880 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f17e400e9f0 tx=0x7f17e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.161+0000 7f17f9ffb640 1 -- 172.21.15.28:0/3460164046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e400cd80 con 0x7f17fc101350 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.161+0000 7f17f9ffb640 1 -- 172.21.15.28:0/3460164046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e4010910 con 0x7f17fc101350 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.161+0000 7f17f9ffb640 1 -- 172.21.15.28:0/3460164046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4018990 con 0x7f17fc101350 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.161+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17fc19d620 con 0x7f17fc101350 2024-09-15T21:57:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.161+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17fc1a4eb0 con 0x7f17fc101350 2024-09-15T21:57:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.162+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17fc100e10 con 0x7f17fc101350 2024-09-15T21:57:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.164+0000 7f17f9ffb640 1 -- 172.21.15.28:0/3460164046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f17e4010ac0 con 0x7f17fc101350 2024-09-15T21:57:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.164+0000 7f17f9ffb640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17e80a3580 0x7f17e80a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.165+0000 7f1800c82640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17e80a3580 0x7f17e80a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.165+0000 7f17f9ffb640 1 -- 172.21.15.28:0/3460164046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f17e40d0ac0 con 0x7f17fc101350 2024-09-15T21:57:29.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.165+0000 7f1800c82640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17e80a3580 0x7f17e80a5a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f17ec000c00 tx=0x7f17ec005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.168+0000 7f17f9ffb640 1 -- 172.21.15.28:0/3460164046 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f17e409cef0 con 0x7f17fc101350 2024-09-15T21:57:29.317 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.353+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f17fc1a5150 con 0x7f17fc101350 2024-09-15T21:57:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.358+0000 7f17f9ffb640 1 -- 172.21.15.28:0/3460164046 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v26)=0 v26) ==== 129+0+0 (secure 0 0 0) 0x7f17e409c7f0 con 0x7f17fc101350 2024-09-15T21:57:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17e80a3580 msgr2=0x7f17e80a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17e80a3580 0x7f17e80a5a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f17ec000c00 tx=0x7f17ec005a20 comp rx=0 tx=0).stop 2024-09-15T21:57:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc101350 msgr2=0x7f17fc198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc101350 0x7f17fc198880 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f17e400e9f0 tx=0x7f17e400eeb0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 shutdown_connections 2024-09-15T21:57:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17e80a3580 0x7f17e80a5a40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17fc1069c0 0x7f17fc19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc101350 0x7f17fc198880 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 --2- 172.21.15.28:0/3460164046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17fc1009b0 0x7f17fc198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 >> 172.21.15.28:0/3460164046 conn(0x7f17fc077990 msgr2=0x7f17fc0ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 shutdown_connections 2024-09-15T21:57:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.360+0000 7f1802ee6640 1 -- 172.21.15.28:0/3460164046 wait complete. 2024-09-15T21:57:29.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:29 smithi028 ceph-mon[30969]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1 KiB/s wr, 25 op/s 2024-09-15T21:57:29.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:29 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1560749663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:29.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-09-15T21:57:29.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:29 smithi203 ceph-mon[35546]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1 KiB/s wr, 25 op/s 2024-09-15T21:57:29.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:29 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1560749663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 -- 172.21.15.28:0/3257436798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c103600 msgr2=0x7fa12c10bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 --2- 172.21.15.28:0/3257436798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c103600 0x7fa12c10bae0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa120009900 tx=0x7fa12002a480 comp rx=0 tx=0).stop 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 -- 172.21.15.28:0/3257436798 shutdown_connections 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 --2- 172.21.15.28:0/3257436798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa12c10c0b0 0x7fa12c10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 --2- 172.21.15.28:0/3257436798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c103600 0x7fa12c10bae0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 --2- 172.21.15.28:0/3257436798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c102ce0 0x7fa12c1030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 -- 172.21.15.28:0/3257436798 >> 172.21.15.28:0/3257436798 conn(0x7fa12c077990 msgr2=0x7fa12c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 -- 172.21.15.28:0/3257436798 shutdown_connections 2024-09-15T21:57:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 -- 172.21.15.28:0/3257436798 wait complete. 2024-09-15T21:57:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.482+0000 7fa1313cf640 1 Processor -- start 2024-09-15T21:57:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa1313cf640 1 -- start start 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa1313cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c102ce0 0x7fa12c100930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa1313cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa12c103600 0x7fa12c100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa1313cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c10c0b0 0x7fa12c1013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa1313cf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c111eb0 con 0x7fa12c10c0b0 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa1313cf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c111d30 con 0x7fa12c102ce0 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa1313cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c112030 con 0x7fa12c103600 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c10c0b0 0x7fa12c1013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c10c0b0 0x7fa12c1013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59834/0 (socket says 172.21.15.28:59834) 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 -- 172.21.15.28:0/2037248027 learned_addr learned my addr 172.21.15.28:0/2037248027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12a7fc640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa12c103600 0x7fa12c100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12affd640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c102ce0 0x7fa12c100930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 -- 172.21.15.28:0/2037248027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa12c103600 msgr2=0x7fa12c100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa12c103600 0x7fa12c100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 -- 172.21.15.28:0/2037248027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c102ce0 msgr2=0x7fa12c100930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c102ce0 0x7fa12c100930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 -- 172.21.15.28:0/2037248027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c1a6840 con 0x7fa12c10c0b0 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12affd640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c102ce0 0x7fa12c100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.483+0000 7fa12b7fe640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c10c0b0 0x7fa12c1013b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa12400eb10 tx=0x7fa12400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.484+0000 7fa113fff640 1 -- 172.21.15.28:0/2037248027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124004b20 con 0x7fa12c10c0b0 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.484+0000 7fa113fff640 1 -- 172.21.15.28:0/2037248027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa124004cc0 con 0x7fa12c10c0b0 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.484+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12c1a6b30 con 0x7fa12c10c0b0 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.484+0000 7fa113fff640 1 -- 172.21.15.28:0/2037248027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124010760 con 0x7fa12c10c0b0 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.484+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12c1a7050 con 0x7fa12c10c0b0 2024-09-15T21:57:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.485+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa12c1030c0 con 0x7fa12c10c0b0 2024-09-15T21:57:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.487+0000 7fa113fff640 1 -- 172.21.15.28:0/2037248027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fa124005000 con 0x7fa12c10c0b0 2024-09-15T21:57:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.487+0000 7fa113fff640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa1140a3580 0x7fa1140a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.487+0000 7fa113fff640 1 -- 172.21.15.28:0/2037248027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa1240d0d70 con 0x7fa12c10c0b0 2024-09-15T21:57:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.488+0000 7fa12affd640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa1140a3580 0x7fa1140a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.488+0000 7fa12affd640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa1140a3580 0x7fa1140a5a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa11c005fd0 tx=0x7fa11c00b5f0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.491+0000 7fa113fff640 1 -- 172.21.15.28:0/2037248027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa12409d1a0 con 0x7fa12c10c0b0 2024-09-15T21:57:29.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:29 smithi174 ceph-mon[35244]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1 KiB/s wr, 25 op/s 2024-09-15T21:57:29.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:29 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1560749663' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.673+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fa12c101e60 con 0x7fa12c10c0b0 2024-09-15T21:57:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.677+0000 7fa113fff640 1 -- 172.21.15.28:0/2037248027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) ==== 138+0+0 (secure 0 0 0) 0x7fa12c101e60 con 0x7fa12c10c0b0 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa1140a3580 msgr2=0x7fa1140a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa1140a3580 0x7fa1140a5a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa11c005fd0 tx=0x7fa11c00b5f0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c10c0b0 msgr2=0x7fa12c1013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c10c0b0 0x7fa12c1013b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa12400eb10 tx=0x7fa12400c2d0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 shutdown_connections 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa1140a3580 0x7fa1140a5a40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa12c10c0b0 0x7fa12c1013b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa12c103600 0x7fa12c100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 --2- 172.21.15.28:0/2037248027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c102ce0 0x7fa12c100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 >> 172.21.15.28:0/2037248027 conn(0x7fa12c077990 msgr2=0x7fa12c0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 shutdown_connections 2024-09-15T21:57:29.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.679+0000 7fa1313cf640 1 -- 172.21.15.28:0/2037248027 wait complete. 2024-09-15T21:57:29.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-09-15T21:57:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 -- 172.21.15.28:0/47665112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 msgr2=0x7f4648101f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 --2- 172.21.15.28:0/47665112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648101f20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4644009930 tx=0x7f464402a420 comp rx=0 tx=0).stop 2024-09-15T21:57:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 -- 172.21.15.28:0/47665112 shutdown_connections 2024-09-15T21:57:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 --2- 172.21.15.28:0/47665112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4648102460 0x7f464810e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 --2- 172.21.15.28:0/47665112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648101f20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 --2- 172.21.15.28:0/47665112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4648107ad0 0x7f4648107eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 -- 172.21.15.28:0/47665112 >> 172.21.15.28:0/47665112 conn(0x7f46480fd820 msgr2=0x7f46480ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 -- 172.21.15.28:0/47665112 shutdown_connections 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 -- 172.21.15.28:0/47665112 wait complete. 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 Processor -- start 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.800+0000 7f464f8ae640 1 -- start start 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4648102460 0x7f4648198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4648107ad0 0x7f464819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4648111ed0 con 0x7f4648101ac0 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4648111d50 con 0x7f4648102460 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4648112050 con 0x7f4648107ad0 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59842/0 (socket says 172.21.15.28:59842) 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 -- 172.21.15.28:0/3290365284 learned_addr learned my addr 172.21.15.28:0/3290365284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464ce49640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4648102460 0x7f4648198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464de4b640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4648107ad0 0x7f464819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 -- 172.21.15.28:0/3290365284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4648107ad0 msgr2=0x7f464819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4648107ad0 0x7f464819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 -- 172.21.15.28:0/3290365284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4648102460 msgr2=0x7f4648198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4648102460 0x7f4648198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 -- 172.21.15.28:0/3290365284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464819d310 con 0x7f4648101ac0 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464ce49640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4648102460 0x7f4648198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464d64a640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648198320 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f463c00d950 tx=0x7f463c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f463a7fc640 1 -- 172.21.15.28:0/3290365284 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c017030 con 0x7f4648101ac0 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464819d600 con 0x7f4648101ac0 2024-09-15T21:57:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f463a7fc640 1 -- 172.21.15.28:0/3290365284 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f463c004020 con 0x7f4648101ac0 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f463a7fc640 1 -- 172.21.15.28:0/3290365284 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0047a0 con 0x7f4648101ac0 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.801+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46481a4e90 con 0x7f4648101ac0 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.802+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4648101f20 con 0x7f4648101ac0 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.804+0000 7f463a7fc640 1 -- 172.21.15.28:0/3290365284 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f463c0041d0 con 0x7f4648101ac0 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.805+0000 7f463a7fc640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f46340a3580 0x7f46340a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.805+0000 7f464ce49640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f46340a3580 0x7f46340a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.805+0000 7f463a7fc640 1 -- 172.21.15.28:0/3290365284 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f463c0cfd60 con 0x7f4648101ac0 2024-09-15T21:57:29.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.805+0000 7f464ce49640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f46340a3580 0x7f46340a5a40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4644000c00 tx=0x7f4644005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.809+0000 7f463a7fc640 1 -- 172.21.15.28:0/3290365284 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f463c09c190 con 0x7f4648101ac0 2024-09-15T21:57:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 -- 172.21.15.28:0/3329150736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 msgr2=0x7f1ae01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/3329150736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae01067e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009900 tx=0x7f1ad402a410 comp rx=0 tx=0).stop 2024-09-15T21:57:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 -- 172.21.15.28:0/3329150736 shutdown_connections 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/3329150736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ae0106f10 0x7f1ae010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/3329150736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae01067e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/3329150736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ae0101de0 0x7f1ae01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 -- 172.21.15.28:0/3329150736 >> 172.21.15.28:0/3329150736 conn(0x7f1ae00fd8c0 msgr2=0x7f1ae00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 -- 172.21.15.28:0/3329150736 shutdown_connections 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.819+0000 7f1ae67fc640 1 -- 172.21.15.28:0/3329150736 wait complete. 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 Processor -- start 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 -- start start 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ae0101de0 0x7f1ae0198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ae0106f10 0x7f1ae019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae0111f70 con 0x7f1ae0102790 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae0111df0 con 0x7f1ae0101de0 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1ae67fc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01120f0 con 0x7f1ae0106f10 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1adf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1adf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae0198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59868/0 (socket says 172.21.15.28:59868) 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.820+0000 7f1adf7fe640 1 -- 172.21.15.28:0/2623956052 learned_addr learned my addr 172.21.15.28:0/2623956052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1adffff640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ae0101de0 0x7f1ae0198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ae0106f10 0x7f1ae019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1adf7fe640 1 -- 172.21.15.28:0/2623956052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ae0106f10 msgr2=0x7f1ae019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ae0106f10 0x7f1ae019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1adf7fe640 1 -- 172.21.15.28:0/2623956052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ae0101de0 msgr2=0x7f1ae0198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ae0101de0 0x7f1ae0198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1adf7fe640 1 -- 172.21.15.28:0/2623956052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae019d3c0 con 0x7f1ae0102790 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae0198950 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4006010 tx=0x7f1ad40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1add7fa640 1 -- 172.21.15.28:0/2623956052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad403f030 con 0x7f1ae0102790 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1add7fa640 1 -- 172.21.15.28:0/2623956052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad4003ce0 con 0x7f1ae0102790 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae019d650 con 0x7f1ae0102790 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1add7fa640 1 -- 172.21.15.28:0/2623956052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4035070 con 0x7f1ae0102790 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.821+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae0074420 con 0x7f1ae0102790 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1add7fa640 1 -- 172.21.15.28:0/2623956052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f1ad4042030 con 0x7f1ae0102790 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 Processor -- start 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 -- start start 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398faf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639905800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b639908130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b639860280 con 0x55b639905d40 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b639807b10 con 0x55b6398fab60 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.824+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b63985c240 con 0x55b6398fb480 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398faf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1adf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b639908130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398faf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56840/0 (socket says 172.21.15.28:56840) 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 -- 172.21.15.28:0/1580494783 learned_addr learned my addr 172.21.15.28:0/1580494783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1adffff640 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639905800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1add7fa640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1ac80a7960 0x7f1ac80a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 -- 172.21.15.28:0/1580494783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 msgr2=0x55b639905800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639905800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 -- 172.21.15.28:0/1580494783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 msgr2=0x55b639908130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b639908130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 -- 172.21.15.28:0/1580494783 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6398141f0 con 0x55b6398fab60 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1add7fa640 1 -- 172.21.15.28:0/2623956052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f1ad40f07a0 con 0x7f1ae0102790 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1ac80a7960 0x7f1ac80a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.825+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398faf40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8007ec0 tx=0x7f1ad800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=918f0285c1a3ad96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ac67fc640 1 -- 172.21.15.28:0/1580494783 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad801a540 con 0x55b6398fab60 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ac67fc640 1 -- 172.21.15.28:0/1580494783 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad801abc0 con 0x55b6398fab60 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ac67fc640 1 -- 172.21.15.28:0/1580494783 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8019aa0 con 0x55b6398fab60 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/1580494783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 msgr2=0x55b6398faf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398faf40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8007ec0 tx=0x7f1ad800f790 comp rx=0 tx=0).stop 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1ac80a7960 0x7f1ac80a9e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ad801e000 tx=0x7f1ad801f000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/1580494783 shutdown_connections 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b639908130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639905800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/1580494783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398faf40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/1580494783 >> 172.21.15.28:0/1580494783 conn(0x55b6398f70f0 msgr2=0x55b6398f74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.826+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/1580494783 shutdown_connections 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.827+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/1580494783 wait complete. 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.827+0000 7f1ae8bd5740 1 -- start start 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.827+0000 7f1ae8bd5740 1 Processor -- start 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.827+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6398fab60 0x55b639902c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.827+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639903150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.827+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b639905d40 0x55b639903690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b639908910 con 0x55b6398fab60 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b63981af10 con 0x55b639905d40 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b639908a90 con 0x55b6398fb480 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1ae8bd5740 10 client.0 _ll_register_callbacks cb 0x55b6397e3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6398fab60 0x55b639902c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6398fab60 0x55b639902c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59894/0 (socket says 172.21.15.28:59894) 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 -- 172.21.15.28:0/3107509497 learned_addr learned my addr 172.21.15.28:0/3107509497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639903150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b639905d40 0x55b639903690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 -- 172.21.15.28:0/3107509497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 msgr2=0x55b639903150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639903150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 -- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b639905d40 msgr2=0x55b639903690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b639905d40 0x55b639903690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 -- 172.21.15.28:0/3107509497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6395d16a0 con 0x55b6398fab60 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1adffff640 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6398fab60 0x55b639902c10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000b9c0 tx=0x7f1ad000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35dce9b99d7636ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1abf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3107509497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0011440 con 0x55b6398fab60 2024-09-15T21:57:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3107509497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f1ad00115e0 con 0x55b6398fab60 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.828+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad801abc0 con 0x55b6398fab60 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3107509497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1ad00117e0 con 0x55b6398fab60 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 10 client.15492 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b639914570 con 0x55b6398fab60 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3107509497 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad801abc0 con 0x55b6398fab60 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3107509497 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1ad001b940 con 0x55b6398fab60 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 10 client.15492 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 10 client.15492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 20 client.15492 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639917e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 4 client.15492 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.829+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b63986d7f0 con 0x55b639915980 2024-09-15T21:57:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.830+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639917e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.830+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639917e00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c034af6504feb7d7 server_cookie=d76d3a872a89af22 in_seq=0 out_seq=0 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.830+0000 7f1abf7fe640 10 client.15492 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3107509497 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b63986d7f0 con 0x55b639915980 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1abf7fe640 10 client.15492 handle_command_reply: tid=0 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 client.15492 shutdown 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6398fab60 msgr2=0x55b639902c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6398fab60 0x55b639902c10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000b9c0 tx=0x7f1ad000bdc0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 shutdown_connections 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639917e00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b639905d40 0x55b639903690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b639903150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1abf7fe640 0 client.15492 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3107509497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6398fab60 0x55b639902c10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 >> 172.21.15.28:0/3107509497 conn(0x55b6398f70f0 msgr2=0x55b6398fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 shutdown_connections 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3107509497 wait complete. 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.831+0000 7f1ae8bd5740 20 client.15492 trim_cache size 0 max 16384 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 Processor -- start 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 -- start start 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398f6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b63991d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63991dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01accd0 con 0x55b639905d40 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01ae810 con 0x55b6398fab60 2024-09-15T21:57:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01ad3a0 con 0x55b6398fb480 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae4d99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63991dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae4d99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63991dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59902/0 (socket says 172.21.15.28:59902) 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1ae4d99640 1 -- 172.21.15.28:0/2466937772 learned_addr learned my addr 172.21.15.28:0/2466937772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b63991d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.849+0000 7f1adffff640 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ae4d99640 1 -- 172.21.15.28:0/2466937772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 msgr2=0x55b63991d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b63991d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ae4d99640 1 -- 172.21.15.28:0/2466937772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 msgr2=0x55b6398f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ae4d99640 1 -- 172.21.15.28:0/2466937772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b639903f90 con 0x55b639905d40 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1adffff640 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63991dc60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8007fd0 tx=0x7f1ad8027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578e96b54657d160 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ac5ffb640 1 -- 172.21.15.28:0/2466937772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8035070 con 0x55b639905d40 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ac5ffb640 1 -- 172.21.15.28:0/2466937772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad8019e10 con 0x55b639905d40 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.850+0000 7f1ac5ffb640 1 -- 172.21.15.28:0/2466937772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad802c040 con 0x55b639905d40 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/2466937772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 msgr2=0x55b63991dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63991dc60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8007fd0 tx=0x7f1ad8027000 comp rx=0 tx=0).stop 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/2466937772 shutdown_connections 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63991dc60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fb480 0x55b63991d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/2466937772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fab60 0x55b6398f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/2466937772 >> 172.21.15.28:0/2466937772 conn(0x55b6398f70f0 msgr2=0x55b6398f5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/2466937772 shutdown_connections 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/2466937772 wait complete. 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.851+0000 7f1ae8bd5740 1 -- start start 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 1 Processor -- start 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fab60 0x55b63990dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fb480 0x55b63990e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63990acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63990acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01ae140 con 0x55b639905d40 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01ada70 con 0x55b6398fb480 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01aacb0 con 0x55b6398fab60 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63990acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59914/0 (socket says 172.21.15.28:59914) 2024-09-15T21:57:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1adffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fab60 0x55b63990dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 -- 172.21.15.28:0/3823254162 learned_addr learned my addr 172.21.15.28:0/3823254162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae8bd5740 10 client.0 _ll_register_callbacks cb 0x7f1ad00042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1adf7fe640 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fb480 0x55b63990e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 -- 172.21.15.28:0/3823254162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fab60 msgr2=0x55b63990dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fab60 0x55b63990dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 -- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fb480 msgr2=0x55b63990e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fb480 0x55b63990e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.852+0000 7f1ae4d99640 1 -- 172.21.15.28:0/3823254162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b63990b1f0 con 0x55b639905d40 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1adffff640 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fab60 0x55b63990dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1ae4d99640 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63990acb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8019400 tx=0x7f1ad8007520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c22f4f14573234b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1abf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3823254162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad803e060 con 0x55b639905d40 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3823254162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f1ad8003ce0 con 0x55b639905d40 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad8019e10 con 0x55b639905d40 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3823254162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1ad8045050 con 0x55b639905d40 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1ae8bd5740 10 client.15498 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b639980f20 con 0x55b639905d40 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.853+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3823254162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad8019e10 con 0x55b639905d40 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3823254162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1ad802f070 con 0x55b639905d40 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1ae8bd5740 10 client.15498 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1ae8bd5740 10 client.15498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1ae8bd5740 20 client.15498 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639983400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1ae8bd5740 4 client.15498 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b639867d00 con 0x55b639915980 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1adffff640 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639983400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1adffff640 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639983400 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc9a0395d493329 server_cookie=ac23940f770f78be in_seq=0 out_seq=0 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.854+0000 7f1abf7fe640 10 client.15498 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.855+0000 7f1abf7fe640 1 -- 172.21.15.28:0/3823254162 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b639867d00 con 0x55b639915980 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.855+0000 7f1abf7fe640 10 client.15498 handle_command_reply: tid=0 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.855+0000 7f1ae8bd5740 1 client.15498 shutdown 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 msgr2=0x55b63990acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63990acb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8019400 tx=0x7f1ad8007520 comp rx=0 tx=0).stop 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 shutdown_connections 2024-09-15T21:57:29.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b639915980 0x55b639983400 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b639905d40 0x55b63990acb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b6398fb480 0x55b63990e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 --2- 172.21.15.28:0/3823254162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6398fab60 0x55b63990dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 >> 172.21.15.28:0/3823254162 conn(0x55b6398f70f0 msgr2=0x55b6398f6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1abf7fe640 0 client.15498 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 shutdown_connections 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 1 -- 172.21.15.28:0/3823254162 wait complete. 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.856+0000 7f1ae8bd5740 20 client.15498 trim_cache size 0 max 16384 2024-09-15T21:57:29.887 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:29.888 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T21:57:29.888 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "b46da9fc-0a37-4239-b008-62982c567489", 2024-09-15T21:57:29.888 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T21:57:29.888 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1ac80a7960 msgr2=0x7f1ac80a9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1ac80a7960 0x7f1ac80a9e20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1ad801e000 tx=0x7f1ad801f000 comp rx=0 tx=0).stop 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 msgr2=0x7f1ae0198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae0198950 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4006010 tx=0x7f1ad40042f0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 shutdown_connections 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1ac80a7960 0x7f1ac80a9e20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ae0106f10 0x7f1ae019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ae0102790 0x7f1ae0198950 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2623956052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ae0101de0 0x7f1ae0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 >> 172.21.15.28:0/2623956052 conn(0x7f1ae00fd8c0 msgr2=0x7f1ae00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 shutdown_connections 2024-09-15T21:57:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:29.857+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2623956052 wait complete. 2024-09-15T21:57:30.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.003+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f46481a5130 con 0x7f4648101ac0 2024-09-15T21:57:30.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.007+0000 7f463a7fc640 1 -- 172.21.15.28:0/3290365284 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) ==== 133+0+0 (secure 0 0 0) 0x7f463c09ba90 con 0x7f4648101ac0 2024-09-15T21:57:30.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.009+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f46340a3580 msgr2=0x7f46340a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.009+0000 7f464f8ae640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f46340a3580 0x7f46340a5a40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4644000c00 tx=0x7f4644005a20 comp rx=0 tx=0).stop 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 msgr2=0x7f4648198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648198320 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f463c00d950 tx=0x7f463c00de10 comp rx=0 tx=0).stop 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 shutdown_connections 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f46340a3580 0x7f46340a5a40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4648107ad0 0x7f464819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4648102460 0x7f4648198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 --2- 172.21.15.28:0/3290365284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4648101ac0 0x7f4648198320 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 >> 172.21.15.28:0/3290365284 conn(0x7f46480fd820 msgr2=0x7f46480fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 shutdown_connections 2024-09-15T21:57:30.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.010+0000 7f464f8ae640 1 -- 172.21.15.28:0/3290365284 wait complete. 2024-09-15T21:57:30.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-09-15T21:57:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 -- 172.21.15.28:0/261499455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f04100880 msgr2=0x7f2f04100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/261499455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f04100880 0x7f2f04100c60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009900 tx=0x7f2f0002a410 comp rx=0 tx=0).stop 2024-09-15T21:57:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 -- 172.21.15.28:0/261499455 shutdown_connections 2024-09-15T21:57:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/261499455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f0410e430 0x7f2f04110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/261499455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f041011a0 0x7f2f0410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/261499455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f04100880 0x7f2f04100c60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 -- 172.21.15.28:0/261499455 >> 172.21.15.28:0/261499455 conn(0x7f2f040fc6b0 msgr2=0x7f2f040fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:30.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 -- 172.21.15.28:0/261499455 shutdown_connections 2024-09-15T21:57:30.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.135+0000 7f2f0bec9640 1 -- 172.21.15.28:0/261499455 wait complete. 2024-09-15T21:57:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 Processor -- start 2024-09-15T21:57:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 -- start start 2024-09-15T21:57:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f04100880 0x7f2f0419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f041011a0 0x7f2f0419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f0410e430 0x7f2f0419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f04114140 con 0x7f2f0410e430 2024-09-15T21:57:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f04113fc0 con 0x7f2f041011a0 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.136+0000 7f2f0bec9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2f041142c0 con 0x7f2f04100880 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f0410e430 0x7f2f0419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f0410e430 0x7f2f0419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59938/0 (socket says 172.21.15.28:59938) 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 -- 172.21.15.28:0/2514862235 learned_addr learned my addr 172.21.15.28:0/2514862235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f09c65640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f04100880 0x7f2f0419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f09464640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f041011a0 0x7f2f0419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 -- 172.21.15.28:0/2514862235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f04100880 msgr2=0x7f2f0419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f04100880 0x7f2f0419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 -- 172.21.15.28:0/2514862235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f041011a0 msgr2=0x7f2f0419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f041011a0 0x7f2f0419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 -- 172.21.15.28:0/2514862235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0419f5a0 con 0x7f2f0410e430 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0a466640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f0410e430 0x7f2f0419ee70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00eb10 tx=0x7f2efc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2efaffd640 1 -- 172.21.15.28:0/2514862235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2efc004b20 con 0x7f2f0410e430 2024-09-15T21:57:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f0419f890 con 0x7f2f0410e430 2024-09-15T21:57:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2efaffd640 1 -- 172.21.15.28:0/2514862235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2efc004cc0 con 0x7f2f0410e430 2024-09-15T21:57:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2efaffd640 1 -- 172.21.15.28:0/2514862235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2efc010790 con 0x7f2f0410e430 2024-09-15T21:57:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.137+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f041a7120 con 0x7f2f0410e430 2024-09-15T21:57:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.139+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f04100c60 con 0x7f2f0410e430 2024-09-15T21:57:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.140+0000 7f2efaffd640 1 -- 172.21.15.28:0/2514862235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f2efc005020 con 0x7f2f0410e430 2024-09-15T21:57:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.141+0000 7f2efaffd640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ef00a3580 0x7f2ef00a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.141+0000 7f2efaffd640 1 -- 172.21.15.28:0/2514862235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2efc0d0a30 con 0x7f2f0410e430 2024-09-15T21:57:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.141+0000 7f2f09c65640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ef00a3580 0x7f2ef00a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.142+0000 7f2f09c65640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ef00a3580 0x7f2ef00a5a40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00005b50 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.148+0000 7f2efaffd640 1 -- 172.21.15.28:0/2514862235 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2efc09ce60 con 0x7f2f0410e430 2024-09-15T21:57:30.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.352+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f2f0410f7c0 con 0x7f2ef00a3580 2024-09-15T21:57:30.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.418+0000 7f2efaffd640 1 -- 172.21.15.28:0/2514862235 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2f0410f7c0 con 0x7f2ef00a3580 2024-09-15T21:57:30.421 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-09-15T21:57:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.420+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ef00a3580 msgr2=0x7f2ef00a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.420+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ef00a3580 0x7f2ef00a5a40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00005b50 comp rx=0 tx=0).stop 2024-09-15T21:57:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.420+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f0410e430 msgr2=0x7f2f0419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.420+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f0410e430 0x7f2f0419ee70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00eb10 tx=0x7f2efc00c2d0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.420+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 shutdown_connections 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.421+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ef00a3580 0x7f2ef00a5a40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.421+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f0410e430 0x7f2f0419ee70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.421+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f041011a0 0x7f2f0419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.421+0000 7f2f0bec9640 1 --2- 172.21.15.28:0/2514862235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f04100880 0x7f2f0419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.421+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 >> 172.21.15.28:0/2514862235 conn(0x7f2f040fc6b0 msgr2=0x7f2f040feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.421+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 shutdown_connections 2024-09-15T21:57:30.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.421+0000 7f2f0bec9640 1 -- 172.21.15.28:0/2514862235 wait complete. 2024-09-15T21:57:30.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.539+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3618710394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 msgr2=0x7fd9d8100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.539+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3618710394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d8100cb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4009900 tx=0x7fd9d402a410 comp rx=0 tx=0).stop 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.539+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3618710394 shutdown_connections 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.539+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3618710394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d810e480 0x7fd9d8110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.539+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3618710394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9d8101280 0x7fd9d810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.539+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3618710394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d8100cb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.539+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3618710394 >> 172.21.15.28:0/3618710394 conn(0x7fd9d80fc6b0 msgr2=0x7fd9d80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3618710394 shutdown_connections 2024-09-15T21:57:30.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3618710394 wait complete. 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 Processor -- start 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 -- start start 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d8101280 0x7fd9d819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9d810e480 0x7fd9d819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d8114100 con 0x7fd9d81008d0 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d8113f80 con 0x7fd9d8101280 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dfcaf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d8114280 con 0x7fd9d810e480 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dda4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dda4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d819a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54146/0 (socket says 172.21.15.28:54146) 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.540+0000 7fd9dda4b640 1 -- 172.21.15.28:0/3399493778 learned_addr learned my addr 172.21.15.28:0/3399493778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9de24c640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9d810e480 0x7fd9d819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dd24a640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d8101280 0x7fd9d819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dda4b640 1 -- 172.21.15.28:0/3399493778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9d810e480 msgr2=0x7fd9d819eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dda4b640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9d810e480 0x7fd9d819eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dda4b640 1 -- 172.21.15.28:0/3399493778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d8101280 msgr2=0x7fd9d819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dda4b640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d8101280 0x7fd9d819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dda4b640 1 -- 172.21.15.28:0/3399493778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d819f5d0 con 0x7fd9d81008d0 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dda4b640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d819a5e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4006010 tx=0x7fd9d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:30.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9ceffd640 1 -- 172.21.15.28:0/3399493778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d403f030 con 0x7fd9d81008d0 2024-09-15T21:57:30.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9ceffd640 1 -- 172.21.15.28:0/3399493778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9d4003ce0 con 0x7fd9d81008d0 2024-09-15T21:57:30.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d819f860 con 0x7fd9d81008d0 2024-09-15T21:57:30.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9ceffd640 1 -- 172.21.15.28:0/3399493778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4035070 con 0x7fd9d81008d0 2024-09-15T21:57:30.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.541+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d81a7150 con 0x7fd9d81008d0 2024-09-15T21:57:30.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.542+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d8100cb0 con 0x7fd9d81008d0 2024-09-15T21:57:30.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.543+0000 7fd9ceffd640 1 -- 172.21.15.28:0/3399493778 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fd9d4042030 con 0x7fd9d81008d0 2024-09-15T21:57:30.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.544+0000 7fd9ceffd640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9c40a3600 0x7fd9c40a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.544+0000 7fd9ceffd640 1 -- 172.21.15.28:0/3399493778 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd9d40f07d0 con 0x7fd9d81008d0 2024-09-15T21:57:30.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.544+0000 7fd9dd24a640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9c40a3600 0x7fd9c40a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.544+0000 7fd9dd24a640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9c40a3600 0x7fd9c40a5ac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8005fd0 tx=0x7fd9c8005850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:30.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.548+0000 7fd9ceffd640 1 -- 172.21.15.28:0/3399493778 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd9d40bcd90 con 0x7fd9d81008d0 2024-09-15T21:57:30.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3460164046' entity='client.admin' 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2037248027' entity='client.admin' 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3290365284' entity='client.admin' 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3460164046' entity='client.admin' 2024-09-15T21:57:30.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2037248027' entity='client.admin' 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3290365284' entity='client.admin' 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.720 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.752+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fd9d8062630 con 0x7fd9c40a3600 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3460164046' entity='client.admin' 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2037248027' entity='client.admin' 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3290365284' entity='client.admin' 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:57:30.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:57:30.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:57:30.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.817+0000 7fd9ceffd640 1 -- 172.21.15.28:0/3399493778 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fd9d8062630 con 0x7fd9c40a3600 2024-09-15T21:57:30.820 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-09-15T21:57:30.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9c40a3600 msgr2=0x7fd9c40a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9c40a3600 0x7fd9c40a5ac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8005fd0 tx=0x7fd9c8005850 comp rx=0 tx=0).stop 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 msgr2=0x7fd9d819a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d819a5e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4006010 tx=0x7fd9d40042f0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 shutdown_connections 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9c40a3600 0x7fd9c40a5ac0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9d810e480 0x7fd9d819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9d8101280 0x7fd9d819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 --2- 172.21.15.28:0/3399493778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9d81008d0 0x7fd9d819a5e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 >> 172.21.15.28:0/3399493778 conn(0x7fd9d80fc6b0 msgr2=0x7fd9d80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:30.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 shutdown_connections 2024-09-15T21:57:30.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.819+0000 7fd9dfcaf640 1 -- 172.21.15.28:0/3399493778 wait complete. 2024-09-15T21:57:30.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-15T21:57:30.872 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 -- 172.21.15.28:0/1337037274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195410c0f0 msgr2=0x7f195410e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 --2- 172.21.15.28:0/1337037274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195410c0f0 0x7f195410e590 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f194c009ca0 tx=0x7f194c02a950 comp rx=0 tx=0).stop 2024-09-15T21:57:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 -- 172.21.15.28:0/1337037274 shutdown_connections 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 --2- 172.21.15.28:0/1337037274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195410c0f0 0x7f195410e590 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 --2- 172.21.15.28:0/1337037274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1954103640 0x7f195410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 --2- 172.21.15.28:0/1337037274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1954102d20 0x7f1954103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 -- 172.21.15.28:0/1337037274 >> 172.21.15.28:0/1337037274 conn(0x7f19540fc6b0 msgr2=0x7f19540fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.975+0000 7f19596e4640 1 -- 172.21.15.28:0/1337037274 shutdown_connections 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 -- 172.21.15.28:0/1337037274 wait complete. 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 Processor -- start 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 -- start start 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1954102d20 0x7f19541982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1954103640 0x7f1954198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f195410c0f0 0x7f195419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1954111ef0 con 0x7f1954103640 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1954111d70 con 0x7f1954102d20 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f19596e4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1954112070 con 0x7f195410c0f0 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f194bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1954103640 0x7f1954198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f194bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1954103640 0x7f1954198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54176/0 (socket says 172.21.15.28:54176) 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.976+0000 7f194bfff640 1 -- 172.21.15.28:0/3642837240 learned_addr learned my addr 172.21.15.28:0/3642837240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f1952ffd640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1954102d20 0x7f19541982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f19537fe640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f195410c0f0 0x7f195419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f194bfff640 1 -- 172.21.15.28:0/3642837240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f195410c0f0 msgr2=0x7f195419cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f194bfff640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f195410c0f0 0x7f195419cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f194bfff640 1 -- 172.21.15.28:0/3642837240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1954102d20 msgr2=0x7f19541982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f194bfff640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1954102d20 0x7f19541982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f194bfff640 1 -- 172.21.15.28:0/3642837240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195419d2e0 con 0x7f1954103640 2024-09-15T21:57:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f194bfff640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1954103640 0x7f1954198830 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f193c00c890 tx=0x7f193c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f1950ff9640 1 -- 172.21.15.28:0/3642837240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c004220 con 0x7f1954103640 2024-09-15T21:57:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f1950ff9640 1 -- 172.21.15.28:0/3642837240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f193c0043c0 con 0x7f1954103640 2024-09-15T21:57:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f1950ff9640 1 -- 172.21.15.28:0/3642837240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c0049e0 con 0x7f1954103640 2024-09-15T21:57:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195419d5d0 con 0x7f1954103640 2024-09-15T21:57:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.977+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19541a4e60 con 0x7f1954103640 2024-09-15T21:57:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.978+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1954103100 con 0x7f1954103640 2024-09-15T21:57:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.980+0000 7f1950ff9640 1 -- 172.21.15.28:0/3642837240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f193c0079b0 con 0x7f1954103640 2024-09-15T21:57:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.980+0000 7f1950ff9640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19400a3580 0x7f19400a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.980+0000 7f1952ffd640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19400a3580 0x7f19400a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.981+0000 7f1950ff9640 1 -- 172.21.15.28:0/3642837240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f193c0d0300 con 0x7f1954103640 2024-09-15T21:57:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.981+0000 7f1952ffd640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19400a3580 0x7f19400a5a40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1944005fd0 tx=0x7f194400b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:30.983+0000 7f1950ff9640 1 -- 172.21.15.28:0/3642837240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f193c014030 con 0x7f1954103640 2024-09-15T21:57:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.206+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f19540fe370 con 0x7f19400a3580 2024-09-15T21:57:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.213+0000 7f1950ff9640 1 -- 172.21.15.28:0/3642837240 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f19540fe370 con 0x7f19400a3580 2024-09-15T21:57:31.216 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-15T00:00:00", "created": "2024-09-15T21:57:30", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19400a3580 msgr2=0x7f19400a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19400a3580 0x7f19400a5a40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1944005fd0 tx=0x7f194400b390 comp rx=0 tx=0).stop 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1954103640 msgr2=0x7f1954198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1954103640 0x7f1954198830 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f193c00c890 tx=0x7f193c00cd50 comp rx=0 tx=0).stop 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 shutdown_connections 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19400a3580 0x7f19400a5a40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f195410c0f0 0x7f195419cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1954103640 0x7f1954198830 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 --2- 172.21.15.28:0/3642837240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1954102d20 0x7f19541982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 >> 172.21.15.28:0/3642837240 conn(0x7f19540fc6b0 msgr2=0x7f19540fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 shutdown_connections 2024-09-15T21:57:31.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.216+0000 7f19596e4640 1 -- 172.21.15.28:0/3642837240 wait complete. 2024-09-15T21:57:31.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-15T21:57:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 -- 172.21.15.28:0/1995270482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501009b0 msgr2=0x7f2b50100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 --2- 172.21.15.28:0/1995270482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501009b0 0x7f2b50100e10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009930 tx=0x7f2b3802a420 comp rx=0 tx=0).stop 2024-09-15T21:57:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 -- 172.21.15.28:0/1995270482 shutdown_connections 2024-09-15T21:57:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 --2- 172.21.15.28:0/1995270482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b50101350 0x7f2b5010d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 --2- 172.21.15.28:0/1995270482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501009b0 0x7f2b50100e10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 --2- 172.21.15.28:0/1995270482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b501069c0 0x7f2b50106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 -- 172.21.15.28:0/1995270482 >> 172.21.15.28:0/1995270482 conn(0x7f2b50077990 msgr2=0x7f2b500fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 -- 172.21.15.28:0/1995270482 shutdown_connections 2024-09-15T21:57:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.335+0000 7f2b55da1640 1 -- 172.21.15.28:0/1995270482 wait complete. 2024-09-15T21:57:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 Processor -- start 2024-09-15T21:57:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 -- start start 2024-09-15T21:57:31.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b501009b0 0x7f2b500fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b50101350 0x7f2b500fd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501069c0 0x7f2b500fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b50110e00 con 0x7f2b501069c0 2024-09-15T21:57:31.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2b50110c80 con 0x7f2b50101350 2024-09-15T21:57:31.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b55da1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2b50110f80 con 0x7f2b501009b0 2024-09-15T21:57:31.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501069c0 0x7f2b500fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501069c0 0x7f2b500fd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54198/0 (socket says 172.21.15.28:54198) 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b4ffff640 1 -- 172.21.15.28:0/179893444 learned_addr learned my addr 172.21.15.28:0/179893444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.336+0000 7f2b4f7fe640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b501009b0 0x7f2b500fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b46ffd640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b50101350 0x7f2b500fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4ffff640 1 -- 172.21.15.28:0/179893444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b501009b0 msgr2=0x7f2b500fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4ffff640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b501009b0 0x7f2b500fcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4ffff640 1 -- 172.21.15.28:0/179893444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b50101350 msgr2=0x7f2b500fd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4ffff640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b50101350 0x7f2b500fd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4ffff640 1 -- 172.21.15.28:0/179893444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b501a68a0 con 0x7f2b501069c0 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4f7fe640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b501009b0 0x7f2b500fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b46ffd640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b50101350 0x7f2b500fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:31.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4ffff640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501069c0 0x7f2b500fd860 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800eaa0 tx=0x7f2b4800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4d7fa640 1 -- 172.21.15.28:0/179893444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4800cdb0 con 0x7f2b501069c0 2024-09-15T21:57:31.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b501a6b90 con 0x7f2b501069c0 2024-09-15T21:57:31.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4d7fa640 1 -- 172.21.15.28:0/179893444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b48005020 con 0x7f2b501069c0 2024-09-15T21:57:31.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b4d7fa640 1 -- 172.21.15.28:0/179893444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b480116b0 con 0x7f2b501069c0 2024-09-15T21:57:31.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.337+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b501a7090 con 0x7f2b501069c0 2024-09-15T21:57:31.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.338+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b50100e10 con 0x7f2b501069c0 2024-09-15T21:57:31.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.339+0000 7f2b4d7fa640 1 -- 172.21.15.28:0/179893444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f2b48005560 con 0x7f2b501069c0 2024-09-15T21:57:31.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.340+0000 7f2b4d7fa640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2b3c0a3580 0x7f2b3c0a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.340+0000 7f2b4f7fe640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2b3c0a3580 0x7f2b3c0a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.340+0000 7f2b4d7fa640 1 -- 172.21.15.28:0/179893444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2b480d0630 con 0x7f2b501069c0 2024-09-15T21:57:31.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.341+0000 7f2b4f7fe640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2b3c0a3580 0x7f2b3c0a5a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b40005fd0 tx=0x7f2b40005850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.348+0000 7f2b4d7fa640 1 -- 172.21.15.28:0/179893444 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2b48016030 con 0x7f2b501069c0 2024-09-15T21:57:31.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 -- 172.21.15.28:0/3130448890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78100e20 msgr2=0x7f3d78101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/3130448890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78100e20 0x7f3d78101200 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009900 tx=0x7f3d6c02a480 comp rx=0 tx=0).stop 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 -- 172.21.15.28:0/3130448890 shutdown_connections 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/3130448890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d78102ef0 0x7f3d7810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/3130448890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d78102570 0x7f3d781029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/3130448890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78100e20 0x7f3d78101200 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 -- 172.21.15.28:0/3130448890 >> 172.21.15.28:0/3130448890 conn(0x7f3d780fc7d0 msgr2=0x7f3d780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 -- 172.21.15.28:0/3130448890 shutdown_connections 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.380+0000 7f3d7ed80640 1 -- 172.21.15.28:0/3130448890 wait complete. 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 Processor -- start 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 -- start start 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d78100e20 0x7f3d78198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78102570 0x7f3d78198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d78102ef0 0x7f3d7819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d78110f80 con 0x7f3d78102570 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d78110e00 con 0x7f3d78102ef0 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7ed80640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d78111100 con 0x7f3d78100e20 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78102570 0x7f3d78198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78102570 0x7f3d78198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54202/0 (socket says 172.21.15.28:54202) 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 -- 172.21.15.28:0/2978538129 learned_addr learned my addr 172.21.15.28:0/2978538129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d78100e20 0x7f3d78198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d78102ef0 0x7f3d7819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 -- 172.21.15.28:0/2978538129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d78100e20 msgr2=0x7f3d78198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d78100e20 0x7f3d78198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 -- 172.21.15.28:0/2978538129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d78102ef0 msgr2=0x7f3d7819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d78102ef0 0x7f3d7819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.381+0000 7f3d77fff640 1 -- 172.21.15.28:0/2978538129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7819d4d0 con 0x7f3d78102570 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.382+0000 7f3d77fff640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78102570 0x7f3d78198a60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400d770 tx=0x7f3d6400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.382+0000 7f3d75ffb640 1 -- 172.21.15.28:0/2978538129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64017070 con 0x7f3d78102570 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.382+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7819d7c0 con 0x7f3d78102570 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.382+0000 7f3d75ffb640 1 -- 172.21.15.28:0/2978538129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d64005500 con 0x7f3d78102570 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.382+0000 7f3d75ffb640 1 -- 172.21.15.28:0/2978538129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64004e00 con 0x7f3d78102570 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.382+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d781a50a0 con 0x7f3d78102570 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 Processor -- start 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 -- start start 2024-09-15T21:57:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x56348809a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634880002e0 con 0x5634880a5680 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563487fa7ce0 con 0x56348809a4a0 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d81159740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563487ffc2a0 con 0x56348809adc0 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d75ffb640 1 -- 172.21.15.28:0/2978538129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f3d64004fa0 con 0x7f3d78102570 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d7d31d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x56348809a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d7d31d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x56348809a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:44890/0 (socket says 172.21.15.28:44890) 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d7d31d640 1 -- 172.21.15.28:0/3892204180 learned_addr learned my addr 172.21.15.28:0/3892204180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d77fff640 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.385+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d7d31d640 1 -- 172.21.15.28:0/3892204180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 msgr2=0x5634880a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d7d31d640 1 -- 172.21.15.28:0/3892204180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 msgr2=0x5634880a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d7d31d640 1 -- 172.21.15.28:0/3892204180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563487f8c440 con 0x56348809a4a0 2024-09-15T21:57:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x56348809a880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d70010370 tx=0x7f3d70010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4e1ed7fe5e2228 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d62ffd640 1 -- 172.21.15.28:0/3892204180 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70016070 con 0x56348809a4a0 2024-09-15T21:57:31.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d62ffd640 1 -- 172.21.15.28:0/3892204180 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d70004d20 con 0x56348809a4a0 2024-09-15T21:57:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d62ffd640 1 -- 172.21.15.28:0/3892204180 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70005420 con 0x56348809a4a0 2024-09-15T21:57:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.386+0000 7f3d75ffb640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3d680a34d0 0x7f3d680a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3d680a34d0 0x7f3d680a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d75ffb640 1 -- 172.21.15.28:0/2978538129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f3d640cfcb0 con 0x7f3d78102570 2024-09-15T21:57:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 -- 172.21.15.28:0/3892204180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 msgr2=0x56348809a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x56348809a880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d70010370 tx=0x7f3d70010830 comp rx=0 tx=0).stop 2024-09-15T21:57:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 -- 172.21.15.28:0/3892204180 shutdown_connections 2024-09-15T21:57:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 --2- 172.21.15.28:0/3892204180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x56348809a880 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 -- 172.21.15.28:0/3892204180 >> 172.21.15.28:0/3892204180 conn(0x563488096a30 msgr2=0x563488096e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 -- 172.21.15.28:0/3892204180 shutdown_connections 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3d680a34d0 0x7f3d680a5990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009230 tx=0x7f3d6c0058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 -- 172.21.15.28:0/3892204180 wait complete. 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.387+0000 7f3d81159740 1 -- start start 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 1 Processor -- start 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634880a8250 con 0x5634880a5680 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563487fbb050 con 0x56348809a4a0 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5634880a83d0 con 0x56348809adc0 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d81159740 10 client.0 _ll_register_callbacks cb 0x563487ffb160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d7d31d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d7d31d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54226/0 (socket says 172.21.15.28:54226) 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d7d31d640 1 -- 172.21.15.28:0/452988358 learned_addr learned my addr 172.21.15.28:0/452988358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d77fff640 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.388+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d7d31d640 1 -- 172.21.15.28:0/452988358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 msgr2=0x5634880a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d7d31d640 1 -- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 msgr2=0x5634880a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880a4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d7d31d640 1 -- 172.21.15.28:0/452988358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563487d713c0 con 0x5634880a5680 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a2760 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3d70006010 tx=0x7f3d70004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521b362a289680f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d57fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d57fff640 1 -- 172.21.15.28:0/452988358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70016070 con 0x5634880a5680 2024-09-15T21:57:31.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d57fff640 1 -- 172.21.15.28:0/452988358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f3d70003dc0 con 0x5634880a5680 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d70004d20 con 0x5634880a5680 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d57fff640 1 -- 172.21.15.28:0/452988358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3d7002f050 con 0x5634880a5680 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d81159740 10 client.15534 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.389+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5634880b3bc0 con 0x5634880a5680 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d57fff640 1 -- 172.21.15.28:0/452988358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d70004d20 con 0x5634880a5680 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d57fff640 1 -- 172.21.15.28:0/452988358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3d7000f040 con 0x5634880a5680 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d81159740 10 client.15534 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d81159740 10 client.15534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d81159740 20 client.15534 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:31.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d81159740 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634880b5020 0x5634880b7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d81159740 4 client.15534 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56348800d850 con 0x5634880b5020 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.390+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634880b5020 0x5634880b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.391+0000 7f3d57fff640 10 client.15534 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.391+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634880b5020 0x5634880b7410 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39a5235faad7182e server_cookie=7a5c6576a6a2ff28 in_seq=0 out_seq=0 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.391+0000 7f3d57fff640 1 -- 172.21.15.28:0/452988358 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56348800d850 con 0x5634880b5020 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d57fff640 10 client.15534 handle_command_reply: tid=0 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 client.15534 shutdown 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 msgr2=0x5634880a2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a2760 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3d70006010 tx=0x7f3d70004c70 comp rx=0 tx=0).stop 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 shutdown_connections 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634880b5020 0x5634880b7410 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634880a5680 0x5634880a2760 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56348809adc0 0x5634880a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 --2- 172.21.15.28:0/452988358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 >> 172.21.15.28:0/452988358 conn(0x563488096a30 msgr2=0x56348809c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d57fff640 0 client.15534 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 shutdown_connections 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 1 -- 172.21.15.28:0/452988358 wait complete. 2024-09-15T21:57:31.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.392+0000 7f3d81159740 20 client.15534 trim_cache size 0 max 16384 2024-09-15T21:57:31.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.409+0000 7f3d81159740 1 Processor -- start 2024-09-15T21:57:31.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.409+0000 7f3d81159740 1 -- start start 2024-09-15T21:57:31.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d81159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d81159740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880bc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d81159740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x5634880bcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d81159740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781aef20 con 0x56348809adc0 2024-09-15T21:57:31.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d81159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781b0a60 con 0x56348809a4a0 2024-09-15T21:57:31.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d81159740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781af5f0 con 0x5634880a5680 2024-09-15T21:57:31.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880bc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880bc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54228/0 (socket says 172.21.15.28:54228) 2024-09-15T21:57:31.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 -- 172.21.15.28:0/1328679372 learned_addr learned my addr 172.21.15.28:0/1328679372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x5634880bcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 -- 172.21.15.28:0/1328679372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 msgr2=0x5634880bcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x5634880bcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 -- 172.21.15.28:0/1328679372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 msgr2=0x5634880b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 -- 172.21.15.28:0/1328679372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634880b7780 con 0x56348809adc0 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.410+0000 7f3d77fff640 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880bc9b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009610 tx=0x7f3d6409b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8ab2f2c3ace2db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d627fc640 1 -- 172.21.15.28:0/1328679372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64020040 con 0x56348809adc0 2024-09-15T21:57:31.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d627fc640 1 -- 172.21.15.28:0/1328679372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d6409ba90 con 0x56348809adc0 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d627fc640 1 -- 172.21.15.28:0/1328679372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d64098100 con 0x56348809adc0 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 -- 172.21.15.28:0/1328679372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 msgr2=0x5634880bc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880bc9b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009610 tx=0x7f3d6409b690 comp rx=0 tx=0).stop 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 -- 172.21.15.28:0/1328679372 shutdown_connections 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x5634880bcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880bc9b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 --2- 172.21.15.28:0/1328679372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 -- 172.21.15.28:0/1328679372 >> 172.21.15.28:0/1328679372 conn(0x563488096a30 msgr2=0x5634880a1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 -- 172.21.15.28:0/1328679372 shutdown_connections 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.411+0000 7f3d81159740 1 -- 172.21.15.28:0/1328679372 wait complete. 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 -- start start 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 Processor -- start 2024-09-15T21:57:31.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880aad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880ab2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x563488117950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781b0390 con 0x56348809adc0 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781afcc0 con 0x56348809a4a0 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781acf00 con 0x5634880a5680 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.412+0000 7f3d81159740 10 client.0 _ll_register_callbacks cb 0x7f3d70009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880ab2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880ab2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54242/0 (socket says 172.21.15.28:54242) 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 -- 172.21.15.28:0/468901100 learned_addr learned my addr 172.21.15.28:0/468901100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d7d31d640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x563488117950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 -- 172.21.15.28:0/468901100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 msgr2=0x563488117950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x563488117950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 -- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 msgr2=0x5634880aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880aad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 -- 172.21.15.28:0/468901100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563488117f90 con 0x56348809adc0 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d77fff640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880ab2b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d6409b790 tx=0x7f3d64005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c187993ad49e9169 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d57fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d57fff640 1 -- 172.21.15.28:0/468901100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d640b9060 con 0x56348809adc0 2024-09-15T21:57:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d57fff640 1 -- 172.21.15.28:0/468901100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f3d640860b0 con 0x56348809adc0 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d6409ba90 con 0x56348809adc0 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.413+0000 7f3d57fff640 1 -- 172.21.15.28:0/468901100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3d64020050 con 0x56348809adc0 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 10 client.15540 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563488123720 con 0x56348809adc0 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d57fff640 1 -- 172.21.15.28:0/468901100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d6409ba90 con 0x56348809adc0 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d57fff640 1 -- 172.21.15.28:0/468901100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3d640b4be0 con 0x56348809adc0 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 10 client.15540 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 10 client.15540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 20 client.15540 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563488123de0 0x5634881261d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 4 client.15540 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T21:57:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.414+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563488007d60 con 0x563488123de0 2024-09-15T21:57:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563488123de0 0x5634881261d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d7cb1c640 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563488123de0 0x5634881261d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a5bbaefaf47f1f server_cookie=82cf0769c0a572b7 in_seq=0 out_seq=0 2024-09-15T21:57:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d57fff640 10 client.15540 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d57fff640 1 -- 172.21.15.28:0/468901100 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563488007d60 con 0x563488123de0 2024-09-15T21:57:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d57fff640 10 client.15540 handle_command_reply: tid=0 2024-09-15T21:57:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 client.15540 shutdown 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 msgr2=0x5634880ab2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880ab2b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d6409b790 tx=0x7f3d64005bb0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 shutdown_connections 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563488123de0 0x5634881261d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634880a5680 0x563488117950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56348809adc0 0x5634880ab2b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d57fff640 0 client.15540 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 --2- 172.21.15.28:0/468901100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56348809a4a0 0x5634880aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.415+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 >> 172.21.15.28:0/468901100 conn(0x563488096a30 msgr2=0x563488095350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 shutdown_connections 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d81159740 1 -- 172.21.15.28:0/468901100 wait complete. 2024-09-15T21:57:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d81159740 20 client.15540 trim_cache size 0 max 16384 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: "b46da9fc-0a37-4239-b008-62982c567489": { 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "b46da9fc-0a37-4239-b008-62982c567489", 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:57:31.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:57:31.426 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3d680a34d0 msgr2=0x7f3d680a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3d680a34d0 0x7f3d680a5990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009230 tx=0x7f3d6c0058d0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78102570 msgr2=0x7f3d78198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78102570 0x7f3d78198a60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400d770 tx=0x7f3d6400dc30 comp rx=0 tx=0).stop 2024-09-15T21:57:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 shutdown_connections 2024-09-15T21:57:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3d680a34d0 0x7f3d680a5990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d78102ef0 0x7f3d7819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d78102570 0x7f3d78198a60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 --2- 172.21.15.28:0/2978538129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d78100e20 0x7f3d78198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.416+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 >> 172.21.15.28:0/2978538129 conn(0x7f3d780fc7d0 msgr2=0x7f3d7806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.417+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 shutdown_connections 2024-09-15T21:57:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.417+0000 7f3d7ed80640 1 -- 172.21.15.28:0/2978538129 wait complete. 2024-09-15T21:57:31.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.562+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f2b50109fa0 con 0x7f2b3c0a3580 2024-09-15T21:57:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.569+0000 7f2b4d7fa640 1 -- 172.21.15.28:0/179893444 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f2b4000bc40 con 0x7f2b3c0a3580 2024-09-15T21:57:31.572 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2b3c0a3580 msgr2=0x7f2b3c0a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2b3c0a3580 0x7f2b3c0a5a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b40005fd0 tx=0x7f2b40005850 comp rx=0 tx=0).stop 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501069c0 msgr2=0x7f2b500fd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501069c0 0x7f2b500fd860 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800eaa0 tx=0x7f2b4800ef60 comp rx=0 tx=0).stop 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 shutdown_connections 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2b3c0a3580 0x7f2b3c0a5a40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b501069c0 0x7f2b500fd860 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b50101350 0x7f2b500fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 --2- 172.21.15.28:0/179893444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b501009b0 0x7f2b500fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 >> 172.21.15.28:0/179893444 conn(0x7f2b50077990 msgr2=0x7f2b50069ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 shutdown_connections 2024-09-15T21:57:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:31.571+0000 7f2b55da1640 1 -- 172.21.15.28:0/179893444 wait complete. 2024-09-15T21:57:31.583 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:31 smithi028 ceph-mon[30969]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 796 B/s wr, 20 op/s 2024-09-15T21:57:31.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:31 smithi028 ceph-mon[30969]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:31.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:31 smithi028 ceph-mon[30969]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:31.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-09-15T21:57:31.614 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-09-15T21:57:31.624 INFO:teuthology.run_tasks:Running task background_exec... 2024-09-15T21:57:31.633 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T21:57:31.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-09-15T21:57:31.657 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-15T21:57:31.670 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T21:57:31.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T21:57:31.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:31 smithi203 ceph-mon[35546]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 796 B/s wr, 20 op/s 2024-09-15T21:57:31.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:31 smithi203 ceph-mon[35546]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:31.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:31 smithi203 ceph-mon[35546]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:31 smithi174 ceph-mon[35244]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 796 B/s wr, 20 op/s 2024-09-15T21:57:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:31 smithi174 ceph-mon[35244]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:31 smithi174 ceph-mon[35244]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:32.072 INFO:tasks.cephfs.filesystem:scrub status for tag:b46da9fc-0a37-4239-b008-62982c567489 - {'path': '/', 'tag': 'b46da9fc-0a37-4239-b008-62982c567489', 'options': 'recursive,force'} 2024-09-15T21:57:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:32.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:32 smithi028 ceph-mon[30969]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:32.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:32 smithi028 ceph-mon[30969]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 -- 172.21.15.28:0/120848914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a01002f0 msgr2=0x7f95a01006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 --2- 172.21.15.28:0/120848914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a01002f0 0x7f95a01006d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9590009930 tx=0x7f959002a420 comp rx=0 tx=0).stop 2024-09-15T21:57:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 -- 172.21.15.28:0/120848914 shutdown_connections 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 --2- 172.21.15.28:0/120848914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95a0106f50 0x7f95a010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 --2- 172.21.15.28:0/120848914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95a0100ca0 0x7f95a0106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 --2- 172.21.15.28:0/120848914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a01002f0 0x7f95a01006d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 -- 172.21.15.28:0/120848914 >> 172.21.15.28:0/120848914 conn(0x7f95a0077210 msgr2=0x7f95a00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 -- 172.21.15.28:0/120848914 shutdown_connections 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.646+0000 7f95a63d7640 1 -- 172.21.15.28:0/120848914 wait complete. 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 Processor -- start 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 -- start start 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95a01002f0 0x7f95a0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a0100ca0 0x7f95a0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95a0106f50 0x7f95a019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:32.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95a0111fa0 con 0x7f95a0100ca0 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f95a0111e20 con 0x7f95a01002f0 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a63d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f95a0112120 con 0x7f95a0106f50 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f959f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a0100ca0 0x7f95a0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f959f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a0100ca0 0x7f95a0198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54266/0 (socket says 172.21.15.28:54266) 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f959f7fe640 1 -- 172.21.15.28:0/2989736699 learned_addr learned my addr 172.21.15.28:0/2989736699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.647+0000 7f95a4974640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95a0106f50 0x7f95a019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959ffff640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95a01002f0 0x7f95a0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959f7fe640 1 -- 172.21.15.28:0/2989736699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95a0106f50 msgr2=0x7f95a019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959f7fe640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95a0106f50 0x7f95a019cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959f7fe640 1 -- 172.21.15.28:0/2989736699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95a01002f0 msgr2=0x7f95a0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959f7fe640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95a01002f0 0x7f95a0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959f7fe640 1 -- 172.21.15.28:0/2989736699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a019d4b0 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959f7fe640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a0100ca0 0x7f95a0198a40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f959400d740 tx=0x7f959400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959d7fa640 1 -- 172.21.15.28:0/2989736699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9594017070 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a019d7a0 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959d7fa640 1 -- 172.21.15.28:0/2989736699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f959400de50 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f959d7fa640 1 -- 172.21.15.28:0/2989736699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9594004e00 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.648+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a01a5080 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.649+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a01006d0 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.650+0000 7f959d7fa640 1 -- 172.21.15.28:0/2989736699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f9594004fa0 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.651+0000 7f959d7fa640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f95880a34d0 0x7f95880a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.651+0000 7f959ffff640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f95880a34d0 0x7f95880a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.651+0000 7f959d7fa640 1 -- 172.21.15.28:0/2989736699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f95940cfc70 con 0x7f95a0100ca0 2024-09-15T21:57:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.652+0000 7f959ffff640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f95880a34d0 0x7f95880a5990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9590009900 tx=0x7f95900058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:32.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.655+0000 7f959d7fa640 1 -- 172.21.15.28:0/2989736699 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f959409ba50 con 0x7f95a0100ca0 2024-09-15T21:57:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:32 smithi203 ceph-mon[35546]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:32.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:32 smithi203 ceph-mon[35546]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:32.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:32 smithi174 ceph-mon[35244]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:32.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:32 smithi174 ceph-mon[35244]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T21:57:32.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.829+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f95a0062630 con 0x7f95880a34d0 2024-09-15T21:57:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.837+0000 7f959d7fa640 1 -- 172.21.15.28:0/2989736699 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339472 (secure 0 0 0) 0x7f95a0062630 con 0x7f95880a34d0 2024-09-15T21:57:32.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:57:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f95880a34d0 msgr2=0x7f95880a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f95880a34d0 0x7f95880a5990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9590009900 tx=0x7f95900058d0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a0100ca0 msgr2=0x7f95a0198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a0100ca0 0x7f95a0198a40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f959400d740 tx=0x7f959400dc00 comp rx=0 tx=0).stop 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 shutdown_connections 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f95880a34d0 0x7f95880a5990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95a0106f50 0x7f95a019cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95a0100ca0 0x7f95a0198a40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 --2- 172.21.15.28:0/2989736699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95a01002f0 0x7f95a0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 >> 172.21.15.28:0/2989736699 conn(0x7f95a0077210 msgr2=0x7f95a0105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 shutdown_connections 2024-09-15T21:57:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:32.841+0000 7f95a63d7640 1 -- 172.21.15.28:0/2989736699 wait complete. 2024-09-15T21:57:33.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:33 smithi028 ceph-mon[30969]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 53 op/s 2024-09-15T21:57:33.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:33 smithi028 ceph-mon[30969]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:33.529 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-15T21:57:32.203913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1394845,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1746,"num_read_kb":1303,"num_write":718,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1044,"ondisk_log_size":1044,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366112,"kb_used_data":16724,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124575712,"statfs":{"total":1151940427776,"available":1151565529088,"internally_reserved":0,"allocated":17125376,"data_stored":10372253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":201076,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":438,"num_read_kb":480,"num_write":146,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.995270"},"pg_stats":[{"pgid":"2.1f","version":"93'14","reported_seq":170,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.857401+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:57:29.857401+0000","last_peered":"2024-09-15T21:57:29.857401+0000","last_clean":"2024-09-15T21:57:29.857401+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:57:29.857401+0000","last_undegraded":"2024-09-15T21:57:29.857401+0000","last_fullsized":"2024-09-15T21:57:29.857401+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280359+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:57:20.280359+0000","last_peered":"2024-09-15T21:57:20.280359+0000","last_clean":"2024-09-15T21:57:20.280359+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:57:20.280359+0000","last_undegraded":"2024-09-15T21:57:20.280359+0000","last_fullsized":"2024-09-15T21:57:20.280359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395705+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:57:20.395705+0000","last_peered":"2024-09-15T21:57:20.395705+0000","last_clean":"2024-09-15T21:57:20.395705+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:57:20.395705+0000","last_undegraded":"2024-09-15T21:57:20.395705+0000","last_fullsized":"2024-09-15T21:57:20.395705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280756+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:57:20.280756+0000","last_peered":"2024-09-15T21:57:20.280756+0000","last_clean":"2024-09-15T21:57:20.280756+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:57:20.280756+0000","last_undegraded":"2024-09-15T21:57:20.280756+0000","last_fullsized":"2024-09-15T21:57:20.280756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467940+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:57:20.467940+0000","last_peered":"2024-09-15T21:57:20.467940+0000","last_clean":"2024-09-15T21:57:20.467940+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:57:20.467940+0000","last_undegraded":"2024-09-15T21:57:20.467940+0000","last_fullsized":"2024-09-15T21:57:20.467940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190109+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:57:20.190109+0000","last_peered":"2024-09-15T21:57:20.190109+0000","last_clean":"2024-09-15T21:57:20.190109+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:57:20.190109+0000","last_undegraded":"2024-09-15T21:57:20.190109+0000","last_fullsized":"2024-09-15T21:57:20.190109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.570575+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:57:20.570575+0000","last_peered":"2024-09-15T21:57:20.570575+0000","last_clean":"2024-09-15T21:57:20.570575+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:57:20.570575+0000","last_undegraded":"2024-09-15T21:57:20.570575+0000","last_fullsized":"2024-09-15T21:57:20.570575+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463433+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:57:20.463433+0000","last_peered":"2024-09-15T21:57:20.463433+0000","last_clean":"2024-09-15T21:57:20.463433+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:57:20.463433+0000","last_undegraded":"2024-09-15T21:57:20.463433+0000","last_fullsized":"2024-09-15T21:57:20.463433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190387+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:57:20.190387+0000","last_peered":"2024-09-15T21:57:20.190387+0000","last_clean":"2024-09-15T21:57:20.190387+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:57:20.190387+0000","last_undegraded":"2024-09-15T21:57:20.190387+0000","last_fullsized":"2024-09-15T21:57:20.190387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472029+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:57:20.472029+0000","last_peered":"2024-09-15T21:57:20.472029+0000","last_clean":"2024-09-15T21:57:20.472029+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:57:20.472029+0000","last_undegraded":"2024-09-15T21:57:20.472029+0000","last_fullsized":"2024-09-15T21:57:20.472029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188210+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:57:20.188210+0000","last_peered":"2024-09-15T21:57:20.188210+0000","last_clean":"2024-09-15T21:57:20.188210+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:57:20.188210+0000","last_undegraded":"2024-09-15T21:57:20.188210+0000","last_fullsized":"2024-09-15T21:57:20.188210+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190320+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:57:20.190320+0000","last_peered":"2024-09-15T21:57:20.190320+0000","last_clean":"2024-09-15T21:57:20.190320+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:57:20.190320+0000","last_undegraded":"2024-09-15T21:57:20.190320+0000","last_fullsized":"2024-09-15T21:57:20.190320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460655+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:57:20.460655+0000","last_peered":"2024-09-15T21:57:20.460655+0000","last_clean":"2024-09-15T21:57:20.460655+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:57:20.460655+0000","last_undegraded":"2024-09-15T21:57:20.460655+0000","last_fullsized":"2024-09-15T21:57:20.460655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278262+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:57:20.278262+0000","last_peered":"2024-09-15T21:57:20.278262+0000","last_clean":"2024-09-15T21:57:20.278262+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:57:20.278262+0000","last_undegraded":"2024-09-15T21:57:20.278262+0000","last_fullsized":"2024-09-15T21:57:20.278262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190739+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:57:20.190739+0000","last_peered":"2024-09-15T21:57:20.190739+0000","last_clean":"2024-09-15T21:57:20.190739+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:57:20.190739+0000","last_undegraded":"2024-09-15T21:57:20.190739+0000","last_fullsized":"2024-09-15T21:57:20.190739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279728+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:57:20.279728+0000","last_peered":"2024-09-15T21:57:20.279728+0000","last_clean":"2024-09-15T21:57:20.279728+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:57:20.279728+0000","last_undegraded":"2024-09-15T21:57:20.279728+0000","last_fullsized":"2024-09-15T21:57:20.279728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"93'5","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471885+0000","last_change":"2024-09-15T21:56:58.926427+0000","last_active":"2024-09-15T21:57:20.471885+0000","last_peered":"2024-09-15T21:57:20.471885+0000","last_clean":"2024-09-15T21:57:20.471885+0000","last_became_active":"2024-09-15T21:56:58.922755+0000","last_became_peered":"2024-09-15T21:56:58.922755+0000","last_unstale":"2024-09-15T21:57:20.471885+0000","last_undegraded":"2024-09-15T21:57:20.471885+0000","last_fullsized":"2024-09-15T21:57:20.471885+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:22.033354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:57:20.190660+0000","last_peered":"2024-09-15T21:57:20.190660+0000","last_clean":"2024-09-15T21:57:20.190660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:57:20.190660+0000","last_undegraded":"2024-09-15T21:57:20.190660+0000","last_fullsized":"2024-09-15T21:57:20.190660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471869+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:57:20.471869+0000","last_peered":"2024-09-15T21:57:20.471869+0000","last_clean":"2024-09-15T21:57:20.471869+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:57:20.471869+0000","last_undegraded":"2024-09-15T21:57:20.471869+0000","last_fullsized":"2024-09-15T21:57:20.471869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279173+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:57:20.279173+0000","last_peered":"2024-09-15T21:57:20.279173+0000","last_clean":"2024-09-15T21:57:20.279173+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:57:20.279173+0000","last_undegraded":"2024-09-15T21:57:20.279173+0000","last_fullsized":"2024-09-15T21:57:20.279173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276125+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:57:20.276125+0000","last_peered":"2024-09-15T21:57:20.276125+0000","last_clean":"2024-09-15T21:57:20.276125+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:57:20.276125+0000","last_undegraded":"2024-09-15T21:57:20.276125+0000","last_fullsized":"2024-09-15T21:57:20.276125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278075+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:57:20.278075+0000","last_peered":"2024-09-15T21:57:20.278075+0000","last_clean":"2024-09-15T21:57:20.278075+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:57:20.278075+0000","last_undegraded":"2024-09-15T21:57:20.278075+0000","last_fullsized":"2024-09-15T21:57:20.278075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462853+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:57:20.462853+0000","last_peered":"2024-09-15T21:57:20.462853+0000","last_clean":"2024-09-15T21:57:20.462853+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:57:20.462853+0000","last_undegraded":"2024-09-15T21:57:20.462853+0000","last_fullsized":"2024-09-15T21:57:20.462853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190763+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:57:20.190763+0000","last_peered":"2024-09-15T21:57:20.190763+0000","last_clean":"2024-09-15T21:57:20.190763+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:57:20.190763+0000","last_undegraded":"2024-09-15T21:57:20.190763+0000","last_fullsized":"2024-09-15T21:57:20.190763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467562+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:57:20.467562+0000","last_peered":"2024-09-15T21:57:20.467562+0000","last_clean":"2024-09-15T21:57:20.467562+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:57:20.467562+0000","last_undegraded":"2024-09-15T21:57:20.467562+0000","last_fullsized":"2024-09-15T21:57:20.467562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190932+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:57:20.190932+0000","last_peered":"2024-09-15T21:57:20.190932+0000","last_clean":"2024-09-15T21:57:20.190932+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:57:20.190932+0000","last_undegraded":"2024-09-15T21:57:20.190932+0000","last_fullsized":"2024-09-15T21:57:20.190932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395988+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:57:20.395988+0000","last_peered":"2024-09-15T21:57:20.395988+0000","last_clean":"2024-09-15T21:57:20.395988+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:57:20.395988+0000","last_undegraded":"2024-09-15T21:57:20.395988+0000","last_fullsized":"2024-09-15T21:57:20.395988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280166+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:57:20.280166+0000","last_peered":"2024-09-15T21:57:20.280166+0000","last_clean":"2024-09-15T21:57:20.280166+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:57:20.280166+0000","last_undegraded":"2024-09-15T21:57:20.280166+0000","last_fullsized":"2024-09-15T21:57:20.280166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'103","reported_seq":230,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.815967+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:57:30.815967+0000","last_peered":"2024-09-15T21:57:30.815967+0000","last_clean":"2024-09-15T21:57:30.815967+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:57:30.815967+0000","last_undegraded":"2024-09-15T21:57:30.815967+0000","last_fullsized":"2024-09-15T21:57:30.815967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":292,"num_read_kb":259,"num_write":187,"num_write_kb":1288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.277996+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:57:20.277996+0000","last_peered":"2024-09-15T21:57:20.277996+0000","last_clean":"2024-09-15T21:57:20.277996+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:57:20.277996+0000","last_undegraded":"2024-09-15T21:57:20.277996+0000","last_fullsized":"2024-09-15T21:57:20.277996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186161+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:57:20.186161+0000","last_peered":"2024-09-15T21:57:20.186161+0000","last_clean":"2024-09-15T21:57:20.186161+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:57:20.186161+0000","last_undegraded":"2024-09-15T21:57:20.186161+0000","last_fullsized":"2024-09-15T21:57:20.186161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278630+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:57:20.278630+0000","last_peered":"2024-09-15T21:57:20.278630+0000","last_clean":"2024-09-15T21:57:20.278630+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:57:20.278630+0000","last_undegraded":"2024-09-15T21:57:20.278630+0000","last_fullsized":"2024-09-15T21:57:20.278630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472019+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:57:20.472019+0000","last_peered":"2024-09-15T21:57:20.472019+0000","last_clean":"2024-09-15T21:57:20.472019+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:57:20.472019+0000","last_undegraded":"2024-09-15T21:57:20.472019+0000","last_fullsized":"2024-09-15T21:57:20.472019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285785+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:57:20.285785+0000","last_peered":"2024-09-15T21:57:20.285785+0000","last_clean":"2024-09-15T21:57:20.285785+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:57:20.285785+0000","last_undegraded":"2024-09-15T21:57:20.285785+0000","last_fullsized":"2024-09-15T21:57:20.285785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276329+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:57:20.276329+0000","last_peered":"2024-09-15T21:57:20.276329+0000","last_clean":"2024-09-15T21:57:20.276329+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:57:20.276329+0000","last_undegraded":"2024-09-15T21:57:20.276329+0000","last_fullsized":"2024-09-15T21:57:20.276329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278087+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:57:20.278087+0000","last_peered":"2024-09-15T21:57:20.278087+0000","last_clean":"2024-09-15T21:57:20.278087+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:57:20.278087+0000","last_undegraded":"2024-09-15T21:57:20.278087+0000","last_fullsized":"2024-09-15T21:57:20.278087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285570+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:57:20.285570+0000","last_peered":"2024-09-15T21:57:20.285570+0000","last_clean":"2024-09-15T21:57:20.285570+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:57:20.285570+0000","last_undegraded":"2024-09-15T21:57:20.285570+0000","last_fullsized":"2024-09-15T21:57:20.285570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.723341+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:57:29.723341+0000","last_peered":"2024-09-15T21:57:29.723341+0000","last_clean":"2024-09-15T21:57:29.723341+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:57:29.723341+0000","last_undegraded":"2024-09-15T21:57:29.723341+0000","last_fullsized":"2024-09-15T21:57:29.723341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185976+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:57:20.185976+0000","last_peered":"2024-09-15T21:57:20.185976+0000","last_clean":"2024-09-15T21:57:20.185976+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:57:20.185976+0000","last_undegraded":"2024-09-15T21:57:20.185976+0000","last_fullsized":"2024-09-15T21:57:20.185976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187498+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:57:20.187498+0000","last_peered":"2024-09-15T21:57:20.187498+0000","last_clean":"2024-09-15T21:57:20.187498+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:57:20.187498+0000","last_undegraded":"2024-09-15T21:57:20.187498+0000","last_fullsized":"2024-09-15T21:57:20.187498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.731968+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:57:29.731968+0000","last_peered":"2024-09-15T21:57:29.731968+0000","last_clean":"2024-09-15T21:57:29.731968+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:57:29.731968+0000","last_undegraded":"2024-09-15T21:57:29.731968+0000","last_fullsized":"2024-09-15T21:57:29.731968+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186270+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:57:20.186270+0000","last_peered":"2024-09-15T21:57:20.186270+0000","last_clean":"2024-09-15T21:57:20.186270+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:57:20.186270+0000","last_undegraded":"2024-09-15T21:57:20.186270+0000","last_fullsized":"2024-09-15T21:57:20.186270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185590+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:57:20.185590+0000","last_peered":"2024-09-15T21:57:20.185590+0000","last_clean":"2024-09-15T21:57:20.185590+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:57:20.185590+0000","last_undegraded":"2024-09-15T21:57:20.185590+0000","last_fullsized":"2024-09-15T21:57:20.185590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460124+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:57:20.460124+0000","last_peered":"2024-09-15T21:57:20.460124+0000","last_clean":"2024-09-15T21:57:20.460124+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:57:20.460124+0000","last_undegraded":"2024-09-15T21:57:20.460124+0000","last_fullsized":"2024-09-15T21:57:20.460124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188088+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:57:20.188088+0000","last_peered":"2024-09-15T21:57:20.188088+0000","last_clean":"2024-09-15T21:57:20.188088+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:57:20.188088+0000","last_undegraded":"2024-09-15T21:57:20.188088+0000","last_fullsized":"2024-09-15T21:57:20.188088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279569+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:57:20.279569+0000","last_peered":"2024-09-15T21:57:20.279569+0000","last_clean":"2024-09-15T21:57:20.279569+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:57:20.279569+0000","last_undegraded":"2024-09-15T21:57:20.279569+0000","last_fullsized":"2024-09-15T21:57:20.279569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190561+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:57:20.190561+0000","last_peered":"2024-09-15T21:57:20.190561+0000","last_clean":"2024-09-15T21:57:20.190561+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:57:20.190561+0000","last_undegraded":"2024-09-15T21:57:20.190561+0000","last_fullsized":"2024-09-15T21:57:20.190561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"93'148","reported_seq":409,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:22.291343+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:57:22.291343+0000","last_peered":"2024-09-15T21:57:22.291343+0000","last_clean":"2024-09-15T21:57:22.291343+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:57:22.291343+0000","last_undegraded":"2024-09-15T21:57:22.291343+0000","last_fullsized":"2024-09-15T21:57:22.291343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460868+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:57:20.460868+0000","last_peered":"2024-09-15T21:57:20.460868+0000","last_clean":"2024-09-15T21:57:20.460868+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:57:20.460868+0000","last_undegraded":"2024-09-15T21:57:20.460868+0000","last_fullsized":"2024-09-15T21:57:20.460868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461076+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:57:20.461076+0000","last_peered":"2024-09-15T21:57:20.461076+0000","last_clean":"2024-09-15T21:57:20.461076+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:57:20.461076+0000","last_undegraded":"2024-09-15T21:57:20.461076+0000","last_fullsized":"2024-09-15T21:57:20.461076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":167,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.731921+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:57:29.731921+0000","last_peered":"2024-09-15T21:57:29.731921+0000","last_clean":"2024-09-15T21:57:29.731921+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:57:29.731921+0000","last_undegraded":"2024-09-15T21:57:29.731921+0000","last_fullsized":"2024-09-15T21:57:29.731921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462918+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:57:20.462918+0000","last_peered":"2024-09-15T21:57:20.462918+0000","last_clean":"2024-09-15T21:57:20.462918+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:57:20.462918+0000","last_undegraded":"2024-09-15T21:57:20.462918+0000","last_fullsized":"2024-09-15T21:57:20.462918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460824+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:57:20.460824+0000","last_peered":"2024-09-15T21:57:20.460824+0000","last_clean":"2024-09-15T21:57:20.460824+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:57:20.460824+0000","last_undegraded":"2024-09-15T21:57:20.460824+0000","last_fullsized":"2024-09-15T21:57:20.460824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'67","reported_seq":137,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.810429+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:57:30.810429+0000","last_peered":"2024-09-15T21:57:30.810429+0000","last_clean":"2024-09-15T21:57:30.810429+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:57:30.810429+0000","last_undegraded":"2024-09-15T21:57:30.810429+0000","last_fullsized":"2024-09-15T21:57:30.810429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":31,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466957+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:57:20.466957+0000","last_peered":"2024-09-15T21:57:20.466957+0000","last_clean":"2024-09-15T21:57:20.466957+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:57:20.466957+0000","last_undegraded":"2024-09-15T21:57:20.466957+0000","last_fullsized":"2024-09-15T21:57:20.466957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460987+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:57:20.460987+0000","last_peered":"2024-09-15T21:57:20.460987+0000","last_clean":"2024-09-15T21:57:20.460987+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:57:20.460987+0000","last_undegraded":"2024-09-15T21:57:20.460987+0000","last_fullsized":"2024-09-15T21:57:20.460987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186150+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:57:20.186150+0000","last_peered":"2024-09-15T21:57:20.186150+0000","last_clean":"2024-09-15T21:57:20.186150+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:57:20.186150+0000","last_undegraded":"2024-09-15T21:57:20.186150+0000","last_fullsized":"2024-09-15T21:57:20.186150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187377+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:57:20.187377+0000","last_peered":"2024-09-15T21:57:20.187377+0000","last_clean":"2024-09-15T21:57:20.187377+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:57:20.187377+0000","last_undegraded":"2024-09-15T21:57:20.187377+0000","last_fullsized":"2024-09-15T21:57:20.187377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472329+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:57:20.472329+0000","last_peered":"2024-09-15T21:57:20.472329+0000","last_clean":"2024-09-15T21:57:20.472329+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:57:20.472329+0000","last_undegraded":"2024-09-15T21:57:20.472329+0000","last_fullsized":"2024-09-15T21:57:20.472329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278620+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:57:20.278620+0000","last_peered":"2024-09-15T21:57:20.278620+0000","last_clean":"2024-09-15T21:57:20.278620+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:57:20.278620+0000","last_undegraded":"2024-09-15T21:57:20.278620+0000","last_fullsized":"2024-09-15T21:57:20.278620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187448+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:57:20.187448+0000","last_peered":"2024-09-15T21:57:20.187448+0000","last_clean":"2024-09-15T21:57:20.187448+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:57:20.187448+0000","last_undegraded":"2024-09-15T21:57:20.187448+0000","last_fullsized":"2024-09-15T21:57:20.187448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187552+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:57:20.187552+0000","last_peered":"2024-09-15T21:57:20.187552+0000","last_clean":"2024-09-15T21:57:20.187552+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:57:20.187552+0000","last_undegraded":"2024-09-15T21:57:20.187552+0000","last_fullsized":"2024-09-15T21:57:20.187552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278144+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:57:20.278144+0000","last_peered":"2024-09-15T21:57:20.278144+0000","last_clean":"2024-09-15T21:57:20.278144+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:57:20.278144+0000","last_undegraded":"2024-09-15T21:57:20.278144+0000","last_fullsized":"2024-09-15T21:57:20.278144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186097+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:57:20.186097+0000","last_peered":"2024-09-15T21:57:20.186097+0000","last_clean":"2024-09-15T21:57:20.186097+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:57:20.186097+0000","last_undegraded":"2024-09-15T21:57:20.186097+0000","last_fullsized":"2024-09-15T21:57:20.186097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460653+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:57:20.460653+0000","last_peered":"2024-09-15T21:57:20.460653+0000","last_clean":"2024-09-15T21:57:20.460653+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:57:20.460653+0000","last_undegraded":"2024-09-15T21:57:20.460653+0000","last_fullsized":"2024-09-15T21:57:20.460653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":164,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.732525+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:57:29.732525+0000","last_peered":"2024-09-15T21:57:29.732525+0000","last_clean":"2024-09-15T21:57:29.732525+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:57:29.732525+0000","last_undegraded":"2024-09-15T21:57:29.732525+0000","last_fullsized":"2024-09-15T21:57:29.732525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276283+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:57:20.276283+0000","last_peered":"2024-09-15T21:57:20.276283+0000","last_clean":"2024-09-15T21:57:20.276283+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:57:20.276283+0000","last_undegraded":"2024-09-15T21:57:20.276283+0000","last_fullsized":"2024-09-15T21:57:20.276283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460781+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:57:20.460781+0000","last_peered":"2024-09-15T21:57:20.460781+0000","last_clean":"2024-09-15T21:57:20.460781+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:57:20.460781+0000","last_undegraded":"2024-09-15T21:57:20.460781+0000","last_fullsized":"2024-09-15T21:57:20.460781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187760+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:57:20.187760+0000","last_peered":"2024-09-15T21:57:20.187760+0000","last_clean":"2024-09-15T21:57:20.187760+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:57:20.187760+0000","last_undegraded":"2024-09-15T21:57:20.187760+0000","last_fullsized":"2024-09-15T21:57:20.187760+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472098+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:57:20.472098+0000","last_peered":"2024-09-15T21:57:20.472098+0000","last_clean":"2024-09-15T21:57:20.472098+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:57:20.472098+0000","last_undegraded":"2024-09-15T21:57:20.472098+0000","last_fullsized":"2024-09-15T21:57:20.472098+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185779+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:57:20.185779+0000","last_peered":"2024-09-15T21:57:20.185779+0000","last_clean":"2024-09-15T21:57:20.185779+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:57:20.185779+0000","last_undegraded":"2024-09-15T21:57:20.185779+0000","last_fullsized":"2024-09-15T21:57:20.185779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395654+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:57:20.395654+0000","last_peered":"2024-09-15T21:57:20.395654+0000","last_clean":"2024-09-15T21:57:20.395654+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:57:20.395654+0000","last_undegraded":"2024-09-15T21:57:20.395654+0000","last_fullsized":"2024-09-15T21:57:20.395654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188109+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:57:20.188109+0000","last_peered":"2024-09-15T21:57:20.188109+0000","last_clean":"2024-09-15T21:57:20.188109+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:57:20.188109+0000","last_undegraded":"2024-09-15T21:57:20.188109+0000","last_fullsized":"2024-09-15T21:57:20.188109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278543+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:57:20.278543+0000","last_peered":"2024-09-15T21:57:20.278543+0000","last_clean":"2024-09-15T21:57:20.278543+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:57:20.278543+0000","last_undegraded":"2024-09-15T21:57:20.278543+0000","last_fullsized":"2024-09-15T21:57:20.278543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'45","reported_seq":328,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.725042+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:57:30.725042+0000","last_peered":"2024-09-15T21:57:30.725042+0000","last_clean":"2024-09-15T21:57:30.725042+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:57:30.725042+0000","last_undegraded":"2024-09-15T21:57:30.725042+0000","last_fullsized":"2024-09-15T21:57:30.725042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72639,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":192,"num_write":47,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275890+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:57:20.275890+0000","last_peered":"2024-09-15T21:57:20.275890+0000","last_clean":"2024-09-15T21:57:20.275890+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:57:20.275890+0000","last_undegraded":"2024-09-15T21:57:20.275890+0000","last_fullsized":"2024-09-15T21:57:20.275890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463470+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:57:20.463470+0000","last_peered":"2024-09-15T21:57:20.463470+0000","last_clean":"2024-09-15T21:57:20.463470+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:57:20.463470+0000","last_undegraded":"2024-09-15T21:57:20.463470+0000","last_fullsized":"2024-09-15T21:57:20.463470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188450+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:57:20.188450+0000","last_peered":"2024-09-15T21:57:20.188450+0000","last_clean":"2024-09-15T21:57:20.188450+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:57:20.188450+0000","last_undegraded":"2024-09-15T21:57:20.188450+0000","last_fullsized":"2024-09-15T21:57:20.188450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190380+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:57:20.190380+0000","last_peered":"2024-09-15T21:57:20.190380+0000","last_clean":"2024-09-15T21:57:20.190380+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:57:20.190380+0000","last_undegraded":"2024-09-15T21:57:20.190380+0000","last_fullsized":"2024-09-15T21:57:20.190380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467982+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:57:20.467982+0000","last_peered":"2024-09-15T21:57:20.467982+0000","last_clean":"2024-09-15T21:57:20.467982+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:57:20.467982+0000","last_undegraded":"2024-09-15T21:57:20.467982+0000","last_fullsized":"2024-09-15T21:57:20.467982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461005+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:57:20.461005+0000","last_peered":"2024-09-15T21:57:20.461005+0000","last_clean":"2024-09-15T21:57:20.461005+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:57:20.461005+0000","last_undegraded":"2024-09-15T21:57:20.461005+0000","last_fullsized":"2024-09-15T21:57:20.461005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275613+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:57:20.275613+0000","last_peered":"2024-09-15T21:57:20.275613+0000","last_clean":"2024-09-15T21:57:20.275613+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:57:20.275613+0000","last_undegraded":"2024-09-15T21:57:20.275613+0000","last_fullsized":"2024-09-15T21:57:20.275613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463309+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:57:20.463309+0000","last_peered":"2024-09-15T21:57:20.463309+0000","last_clean":"2024-09-15T21:57:20.463309+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:57:20.463309+0000","last_undegraded":"2024-09-15T21:57:20.463309+0000","last_fullsized":"2024-09-15T21:57:20.463309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'8","reported_seq":275,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.724567+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:57:30.724567+0000","last_peered":"2024-09-15T21:57:30.724567+0000","last_clean":"2024-09-15T21:57:30.724567+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:57:30.724567+0000","last_undegraded":"2024-09-15T21:57:30.724567+0000","last_fullsized":"2024-09-15T21:57:30.724567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278776+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:57:20.278776+0000","last_peered":"2024-09-15T21:57:20.278776+0000","last_clean":"2024-09-15T21:57:20.278776+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:57:20.278776+0000","last_undegraded":"2024-09-15T21:57:20.278776+0000","last_fullsized":"2024-09-15T21:57:20.278776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467332+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:57:20.467332+0000","last_peered":"2024-09-15T21:57:20.467332+0000","last_clean":"2024-09-15T21:57:20.467332+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:57:20.467332+0000","last_undegraded":"2024-09-15T21:57:20.467332+0000","last_fullsized":"2024-09-15T21:57:20.467332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275877+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:57:20.275877+0000","last_peered":"2024-09-15T21:57:20.275877+0000","last_clean":"2024-09-15T21:57:20.275877+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:57:20.275877+0000","last_undegraded":"2024-09-15T21:57:20.275877+0000","last_fullsized":"2024-09-15T21:57:20.275877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187675+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:57:20.187675+0000","last_peered":"2024-09-15T21:57:20.187675+0000","last_clean":"2024-09-15T21:57:20.187675+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:57:20.187675+0000","last_undegraded":"2024-09-15T21:57:20.187675+0000","last_fullsized":"2024-09-15T21:57:20.187675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472373+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:57:20.472373+0000","last_peered":"2024-09-15T21:57:20.472373+0000","last_clean":"2024-09-15T21:57:20.472373+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:57:20.472373+0000","last_undegraded":"2024-09-15T21:57:20.472373+0000","last_fullsized":"2024-09-15T21:57:20.472373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278428+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:57:20.278428+0000","last_peered":"2024-09-15T21:57:20.278428+0000","last_clean":"2024-09-15T21:57:20.278428+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:57:20.278428+0000","last_undegraded":"2024-09-15T21:57:20.278428+0000","last_fullsized":"2024-09-15T21:57:20.278428+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185660+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:57:20.185660+0000","last_peered":"2024-09-15T21:57:20.185660+0000","last_clean":"2024-09-15T21:57:20.185660+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:57:20.185660+0000","last_undegraded":"2024-09-15T21:57:20.185660+0000","last_fullsized":"2024-09-15T21:57:20.185660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285631+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:57:20.285631+0000","last_peered":"2024-09-15T21:57:20.285631+0000","last_clean":"2024-09-15T21:57:20.285631+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:57:20.285631+0000","last_undegraded":"2024-09-15T21:57:20.285631+0000","last_fullsized":"2024-09-15T21:57:20.285631+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187959+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:57:20.187959+0000","last_peered":"2024-09-15T21:57:20.187959+0000","last_clean":"2024-09-15T21:57:20.187959+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:57:20.187959+0000","last_undegraded":"2024-09-15T21:57:20.187959+0000","last_fullsized":"2024-09-15T21:57:20.187959+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396115+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:57:20.396115+0000","last_peered":"2024-09-15T21:57:20.396115+0000","last_clean":"2024-09-15T21:57:20.396115+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:57:20.396115+0000","last_undegraded":"2024-09-15T21:57:20.396115+0000","last_fullsized":"2024-09-15T21:57:20.396115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'10","reported_seq":171,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.723820+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:57:29.723820+0000","last_peered":"2024-09-15T21:57:29.723820+0000","last_clean":"2024-09-15T21:57:29.723820+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:57:29.723820+0000","last_undegraded":"2024-09-15T21:57:29.723820+0000","last_fullsized":"2024-09-15T21:57:29.723820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":7,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278237+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:57:20.278237+0000","last_peered":"2024-09-15T21:57:20.278237+0000","last_clean":"2024-09-15T21:57:20.278237+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:57:20.278237+0000","last_undegraded":"2024-09-15T21:57:20.278237+0000","last_fullsized":"2024-09-15T21:57:20.278237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396197+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:57:20.396197+0000","last_peered":"2024-09-15T21:57:20.396197+0000","last_clean":"2024-09-15T21:57:20.396197+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:57:20.396197+0000","last_undegraded":"2024-09-15T21:57:20.396197+0000","last_fullsized":"2024-09-15T21:57:20.396197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471807+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:57:20.471807+0000","last_peered":"2024-09-15T21:57:20.471807+0000","last_clean":"2024-09-15T21:57:20.471807+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:57:20.471807+0000","last_undegraded":"2024-09-15T21:57:20.471807+0000","last_fullsized":"2024-09-15T21:57:20.471807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396260+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:57:20.396260+0000","last_peered":"2024-09-15T21:57:20.396260+0000","last_clean":"2024-09-15T21:57:20.396260+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:57:20.396260+0000","last_undegraded":"2024-09-15T21:57:20.396260+0000","last_fullsized":"2024-09-15T21:57:20.396260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395966+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:57:20.395966+0000","last_peered":"2024-09-15T21:57:20.395966+0000","last_clean":"2024-09-15T21:57:20.395966+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:57:20.395966+0000","last_undegraded":"2024-09-15T21:57:20.395966+0000","last_fullsized":"2024-09-15T21:57:20.395966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466952+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:57:20.466952+0000","last_peered":"2024-09-15T21:57:20.466952+0000","last_clean":"2024-09-15T21:57:20.466952+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:57:20.466952+0000","last_undegraded":"2024-09-15T21:57:20.466952+0000","last_fullsized":"2024-09-15T21:57:20.466952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186090+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:57:20.186090+0000","last_peered":"2024-09-15T21:57:20.186090+0000","last_clean":"2024-09-15T21:57:20.186090+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:57:20.186090+0000","last_undegraded":"2024-09-15T21:57:20.186090+0000","last_fullsized":"2024-09-15T21:57:20.186090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472240+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:57:20.472240+0000","last_peered":"2024-09-15T21:57:20.472240+0000","last_clean":"2024-09-15T21:57:20.472240+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:57:20.472240+0000","last_undegraded":"2024-09-15T21:57:20.472240+0000","last_fullsized":"2024-09-15T21:57:20.472240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396338+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:57:20.396338+0000","last_peered":"2024-09-15T21:57:20.396338+0000","last_clean":"2024-09-15T21:57:20.396338+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:57:20.396338+0000","last_undegraded":"2024-09-15T21:57:20.396338+0000","last_fullsized":"2024-09-15T21:57:20.396338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280054+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:57:20.280054+0000","last_peered":"2024-09-15T21:57:20.280054+0000","last_clean":"2024-09-15T21:57:20.280054+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:57:20.280054+0000","last_undegraded":"2024-09-15T21:57:20.280054+0000","last_fullsized":"2024-09-15T21:57:20.280054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187475+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:57:20.187475+0000","last_peered":"2024-09-15T21:57:20.187475+0000","last_clean":"2024-09-15T21:57:20.187475+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:57:20.187475+0000","last_undegraded":"2024-09-15T21:57:20.187475+0000","last_fullsized":"2024-09-15T21:57:20.187475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278973+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:57:20.278973+0000","last_peered":"2024-09-15T21:57:20.278973+0000","last_clean":"2024-09-15T21:57:20.278973+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:57:20.278973+0000","last_undegraded":"2024-09-15T21:57:20.278973+0000","last_fullsized":"2024-09-15T21:57:20.278973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463016+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:57:20.463016+0000","last_peered":"2024-09-15T21:57:20.463016+0000","last_clean":"2024-09-15T21:57:20.463016+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:57:20.463016+0000","last_undegraded":"2024-09-15T21:57:20.463016+0000","last_fullsized":"2024-09-15T21:57:20.463016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186044+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:57:20.186044+0000","last_peered":"2024-09-15T21:57:20.186044+0000","last_clean":"2024-09-15T21:57:20.186044+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:57:20.186044+0000","last_undegraded":"2024-09-15T21:57:20.186044+0000","last_fullsized":"2024-09-15T21:57:20.186044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472203+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:57:20.472203+0000","last_peered":"2024-09-15T21:57:20.472203+0000","last_clean":"2024-09-15T21:57:20.472203+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:57:20.472203+0000","last_undegraded":"2024-09-15T21:57:20.472203+0000","last_fullsized":"2024-09-15T21:57:20.472203+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396319+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:57:20.396319+0000","last_peered":"2024-09-15T21:57:20.396319+0000","last_clean":"2024-09-15T21:57:20.396319+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:57:20.396319+0000","last_undegraded":"2024-09-15T21:57:20.396319+0000","last_fullsized":"2024-09-15T21:57:20.396319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276457+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:57:20.276457+0000","last_peered":"2024-09-15T21:57:20.276457+0000","last_clean":"2024-09-15T21:57:20.276457+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:57:20.276457+0000","last_undegraded":"2024-09-15T21:57:20.276457+0000","last_fullsized":"2024-09-15T21:57:20.276457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472556+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:57:20.472556+0000","last_peered":"2024-09-15T21:57:20.472556+0000","last_clean":"2024-09-15T21:57:20.472556+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:57:20.472556+0000","last_undegraded":"2024-09-15T21:57:20.472556+0000","last_fullsized":"2024-09-15T21:57:20.472556+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462857+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:57:20.462857+0000","last_peered":"2024-09-15T21:57:20.462857+0000","last_clean":"2024-09-15T21:57:20.462857+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:57:20.462857+0000","last_undegraded":"2024-09-15T21:57:20.462857+0000","last_fullsized":"2024-09-15T21:57:20.462857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466723+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:57:20.466723+0000","last_peered":"2024-09-15T21:57:20.466723+0000","last_clean":"2024-09-15T21:57:20.466723+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:57:20.466723+0000","last_undegraded":"2024-09-15T21:57:20.466723+0000","last_fullsized":"2024-09-15T21:57:20.466723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466886+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:57:20.466886+0000","last_peered":"2024-09-15T21:57:20.466886+0000","last_clean":"2024-09-15T21:57:20.466886+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:57:20.466886+0000","last_undegraded":"2024-09-15T21:57:20.466886+0000","last_fullsized":"2024-09-15T21:57:20.466886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186220+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:57:20.186220+0000","last_peered":"2024-09-15T21:57:20.186220+0000","last_clean":"2024-09-15T21:57:20.186220+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:57:20.186220+0000","last_undegraded":"2024-09-15T21:57:20.186220+0000","last_fullsized":"2024-09-15T21:57:20.186220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276015+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:57:20.276015+0000","last_peered":"2024-09-15T21:57:20.276015+0000","last_clean":"2024-09-15T21:57:20.276015+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:57:20.276015+0000","last_undegraded":"2024-09-15T21:57:20.276015+0000","last_fullsized":"2024-09-15T21:57:20.276015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461118+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:57:20.461118+0000","last_peered":"2024-09-15T21:57:20.461118+0000","last_clean":"2024-09-15T21:57:20.461118+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:57:20.461118+0000","last_undegraded":"2024-09-15T21:57:20.461118+0000","last_fullsized":"2024-09-15T21:57:20.461118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.286112+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:57:20.286112+0000","last_peered":"2024-09-15T21:57:20.286112+0000","last_clean":"2024-09-15T21:57:20.286112+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:57:20.286112+0000","last_undegraded":"2024-09-15T21:57:20.286112+0000","last_fullsized":"2024-09-15T21:57:20.286112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463385+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:57:20.463385+0000","last_peered":"2024-09-15T21:57:20.463385+0000","last_clean":"2024-09-15T21:57:20.463385+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:57:20.463385+0000","last_undegraded":"2024-09-15T21:57:20.463385+0000","last_fullsized":"2024-09-15T21:57:20.463385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278238+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:57:20.278238+0000","last_peered":"2024-09-15T21:57:20.278238+0000","last_clean":"2024-09-15T21:57:20.278238+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:57:20.278238+0000","last_undegraded":"2024-09-15T21:57:20.278238+0000","last_fullsized":"2024-09-15T21:57:20.278238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460583+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:57:20.460583+0000","last_peered":"2024-09-15T21:57:20.460583+0000","last_clean":"2024-09-15T21:57:20.460583+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:57:20.460583+0000","last_undegraded":"2024-09-15T21:57:20.460583+0000","last_fullsized":"2024-09-15T21:57:20.460583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285573+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:57:20.285573+0000","last_peered":"2024-09-15T21:57:20.285573+0000","last_clean":"2024-09-15T21:57:20.285573+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:57:20.285573+0000","last_undegraded":"2024-09-15T21:57:20.285573+0000","last_fullsized":"2024-09-15T21:57:20.285573+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278231+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:57:20.278231+0000","last_peered":"2024-09-15T21:57:20.278231+0000","last_clean":"2024-09-15T21:57:20.278231+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:57:20.278231+0000","last_undegraded":"2024-09-15T21:57:20.278231+0000","last_fullsized":"2024-09-15T21:57:20.278231+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190742+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:57:20.190742+0000","last_peered":"2024-09-15T21:57:20.190742+0000","last_clean":"2024-09-15T21:57:20.190742+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:57:20.190742+0000","last_undegraded":"2024-09-15T21:57:20.190742+0000","last_fullsized":"2024-09-15T21:57:20.190742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467111+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:57:20.467111+0000","last_peered":"2024-09-15T21:57:20.467111+0000","last_clean":"2024-09-15T21:57:20.467111+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:57:20.467111+0000","last_undegraded":"2024-09-15T21:57:20.467111+0000","last_fullsized":"2024-09-15T21:57:20.467111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467430+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:57:20.467430+0000","last_peered":"2024-09-15T21:57:20.467430+0000","last_clean":"2024-09-15T21:57:20.467430+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:57:20.467430+0000","last_undegraded":"2024-09-15T21:57:20.467430+0000","last_fullsized":"2024-09-15T21:57:20.467430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472483+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:57:20.472483+0000","last_peered":"2024-09-15T21:57:20.472483+0000","last_clean":"2024-09-15T21:57:20.472483+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:57:20.472483+0000","last_undegraded":"2024-09-15T21:57:20.472483+0000","last_fullsized":"2024-09-15T21:57:20.472483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":804243,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1509,"num_read_kb":973,"num_write":515,"num_write_kb":2400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2777088,"data_stored":2412729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514540,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1081344,"data_stored":536154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776176,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29096,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716056,"statfs":{"total":95995035648,"available":95965241344,"internally_reserved":0,"allocated":2121728,"data_stored":1531310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939703,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1490944,"data_stored":923695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168635,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36796,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708356,"statfs":{"total":95995035648,"available":95957356544,"internally_reserved":0,"allocated":1486848,"data_stored":927587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397569,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37004,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708148,"statfs":{"total":95995035648,"available":95957143552,"internally_reserved":0,"allocated":1695744,"data_stored":1133866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561092,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":530472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790027,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29132,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716020,"statfs":{"total":95995035648,"available":95965204480,"internally_reserved":0,"allocated":2035712,"data_stored":1456501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953552,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":534939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1617920,"data_stored":1054603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1175552,"data_stored":606523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":604141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":532462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":77166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":753664,"data_stored":721878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":332680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":403393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:57:33.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T21:57:33.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:33 smithi203 ceph-mon[35546]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 53 op/s 2024-09-15T21:57:33.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:33 smithi203 ceph-mon[35546]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:33.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:33 smithi174 ceph-mon[35244]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 53 op/s 2024-09-15T21:57:33.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:33 smithi174 ceph-mon[35244]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:33.794 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.308+0000 7f67b6615640 1 -- 172.21.15.28:0/513112193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0101420 msgr2=0x7f67b010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.308+0000 7f67b6615640 1 --2- 172.21.15.28:0/513112193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0101420 0x7f67b010d950 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009cd0 tx=0x7f67a802a8e0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.308+0000 7f67b6615640 1 -- 172.21.15.28:0/513112193 shutdown_connections 2024-09-15T21:57:34.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.308+0000 7f67b6615640 1 --2- 172.21.15.28:0/513112193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0101420 0x7f67b010d950 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.308+0000 7f67b6615640 1 --2- 172.21.15.28:0/513112193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67b0100a80 0x7f67b0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.308+0000 7f67b6615640 1 --2- 172.21.15.28:0/513112193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67b0106a80 0x7f67b0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.308+0000 7f67b6615640 1 -- 172.21.15.28:0/513112193 >> 172.21.15.28:0/513112193 conn(0x7f67b00fc7a0 msgr2=0x7f67b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.309+0000 7f67b6615640 1 -- 172.21.15.28:0/513112193 shutdown_connections 2024-09-15T21:57:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.309+0000 7f67b6615640 1 -- 172.21.15.28:0/513112193 wait complete. 2024-09-15T21:57:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.309+0000 7f67b6615640 1 Processor -- start 2024-09-15T21:57:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.309+0000 7f67b6615640 1 -- start start 2024-09-15T21:57:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67b6615640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0100a80 0x7f67b0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67b6615640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67b0101420 0x7f67b01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67b6615640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67b0106a80 0x7f67b019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67b6615640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67b0110e80 con 0x7f67b0100a80 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67b6615640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67b0110d00 con 0x7f67b0106a80 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67b6615640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67b0111000 con 0x7f67b0101420 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0100a80 0x7f67b0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0100a80 0x7f67b0198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54276/0 (socket says 172.21.15.28:54276) 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 -- 172.21.15.28:0/1389742125 learned_addr learned my addr 172.21.15.28:0/1389742125 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67a7fff640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67b0101420 0x7f67b01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67b4bb2640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67b0106a80 0x7f67b019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 -- 172.21.15.28:0/1389742125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67b0101420 msgr2=0x7f67b01989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67b0101420 0x7f67b01989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 -- 172.21.15.28:0/1389742125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67b0106a80 msgr2=0x7f67b019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67b0106a80 0x7f67b019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.310+0000 7f67affff640 1 -- 172.21.15.28:0/1389742125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b019d410 con 0x7f67b0100a80 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.311+0000 7f67affff640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0100a80 0x7f67b0198460 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f67a000e9f0 tx=0x7f67a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.311+0000 7f67adffb640 1 -- 172.21.15.28:0/1389742125 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a000cd80 con 0x7f67b0100a80 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.311+0000 7f67adffb640 1 -- 172.21.15.28:0/1389742125 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67a0010910 con 0x7f67b0100a80 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.311+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b019d700 con 0x7f67b0100a80 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.311+0000 7f67adffb640 1 -- 172.21.15.28:0/1389742125 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0018990 con 0x7f67b0100a80 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.311+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b0102580 con 0x7f67b0100a80 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.312+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b0100ee0 con 0x7f67b0100a80 2024-09-15T21:57:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.313+0000 7f67adffb640 1 -- 172.21.15.28:0/1389742125 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f67a0010ac0 con 0x7f67b0100a80 2024-09-15T21:57:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.314+0000 7f67adffb640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f679c0a3550 0x7f679c0a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.314+0000 7f67a7fff640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f679c0a3550 0x7f679c0a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.314+0000 7f67adffb640 1 -- 172.21.15.28:0/1389742125 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f67a00d0470 con 0x7f67b0100a80 2024-09-15T21:57:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.314+0000 7f67a7fff640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f679c0a3550 0x7f679c0a5a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f67b00fea90 tx=0x7f679800b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:34.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.318+0000 7f67adffb640 1 -- 172.21.15.28:0/1389742125 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f67a009c2f0 con 0x7f67b0100a80 2024-09-15T21:57:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.489+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f67b006af40 con 0x7f679c0a3550 2024-09-15T21:57:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.497+0000 7f67adffb640 1 -- 172.21.15.28:0/1389742125 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339473 (secure 0 0 0) 0x7f67b006af40 con 0x7f679c0a3550 2024-09-15T21:57:34.502 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:57:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f679c0a3550 msgr2=0x7f679c0a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f679c0a3550 0x7f679c0a5a10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f67b00fea90 tx=0x7f679800b390 comp rx=0 tx=0).stop 2024-09-15T21:57:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0100a80 msgr2=0x7f67b0198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0100a80 0x7f67b0198460 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f67a000e9f0 tx=0x7f67a000eeb0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 shutdown_connections 2024-09-15T21:57:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f679c0a3550 0x7f679c0a5a10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67b0106a80 0x7f67b019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67b0101420 0x7f67b01989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 --2- 172.21.15.28:0/1389742125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67b0100a80 0x7f67b0198460 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 >> 172.21.15.28:0/1389742125 conn(0x7f67b00fc7a0 msgr2=0x7f67b006a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 shutdown_connections 2024-09-15T21:57:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:34.501+0000 7f67b6615640 1 -- 172.21.15.28:0/1389742125 wait complete. 2024-09-15T21:57:35.190 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-09-15T21:57:34.204598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1394845,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1746,"num_read_kb":1303,"num_write":718,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1044,"ondisk_log_size":1044,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366112,"kb_used_data":16724,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124575712,"statfs":{"total":1151940427776,"available":1151565529088,"internally_reserved":0,"allocated":17125376,"data_stored":10372253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":201076,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":438,"num_read_kb":480,"num_write":146,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.1f","version":"93'14","reported_seq":170,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.857401+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:57:29.857401+0000","last_peered":"2024-09-15T21:57:29.857401+0000","last_clean":"2024-09-15T21:57:29.857401+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:57:29.857401+0000","last_undegraded":"2024-09-15T21:57:29.857401+0000","last_fullsized":"2024-09-15T21:57:29.857401+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280359+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:57:20.280359+0000","last_peered":"2024-09-15T21:57:20.280359+0000","last_clean":"2024-09-15T21:57:20.280359+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:57:20.280359+0000","last_undegraded":"2024-09-15T21:57:20.280359+0000","last_fullsized":"2024-09-15T21:57:20.280359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395705+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:57:20.395705+0000","last_peered":"2024-09-15T21:57:20.395705+0000","last_clean":"2024-09-15T21:57:20.395705+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:57:20.395705+0000","last_undegraded":"2024-09-15T21:57:20.395705+0000","last_fullsized":"2024-09-15T21:57:20.395705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280756+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:57:20.280756+0000","last_peered":"2024-09-15T21:57:20.280756+0000","last_clean":"2024-09-15T21:57:20.280756+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:57:20.280756+0000","last_undegraded":"2024-09-15T21:57:20.280756+0000","last_fullsized":"2024-09-15T21:57:20.280756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467940+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:57:20.467940+0000","last_peered":"2024-09-15T21:57:20.467940+0000","last_clean":"2024-09-15T21:57:20.467940+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:57:20.467940+0000","last_undegraded":"2024-09-15T21:57:20.467940+0000","last_fullsized":"2024-09-15T21:57:20.467940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190109+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:57:20.190109+0000","last_peered":"2024-09-15T21:57:20.190109+0000","last_clean":"2024-09-15T21:57:20.190109+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:57:20.190109+0000","last_undegraded":"2024-09-15T21:57:20.190109+0000","last_fullsized":"2024-09-15T21:57:20.190109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.570575+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:57:20.570575+0000","last_peered":"2024-09-15T21:57:20.570575+0000","last_clean":"2024-09-15T21:57:20.570575+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:57:20.570575+0000","last_undegraded":"2024-09-15T21:57:20.570575+0000","last_fullsized":"2024-09-15T21:57:20.570575+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463433+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:57:20.463433+0000","last_peered":"2024-09-15T21:57:20.463433+0000","last_clean":"2024-09-15T21:57:20.463433+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:57:20.463433+0000","last_undegraded":"2024-09-15T21:57:20.463433+0000","last_fullsized":"2024-09-15T21:57:20.463433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190387+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:57:20.190387+0000","last_peered":"2024-09-15T21:57:20.190387+0000","last_clean":"2024-09-15T21:57:20.190387+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:57:20.190387+0000","last_undegraded":"2024-09-15T21:57:20.190387+0000","last_fullsized":"2024-09-15T21:57:20.190387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472029+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:57:20.472029+0000","last_peered":"2024-09-15T21:57:20.472029+0000","last_clean":"2024-09-15T21:57:20.472029+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:57:20.472029+0000","last_undegraded":"2024-09-15T21:57:20.472029+0000","last_fullsized":"2024-09-15T21:57:20.472029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188210+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:57:20.188210+0000","last_peered":"2024-09-15T21:57:20.188210+0000","last_clean":"2024-09-15T21:57:20.188210+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:57:20.188210+0000","last_undegraded":"2024-09-15T21:57:20.188210+0000","last_fullsized":"2024-09-15T21:57:20.188210+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190320+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:57:20.190320+0000","last_peered":"2024-09-15T21:57:20.190320+0000","last_clean":"2024-09-15T21:57:20.190320+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:57:20.190320+0000","last_undegraded":"2024-09-15T21:57:20.190320+0000","last_fullsized":"2024-09-15T21:57:20.190320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460655+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:57:20.460655+0000","last_peered":"2024-09-15T21:57:20.460655+0000","last_clean":"2024-09-15T21:57:20.460655+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:57:20.460655+0000","last_undegraded":"2024-09-15T21:57:20.460655+0000","last_fullsized":"2024-09-15T21:57:20.460655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278262+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:57:20.278262+0000","last_peered":"2024-09-15T21:57:20.278262+0000","last_clean":"2024-09-15T21:57:20.278262+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:57:20.278262+0000","last_undegraded":"2024-09-15T21:57:20.278262+0000","last_fullsized":"2024-09-15T21:57:20.278262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190739+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:57:20.190739+0000","last_peered":"2024-09-15T21:57:20.190739+0000","last_clean":"2024-09-15T21:57:20.190739+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:57:20.190739+0000","last_undegraded":"2024-09-15T21:57:20.190739+0000","last_fullsized":"2024-09-15T21:57:20.190739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279728+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:57:20.279728+0000","last_peered":"2024-09-15T21:57:20.279728+0000","last_clean":"2024-09-15T21:57:20.279728+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:57:20.279728+0000","last_undegraded":"2024-09-15T21:57:20.279728+0000","last_fullsized":"2024-09-15T21:57:20.279728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"93'5","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471885+0000","last_change":"2024-09-15T21:56:58.926427+0000","last_active":"2024-09-15T21:57:20.471885+0000","last_peered":"2024-09-15T21:57:20.471885+0000","last_clean":"2024-09-15T21:57:20.471885+0000","last_became_active":"2024-09-15T21:56:58.922755+0000","last_became_peered":"2024-09-15T21:56:58.922755+0000","last_unstale":"2024-09-15T21:57:20.471885+0000","last_undegraded":"2024-09-15T21:57:20.471885+0000","last_fullsized":"2024-09-15T21:57:20.471885+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:22.033354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:57:20.190660+0000","last_peered":"2024-09-15T21:57:20.190660+0000","last_clean":"2024-09-15T21:57:20.190660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:57:20.190660+0000","last_undegraded":"2024-09-15T21:57:20.190660+0000","last_fullsized":"2024-09-15T21:57:20.190660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471869+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:57:20.471869+0000","last_peered":"2024-09-15T21:57:20.471869+0000","last_clean":"2024-09-15T21:57:20.471869+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:57:20.471869+0000","last_undegraded":"2024-09-15T21:57:20.471869+0000","last_fullsized":"2024-09-15T21:57:20.471869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279173+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:57:20.279173+0000","last_peered":"2024-09-15T21:57:20.279173+0000","last_clean":"2024-09-15T21:57:20.279173+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:57:20.279173+0000","last_undegraded":"2024-09-15T21:57:20.279173+0000","last_fullsized":"2024-09-15T21:57:20.279173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276125+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:57:20.276125+0000","last_peered":"2024-09-15T21:57:20.276125+0000","last_clean":"2024-09-15T21:57:20.276125+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:57:20.276125+0000","last_undegraded":"2024-09-15T21:57:20.276125+0000","last_fullsized":"2024-09-15T21:57:20.276125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278075+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:57:20.278075+0000","last_peered":"2024-09-15T21:57:20.278075+0000","last_clean":"2024-09-15T21:57:20.278075+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:57:20.278075+0000","last_undegraded":"2024-09-15T21:57:20.278075+0000","last_fullsized":"2024-09-15T21:57:20.278075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462853+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:57:20.462853+0000","last_peered":"2024-09-15T21:57:20.462853+0000","last_clean":"2024-09-15T21:57:20.462853+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:57:20.462853+0000","last_undegraded":"2024-09-15T21:57:20.462853+0000","last_fullsized":"2024-09-15T21:57:20.462853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190763+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:57:20.190763+0000","last_peered":"2024-09-15T21:57:20.190763+0000","last_clean":"2024-09-15T21:57:20.190763+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:57:20.190763+0000","last_undegraded":"2024-09-15T21:57:20.190763+0000","last_fullsized":"2024-09-15T21:57:20.190763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467562+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:57:20.467562+0000","last_peered":"2024-09-15T21:57:20.467562+0000","last_clean":"2024-09-15T21:57:20.467562+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:57:20.467562+0000","last_undegraded":"2024-09-15T21:57:20.467562+0000","last_fullsized":"2024-09-15T21:57:20.467562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190932+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:57:20.190932+0000","last_peered":"2024-09-15T21:57:20.190932+0000","last_clean":"2024-09-15T21:57:20.190932+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:57:20.190932+0000","last_undegraded":"2024-09-15T21:57:20.190932+0000","last_fullsized":"2024-09-15T21:57:20.190932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395988+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:57:20.395988+0000","last_peered":"2024-09-15T21:57:20.395988+0000","last_clean":"2024-09-15T21:57:20.395988+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:57:20.395988+0000","last_undegraded":"2024-09-15T21:57:20.395988+0000","last_fullsized":"2024-09-15T21:57:20.395988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280166+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:57:20.280166+0000","last_peered":"2024-09-15T21:57:20.280166+0000","last_clean":"2024-09-15T21:57:20.280166+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:57:20.280166+0000","last_undegraded":"2024-09-15T21:57:20.280166+0000","last_fullsized":"2024-09-15T21:57:20.280166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'103","reported_seq":230,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.815967+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:57:30.815967+0000","last_peered":"2024-09-15T21:57:30.815967+0000","last_clean":"2024-09-15T21:57:30.815967+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:57:30.815967+0000","last_undegraded":"2024-09-15T21:57:30.815967+0000","last_fullsized":"2024-09-15T21:57:30.815967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":292,"num_read_kb":259,"num_write":187,"num_write_kb":1288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.277996+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:57:20.277996+0000","last_peered":"2024-09-15T21:57:20.277996+0000","last_clean":"2024-09-15T21:57:20.277996+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:57:20.277996+0000","last_undegraded":"2024-09-15T21:57:20.277996+0000","last_fullsized":"2024-09-15T21:57:20.277996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186161+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:57:20.186161+0000","last_peered":"2024-09-15T21:57:20.186161+0000","last_clean":"2024-09-15T21:57:20.186161+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:57:20.186161+0000","last_undegraded":"2024-09-15T21:57:20.186161+0000","last_fullsized":"2024-09-15T21:57:20.186161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278630+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:57:20.278630+0000","last_peered":"2024-09-15T21:57:20.278630+0000","last_clean":"2024-09-15T21:57:20.278630+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:57:20.278630+0000","last_undegraded":"2024-09-15T21:57:20.278630+0000","last_fullsized":"2024-09-15T21:57:20.278630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472019+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:57:20.472019+0000","last_peered":"2024-09-15T21:57:20.472019+0000","last_clean":"2024-09-15T21:57:20.472019+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:57:20.472019+0000","last_undegraded":"2024-09-15T21:57:20.472019+0000","last_fullsized":"2024-09-15T21:57:20.472019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285785+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:57:20.285785+0000","last_peered":"2024-09-15T21:57:20.285785+0000","last_clean":"2024-09-15T21:57:20.285785+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:57:20.285785+0000","last_undegraded":"2024-09-15T21:57:20.285785+0000","last_fullsized":"2024-09-15T21:57:20.285785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276329+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:57:20.276329+0000","last_peered":"2024-09-15T21:57:20.276329+0000","last_clean":"2024-09-15T21:57:20.276329+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:57:20.276329+0000","last_undegraded":"2024-09-15T21:57:20.276329+0000","last_fullsized":"2024-09-15T21:57:20.276329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278087+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:57:20.278087+0000","last_peered":"2024-09-15T21:57:20.278087+0000","last_clean":"2024-09-15T21:57:20.278087+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:57:20.278087+0000","last_undegraded":"2024-09-15T21:57:20.278087+0000","last_fullsized":"2024-09-15T21:57:20.278087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285570+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:57:20.285570+0000","last_peered":"2024-09-15T21:57:20.285570+0000","last_clean":"2024-09-15T21:57:20.285570+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:57:20.285570+0000","last_undegraded":"2024-09-15T21:57:20.285570+0000","last_fullsized":"2024-09-15T21:57:20.285570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.723341+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:57:29.723341+0000","last_peered":"2024-09-15T21:57:29.723341+0000","last_clean":"2024-09-15T21:57:29.723341+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:57:29.723341+0000","last_undegraded":"2024-09-15T21:57:29.723341+0000","last_fullsized":"2024-09-15T21:57:29.723341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185976+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:57:20.185976+0000","last_peered":"2024-09-15T21:57:20.185976+0000","last_clean":"2024-09-15T21:57:20.185976+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:57:20.185976+0000","last_undegraded":"2024-09-15T21:57:20.185976+0000","last_fullsized":"2024-09-15T21:57:20.185976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187498+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:57:20.187498+0000","last_peered":"2024-09-15T21:57:20.187498+0000","last_clean":"2024-09-15T21:57:20.187498+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:57:20.187498+0000","last_undegraded":"2024-09-15T21:57:20.187498+0000","last_fullsized":"2024-09-15T21:57:20.187498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.731968+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:57:29.731968+0000","last_peered":"2024-09-15T21:57:29.731968+0000","last_clean":"2024-09-15T21:57:29.731968+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:57:29.731968+0000","last_undegraded":"2024-09-15T21:57:29.731968+0000","last_fullsized":"2024-09-15T21:57:29.731968+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186270+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:57:20.186270+0000","last_peered":"2024-09-15T21:57:20.186270+0000","last_clean":"2024-09-15T21:57:20.186270+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:57:20.186270+0000","last_undegraded":"2024-09-15T21:57:20.186270+0000","last_fullsized":"2024-09-15T21:57:20.186270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185590+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:57:20.185590+0000","last_peered":"2024-09-15T21:57:20.185590+0000","last_clean":"2024-09-15T21:57:20.185590+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:57:20.185590+0000","last_undegraded":"2024-09-15T21:57:20.185590+0000","last_fullsized":"2024-09-15T21:57:20.185590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460124+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:57:20.460124+0000","last_peered":"2024-09-15T21:57:20.460124+0000","last_clean":"2024-09-15T21:57:20.460124+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:57:20.460124+0000","last_undegraded":"2024-09-15T21:57:20.460124+0000","last_fullsized":"2024-09-15T21:57:20.460124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188088+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:57:20.188088+0000","last_peered":"2024-09-15T21:57:20.188088+0000","last_clean":"2024-09-15T21:57:20.188088+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:57:20.188088+0000","last_undegraded":"2024-09-15T21:57:20.188088+0000","last_fullsized":"2024-09-15T21:57:20.188088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279569+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:57:20.279569+0000","last_peered":"2024-09-15T21:57:20.279569+0000","last_clean":"2024-09-15T21:57:20.279569+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:57:20.279569+0000","last_undegraded":"2024-09-15T21:57:20.279569+0000","last_fullsized":"2024-09-15T21:57:20.279569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190561+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:57:20.190561+0000","last_peered":"2024-09-15T21:57:20.190561+0000","last_clean":"2024-09-15T21:57:20.190561+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:57:20.190561+0000","last_undegraded":"2024-09-15T21:57:20.190561+0000","last_fullsized":"2024-09-15T21:57:20.190561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"93'148","reported_seq":409,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:22.291343+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:57:22.291343+0000","last_peered":"2024-09-15T21:57:22.291343+0000","last_clean":"2024-09-15T21:57:22.291343+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:57:22.291343+0000","last_undegraded":"2024-09-15T21:57:22.291343+0000","last_fullsized":"2024-09-15T21:57:22.291343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460868+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:57:20.460868+0000","last_peered":"2024-09-15T21:57:20.460868+0000","last_clean":"2024-09-15T21:57:20.460868+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:57:20.460868+0000","last_undegraded":"2024-09-15T21:57:20.460868+0000","last_fullsized":"2024-09-15T21:57:20.460868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461076+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:57:20.461076+0000","last_peered":"2024-09-15T21:57:20.461076+0000","last_clean":"2024-09-15T21:57:20.461076+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:57:20.461076+0000","last_undegraded":"2024-09-15T21:57:20.461076+0000","last_fullsized":"2024-09-15T21:57:20.461076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":167,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.731921+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:57:29.731921+0000","last_peered":"2024-09-15T21:57:29.731921+0000","last_clean":"2024-09-15T21:57:29.731921+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:57:29.731921+0000","last_undegraded":"2024-09-15T21:57:29.731921+0000","last_fullsized":"2024-09-15T21:57:29.731921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462918+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:57:20.462918+0000","last_peered":"2024-09-15T21:57:20.462918+0000","last_clean":"2024-09-15T21:57:20.462918+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:57:20.462918+0000","last_undegraded":"2024-09-15T21:57:20.462918+0000","last_fullsized":"2024-09-15T21:57:20.462918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460824+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:57:20.460824+0000","last_peered":"2024-09-15T21:57:20.460824+0000","last_clean":"2024-09-15T21:57:20.460824+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:57:20.460824+0000","last_undegraded":"2024-09-15T21:57:20.460824+0000","last_fullsized":"2024-09-15T21:57:20.460824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'67","reported_seq":137,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.810429+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:57:30.810429+0000","last_peered":"2024-09-15T21:57:30.810429+0000","last_clean":"2024-09-15T21:57:30.810429+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:57:30.810429+0000","last_undegraded":"2024-09-15T21:57:30.810429+0000","last_fullsized":"2024-09-15T21:57:30.810429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":31,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466957+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:57:20.466957+0000","last_peered":"2024-09-15T21:57:20.466957+0000","last_clean":"2024-09-15T21:57:20.466957+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:57:20.466957+0000","last_undegraded":"2024-09-15T21:57:20.466957+0000","last_fullsized":"2024-09-15T21:57:20.466957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460987+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:57:20.460987+0000","last_peered":"2024-09-15T21:57:20.460987+0000","last_clean":"2024-09-15T21:57:20.460987+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:57:20.460987+0000","last_undegraded":"2024-09-15T21:57:20.460987+0000","last_fullsized":"2024-09-15T21:57:20.460987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186150+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:57:20.186150+0000","last_peered":"2024-09-15T21:57:20.186150+0000","last_clean":"2024-09-15T21:57:20.186150+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:57:20.186150+0000","last_undegraded":"2024-09-15T21:57:20.186150+0000","last_fullsized":"2024-09-15T21:57:20.186150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187377+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:57:20.187377+0000","last_peered":"2024-09-15T21:57:20.187377+0000","last_clean":"2024-09-15T21:57:20.187377+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:57:20.187377+0000","last_undegraded":"2024-09-15T21:57:20.187377+0000","last_fullsized":"2024-09-15T21:57:20.187377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472329+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:57:20.472329+0000","last_peered":"2024-09-15T21:57:20.472329+0000","last_clean":"2024-09-15T21:57:20.472329+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:57:20.472329+0000","last_undegraded":"2024-09-15T21:57:20.472329+0000","last_fullsized":"2024-09-15T21:57:20.472329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278620+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:57:20.278620+0000","last_peered":"2024-09-15T21:57:20.278620+0000","last_clean":"2024-09-15T21:57:20.278620+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:57:20.278620+0000","last_undegraded":"2024-09-15T21:57:20.278620+0000","last_fullsized":"2024-09-15T21:57:20.278620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187448+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:57:20.187448+0000","last_peered":"2024-09-15T21:57:20.187448+0000","last_clean":"2024-09-15T21:57:20.187448+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:57:20.187448+0000","last_undegraded":"2024-09-15T21:57:20.187448+0000","last_fullsized":"2024-09-15T21:57:20.187448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187552+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:57:20.187552+0000","last_peered":"2024-09-15T21:57:20.187552+0000","last_clean":"2024-09-15T21:57:20.187552+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:57:20.187552+0000","last_undegraded":"2024-09-15T21:57:20.187552+0000","last_fullsized":"2024-09-15T21:57:20.187552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278144+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:57:20.278144+0000","last_peered":"2024-09-15T21:57:20.278144+0000","last_clean":"2024-09-15T21:57:20.278144+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:57:20.278144+0000","last_undegraded":"2024-09-15T21:57:20.278144+0000","last_fullsized":"2024-09-15T21:57:20.278144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186097+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:57:20.186097+0000","last_peered":"2024-09-15T21:57:20.186097+0000","last_clean":"2024-09-15T21:57:20.186097+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:57:20.186097+0000","last_undegraded":"2024-09-15T21:57:20.186097+0000","last_fullsized":"2024-09-15T21:57:20.186097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460653+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:57:20.460653+0000","last_peered":"2024-09-15T21:57:20.460653+0000","last_clean":"2024-09-15T21:57:20.460653+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:57:20.460653+0000","last_undegraded":"2024-09-15T21:57:20.460653+0000","last_fullsized":"2024-09-15T21:57:20.460653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":164,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.732525+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:57:29.732525+0000","last_peered":"2024-09-15T21:57:29.732525+0000","last_clean":"2024-09-15T21:57:29.732525+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:57:29.732525+0000","last_undegraded":"2024-09-15T21:57:29.732525+0000","last_fullsized":"2024-09-15T21:57:29.732525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276283+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:57:20.276283+0000","last_peered":"2024-09-15T21:57:20.276283+0000","last_clean":"2024-09-15T21:57:20.276283+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:57:20.276283+0000","last_undegraded":"2024-09-15T21:57:20.276283+0000","last_fullsized":"2024-09-15T21:57:20.276283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460781+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:57:20.460781+0000","last_peered":"2024-09-15T21:57:20.460781+0000","last_clean":"2024-09-15T21:57:20.460781+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:57:20.460781+0000","last_undegraded":"2024-09-15T21:57:20.460781+0000","last_fullsized":"2024-09-15T21:57:20.460781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187760+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:57:20.187760+0000","last_peered":"2024-09-15T21:57:20.187760+0000","last_clean":"2024-09-15T21:57:20.187760+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:57:20.187760+0000","last_undegraded":"2024-09-15T21:57:20.187760+0000","last_fullsized":"2024-09-15T21:57:20.187760+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472098+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:57:20.472098+0000","last_peered":"2024-09-15T21:57:20.472098+0000","last_clean":"2024-09-15T21:57:20.472098+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:57:20.472098+0000","last_undegraded":"2024-09-15T21:57:20.472098+0000","last_fullsized":"2024-09-15T21:57:20.472098+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185779+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:57:20.185779+0000","last_peered":"2024-09-15T21:57:20.185779+0000","last_clean":"2024-09-15T21:57:20.185779+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:57:20.185779+0000","last_undegraded":"2024-09-15T21:57:20.185779+0000","last_fullsized":"2024-09-15T21:57:20.185779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395654+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:57:20.395654+0000","last_peered":"2024-09-15T21:57:20.395654+0000","last_clean":"2024-09-15T21:57:20.395654+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:57:20.395654+0000","last_undegraded":"2024-09-15T21:57:20.395654+0000","last_fullsized":"2024-09-15T21:57:20.395654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188109+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:57:20.188109+0000","last_peered":"2024-09-15T21:57:20.188109+0000","last_clean":"2024-09-15T21:57:20.188109+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:57:20.188109+0000","last_undegraded":"2024-09-15T21:57:20.188109+0000","last_fullsized":"2024-09-15T21:57:20.188109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278543+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:57:20.278543+0000","last_peered":"2024-09-15T21:57:20.278543+0000","last_clean":"2024-09-15T21:57:20.278543+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:57:20.278543+0000","last_undegraded":"2024-09-15T21:57:20.278543+0000","last_fullsized":"2024-09-15T21:57:20.278543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'45","reported_seq":328,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.725042+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:57:30.725042+0000","last_peered":"2024-09-15T21:57:30.725042+0000","last_clean":"2024-09-15T21:57:30.725042+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:57:30.725042+0000","last_undegraded":"2024-09-15T21:57:30.725042+0000","last_fullsized":"2024-09-15T21:57:30.725042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72639,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":192,"num_write":47,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275890+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:57:20.275890+0000","last_peered":"2024-09-15T21:57:20.275890+0000","last_clean":"2024-09-15T21:57:20.275890+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:57:20.275890+0000","last_undegraded":"2024-09-15T21:57:20.275890+0000","last_fullsized":"2024-09-15T21:57:20.275890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463470+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:57:20.463470+0000","last_peered":"2024-09-15T21:57:20.463470+0000","last_clean":"2024-09-15T21:57:20.463470+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:57:20.463470+0000","last_undegraded":"2024-09-15T21:57:20.463470+0000","last_fullsized":"2024-09-15T21:57:20.463470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188450+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:57:20.188450+0000","last_peered":"2024-09-15T21:57:20.188450+0000","last_clean":"2024-09-15T21:57:20.188450+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:57:20.188450+0000","last_undegraded":"2024-09-15T21:57:20.188450+0000","last_fullsized":"2024-09-15T21:57:20.188450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190380+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:57:20.190380+0000","last_peered":"2024-09-15T21:57:20.190380+0000","last_clean":"2024-09-15T21:57:20.190380+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:57:20.190380+0000","last_undegraded":"2024-09-15T21:57:20.190380+0000","last_fullsized":"2024-09-15T21:57:20.190380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467982+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:57:20.467982+0000","last_peered":"2024-09-15T21:57:20.467982+0000","last_clean":"2024-09-15T21:57:20.467982+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:57:20.467982+0000","last_undegraded":"2024-09-15T21:57:20.467982+0000","last_fullsized":"2024-09-15T21:57:20.467982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461005+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:57:20.461005+0000","last_peered":"2024-09-15T21:57:20.461005+0000","last_clean":"2024-09-15T21:57:20.461005+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:57:20.461005+0000","last_undegraded":"2024-09-15T21:57:20.461005+0000","last_fullsized":"2024-09-15T21:57:20.461005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275613+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:57:20.275613+0000","last_peered":"2024-09-15T21:57:20.275613+0000","last_clean":"2024-09-15T21:57:20.275613+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:57:20.275613+0000","last_undegraded":"2024-09-15T21:57:20.275613+0000","last_fullsized":"2024-09-15T21:57:20.275613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463309+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:57:20.463309+0000","last_peered":"2024-09-15T21:57:20.463309+0000","last_clean":"2024-09-15T21:57:20.463309+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:57:20.463309+0000","last_undegraded":"2024-09-15T21:57:20.463309+0000","last_fullsized":"2024-09-15T21:57:20.463309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'8","reported_seq":275,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.724567+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:57:30.724567+0000","last_peered":"2024-09-15T21:57:30.724567+0000","last_clean":"2024-09-15T21:57:30.724567+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:57:30.724567+0000","last_undegraded":"2024-09-15T21:57:30.724567+0000","last_fullsized":"2024-09-15T21:57:30.724567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278776+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:57:20.278776+0000","last_peered":"2024-09-15T21:57:20.278776+0000","last_clean":"2024-09-15T21:57:20.278776+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:57:20.278776+0000","last_undegraded":"2024-09-15T21:57:20.278776+0000","last_fullsized":"2024-09-15T21:57:20.278776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467332+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:57:20.467332+0000","last_peered":"2024-09-15T21:57:20.467332+0000","last_clean":"2024-09-15T21:57:20.467332+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:57:20.467332+0000","last_undegraded":"2024-09-15T21:57:20.467332+0000","last_fullsized":"2024-09-15T21:57:20.467332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275877+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:57:20.275877+0000","last_peered":"2024-09-15T21:57:20.275877+0000","last_clean":"2024-09-15T21:57:20.275877+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:57:20.275877+0000","last_undegraded":"2024-09-15T21:57:20.275877+0000","last_fullsized":"2024-09-15T21:57:20.275877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187675+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:57:20.187675+0000","last_peered":"2024-09-15T21:57:20.187675+0000","last_clean":"2024-09-15T21:57:20.187675+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:57:20.187675+0000","last_undegraded":"2024-09-15T21:57:20.187675+0000","last_fullsized":"2024-09-15T21:57:20.187675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472373+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:57:20.472373+0000","last_peered":"2024-09-15T21:57:20.472373+0000","last_clean":"2024-09-15T21:57:20.472373+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:57:20.472373+0000","last_undegraded":"2024-09-15T21:57:20.472373+0000","last_fullsized":"2024-09-15T21:57:20.472373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278428+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:57:20.278428+0000","last_peered":"2024-09-15T21:57:20.278428+0000","last_clean":"2024-09-15T21:57:20.278428+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:57:20.278428+0000","last_undegraded":"2024-09-15T21:57:20.278428+0000","last_fullsized":"2024-09-15T21:57:20.278428+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185660+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:57:20.185660+0000","last_peered":"2024-09-15T21:57:20.185660+0000","last_clean":"2024-09-15T21:57:20.185660+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:57:20.185660+0000","last_undegraded":"2024-09-15T21:57:20.185660+0000","last_fullsized":"2024-09-15T21:57:20.185660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285631+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:57:20.285631+0000","last_peered":"2024-09-15T21:57:20.285631+0000","last_clean":"2024-09-15T21:57:20.285631+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:57:20.285631+0000","last_undegraded":"2024-09-15T21:57:20.285631+0000","last_fullsized":"2024-09-15T21:57:20.285631+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187959+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:57:20.187959+0000","last_peered":"2024-09-15T21:57:20.187959+0000","last_clean":"2024-09-15T21:57:20.187959+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:57:20.187959+0000","last_undegraded":"2024-09-15T21:57:20.187959+0000","last_fullsized":"2024-09-15T21:57:20.187959+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396115+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:57:20.396115+0000","last_peered":"2024-09-15T21:57:20.396115+0000","last_clean":"2024-09-15T21:57:20.396115+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:57:20.396115+0000","last_undegraded":"2024-09-15T21:57:20.396115+0000","last_fullsized":"2024-09-15T21:57:20.396115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'10","reported_seq":171,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.723820+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:57:29.723820+0000","last_peered":"2024-09-15T21:57:29.723820+0000","last_clean":"2024-09-15T21:57:29.723820+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:57:29.723820+0000","last_undegraded":"2024-09-15T21:57:29.723820+0000","last_fullsized":"2024-09-15T21:57:29.723820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":7,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278237+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:57:20.278237+0000","last_peered":"2024-09-15T21:57:20.278237+0000","last_clean":"2024-09-15T21:57:20.278237+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:57:20.278237+0000","last_undegraded":"2024-09-15T21:57:20.278237+0000","last_fullsized":"2024-09-15T21:57:20.278237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396197+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:57:20.396197+0000","last_peered":"2024-09-15T21:57:20.396197+0000","last_clean":"2024-09-15T21:57:20.396197+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:57:20.396197+0000","last_undegraded":"2024-09-15T21:57:20.396197+0000","last_fullsized":"2024-09-15T21:57:20.396197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471807+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:57:20.471807+0000","last_peered":"2024-09-15T21:57:20.471807+0000","last_clean":"2024-09-15T21:57:20.471807+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:57:20.471807+0000","last_undegraded":"2024-09-15T21:57:20.471807+0000","last_fullsized":"2024-09-15T21:57:20.471807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396260+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:57:20.396260+0000","last_peered":"2024-09-15T21:57:20.396260+0000","last_clean":"2024-09-15T21:57:20.396260+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:57:20.396260+0000","last_undegraded":"2024-09-15T21:57:20.396260+0000","last_fullsized":"2024-09-15T21:57:20.396260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395966+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:57:20.395966+0000","last_peered":"2024-09-15T21:57:20.395966+0000","last_clean":"2024-09-15T21:57:20.395966+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:57:20.395966+0000","last_undegraded":"2024-09-15T21:57:20.395966+0000","last_fullsized":"2024-09-15T21:57:20.395966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466952+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:57:20.466952+0000","last_peered":"2024-09-15T21:57:20.466952+0000","last_clean":"2024-09-15T21:57:20.466952+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:57:20.466952+0000","last_undegraded":"2024-09-15T21:57:20.466952+0000","last_fullsized":"2024-09-15T21:57:20.466952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186090+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:57:20.186090+0000","last_peered":"2024-09-15T21:57:20.186090+0000","last_clean":"2024-09-15T21:57:20.186090+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:57:20.186090+0000","last_undegraded":"2024-09-15T21:57:20.186090+0000","last_fullsized":"2024-09-15T21:57:20.186090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472240+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:57:20.472240+0000","last_peered":"2024-09-15T21:57:20.472240+0000","last_clean":"2024-09-15T21:57:20.472240+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:57:20.472240+0000","last_undegraded":"2024-09-15T21:57:20.472240+0000","last_fullsized":"2024-09-15T21:57:20.472240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396338+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:57:20.396338+0000","last_peered":"2024-09-15T21:57:20.396338+0000","last_clean":"2024-09-15T21:57:20.396338+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:57:20.396338+0000","last_undegraded":"2024-09-15T21:57:20.396338+0000","last_fullsized":"2024-09-15T21:57:20.396338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280054+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:57:20.280054+0000","last_peered":"2024-09-15T21:57:20.280054+0000","last_clean":"2024-09-15T21:57:20.280054+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:57:20.280054+0000","last_undegraded":"2024-09-15T21:57:20.280054+0000","last_fullsized":"2024-09-15T21:57:20.280054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187475+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:57:20.187475+0000","last_peered":"2024-09-15T21:57:20.187475+0000","last_clean":"2024-09-15T21:57:20.187475+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:57:20.187475+0000","last_undegraded":"2024-09-15T21:57:20.187475+0000","last_fullsized":"2024-09-15T21:57:20.187475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278973+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:57:20.278973+0000","last_peered":"2024-09-15T21:57:20.278973+0000","last_clean":"2024-09-15T21:57:20.278973+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:57:20.278973+0000","last_undegraded":"2024-09-15T21:57:20.278973+0000","last_fullsized":"2024-09-15T21:57:20.278973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463016+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:57:20.463016+0000","last_peered":"2024-09-15T21:57:20.463016+0000","last_clean":"2024-09-15T21:57:20.463016+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:57:20.463016+0000","last_undegraded":"2024-09-15T21:57:20.463016+0000","last_fullsized":"2024-09-15T21:57:20.463016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186044+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:57:20.186044+0000","last_peered":"2024-09-15T21:57:20.186044+0000","last_clean":"2024-09-15T21:57:20.186044+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:57:20.186044+0000","last_undegraded":"2024-09-15T21:57:20.186044+0000","last_fullsized":"2024-09-15T21:57:20.186044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472203+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:57:20.472203+0000","last_peered":"2024-09-15T21:57:20.472203+0000","last_clean":"2024-09-15T21:57:20.472203+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:57:20.472203+0000","last_undegraded":"2024-09-15T21:57:20.472203+0000","last_fullsized":"2024-09-15T21:57:20.472203+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396319+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:57:20.396319+0000","last_peered":"2024-09-15T21:57:20.396319+0000","last_clean":"2024-09-15T21:57:20.396319+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:57:20.396319+0000","last_undegraded":"2024-09-15T21:57:20.396319+0000","last_fullsized":"2024-09-15T21:57:20.396319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276457+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:57:20.276457+0000","last_peered":"2024-09-15T21:57:20.276457+0000","last_clean":"2024-09-15T21:57:20.276457+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:57:20.276457+0000","last_undegraded":"2024-09-15T21:57:20.276457+0000","last_fullsized":"2024-09-15T21:57:20.276457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472556+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:57:20.472556+0000","last_peered":"2024-09-15T21:57:20.472556+0000","last_clean":"2024-09-15T21:57:20.472556+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:57:20.472556+0000","last_undegraded":"2024-09-15T21:57:20.472556+0000","last_fullsized":"2024-09-15T21:57:20.472556+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462857+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:57:20.462857+0000","last_peered":"2024-09-15T21:57:20.462857+0000","last_clean":"2024-09-15T21:57:20.462857+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:57:20.462857+0000","last_undegraded":"2024-09-15T21:57:20.462857+0000","last_fullsized":"2024-09-15T21:57:20.462857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466723+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:57:20.466723+0000","last_peered":"2024-09-15T21:57:20.466723+0000","last_clean":"2024-09-15T21:57:20.466723+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:57:20.466723+0000","last_undegraded":"2024-09-15T21:57:20.466723+0000","last_fullsized":"2024-09-15T21:57:20.466723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466886+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:57:20.466886+0000","last_peered":"2024-09-15T21:57:20.466886+0000","last_clean":"2024-09-15T21:57:20.466886+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:57:20.466886+0000","last_undegraded":"2024-09-15T21:57:20.466886+0000","last_fullsized":"2024-09-15T21:57:20.466886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186220+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:57:20.186220+0000","last_peered":"2024-09-15T21:57:20.186220+0000","last_clean":"2024-09-15T21:57:20.186220+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:57:20.186220+0000","last_undegraded":"2024-09-15T21:57:20.186220+0000","last_fullsized":"2024-09-15T21:57:20.186220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276015+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:57:20.276015+0000","last_peered":"2024-09-15T21:57:20.276015+0000","last_clean":"2024-09-15T21:57:20.276015+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:57:20.276015+0000","last_undegraded":"2024-09-15T21:57:20.276015+0000","last_fullsized":"2024-09-15T21:57:20.276015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461118+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:57:20.461118+0000","last_peered":"2024-09-15T21:57:20.461118+0000","last_clean":"2024-09-15T21:57:20.461118+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:57:20.461118+0000","last_undegraded":"2024-09-15T21:57:20.461118+0000","last_fullsized":"2024-09-15T21:57:20.461118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.286112+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:57:20.286112+0000","last_peered":"2024-09-15T21:57:20.286112+0000","last_clean":"2024-09-15T21:57:20.286112+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:57:20.286112+0000","last_undegraded":"2024-09-15T21:57:20.286112+0000","last_fullsized":"2024-09-15T21:57:20.286112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463385+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:57:20.463385+0000","last_peered":"2024-09-15T21:57:20.463385+0000","last_clean":"2024-09-15T21:57:20.463385+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:57:20.463385+0000","last_undegraded":"2024-09-15T21:57:20.463385+0000","last_fullsized":"2024-09-15T21:57:20.463385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278238+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:57:20.278238+0000","last_peered":"2024-09-15T21:57:20.278238+0000","last_clean":"2024-09-15T21:57:20.278238+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:57:20.278238+0000","last_undegraded":"2024-09-15T21:57:20.278238+0000","last_fullsized":"2024-09-15T21:57:20.278238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460583+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:57:20.460583+0000","last_peered":"2024-09-15T21:57:20.460583+0000","last_clean":"2024-09-15T21:57:20.460583+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:57:20.460583+0000","last_undegraded":"2024-09-15T21:57:20.460583+0000","last_fullsized":"2024-09-15T21:57:20.460583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285573+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:57:20.285573+0000","last_peered":"2024-09-15T21:57:20.285573+0000","last_clean":"2024-09-15T21:57:20.285573+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:57:20.285573+0000","last_undegraded":"2024-09-15T21:57:20.285573+0000","last_fullsized":"2024-09-15T21:57:20.285573+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278231+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:57:20.278231+0000","last_peered":"2024-09-15T21:57:20.278231+0000","last_clean":"2024-09-15T21:57:20.278231+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:57:20.278231+0000","last_undegraded":"2024-09-15T21:57:20.278231+0000","last_fullsized":"2024-09-15T21:57:20.278231+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190742+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:57:20.190742+0000","last_peered":"2024-09-15T21:57:20.190742+0000","last_clean":"2024-09-15T21:57:20.190742+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:57:20.190742+0000","last_undegraded":"2024-09-15T21:57:20.190742+0000","last_fullsized":"2024-09-15T21:57:20.190742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467111+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:57:20.467111+0000","last_peered":"2024-09-15T21:57:20.467111+0000","last_clean":"2024-09-15T21:57:20.467111+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:57:20.467111+0000","last_undegraded":"2024-09-15T21:57:20.467111+0000","last_fullsized":"2024-09-15T21:57:20.467111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467430+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:57:20.467430+0000","last_peered":"2024-09-15T21:57:20.467430+0000","last_clean":"2024-09-15T21:57:20.467430+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:57:20.467430+0000","last_undegraded":"2024-09-15T21:57:20.467430+0000","last_fullsized":"2024-09-15T21:57:20.467430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472483+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:57:20.472483+0000","last_peered":"2024-09-15T21:57:20.472483+0000","last_clean":"2024-09-15T21:57:20.472483+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:57:20.472483+0000","last_undegraded":"2024-09-15T21:57:20.472483+0000","last_fullsized":"2024-09-15T21:57:20.472483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":804243,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1509,"num_read_kb":973,"num_write":515,"num_write_kb":2400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2777088,"data_stored":2412729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514540,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1081344,"data_stored":536154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776176,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29096,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716056,"statfs":{"total":95995035648,"available":95965241344,"internally_reserved":0,"allocated":2121728,"data_stored":1531310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939703,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1490944,"data_stored":923695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168635,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36796,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708356,"statfs":{"total":95995035648,"available":95957356544,"internally_reserved":0,"allocated":1486848,"data_stored":927587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397569,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37004,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708148,"statfs":{"total":95995035648,"available":95957143552,"internally_reserved":0,"allocated":1695744,"data_stored":1133866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561092,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":530472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790027,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29132,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716020,"statfs":{"total":95995035648,"available":95965204480,"internally_reserved":0,"allocated":2035712,"data_stored":1456501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953552,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":534939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1617920,"data_stored":1054603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1175552,"data_stored":606523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":604141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":532462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":77166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":753664,"data_stored":721878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":332680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":403393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:57:35.199 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T21:57:35.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T21:57:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:35 smithi028 ceph-mon[30969]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-09-15T21:57:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:35 smithi028 ceph-mon[30969]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:35.460 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:35.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:35 smithi203 ceph-mon[35546]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-09-15T21:57:35.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:35 smithi203 ceph-mon[35546]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:35.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:35 smithi174 ceph-mon[35244]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-09-15T21:57:35.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:35 smithi174 ceph-mon[35244]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 -- 172.21.15.28:0/4132287167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106f10 msgr2=0x7f280010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 --2- 172.21.15.28:0/4132287167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106f10 0x7f280010ea90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009ca0 tx=0x7f27f802a8c0 comp rx=0 tx=0).stop 2024-09-15T21:57:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 -- 172.21.15.28:0/4132287167 shutdown_connections 2024-09-15T21:57:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 --2- 172.21.15.28:0/4132287167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800106f10 0x7f280010ea90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 --2- 172.21.15.28:0/4132287167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2800102790 0x7f28001067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 --2- 172.21.15.28:0/4132287167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2800101de0 0x7f28001021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 -- 172.21.15.28:0/4132287167 >> 172.21.15.28:0/4132287167 conn(0x7f28000fd8c0 msgr2=0x7f28000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 -- 172.21.15.28:0/4132287167 shutdown_connections 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 -- 172.21.15.28:0/4132287167 wait complete. 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 Processor -- start 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.934+0000 7f280744a640 1 -- start start 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f280744a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800101de0 0x7f28000744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f280744a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2800102790 0x7f2800074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f280744a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2800106f10 0x7f2800074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f280744a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2800111f70 con 0x7f2800101de0 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f280744a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2800111df0 con 0x7f2800102790 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f280744a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f28001120f0 con 0x7f2800106f10 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800101de0 0x7f28000744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:35.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800101de0 0x7f28000744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54298/0 (socket says 172.21.15.28:54298) 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 -- 172.21.15.28:0/383526627 learned_addr learned my addr 172.21.15.28:0/383526627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28049e5640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2800102790 0x7f2800074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28059e7640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2800106f10 0x7f2800074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 -- 172.21.15.28:0/383526627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2800106f10 msgr2=0x7f2800074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2800106f10 0x7f2800074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 -- 172.21.15.28:0/383526627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2800102790 msgr2=0x7f2800074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2800102790 0x7f2800074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 -- 172.21.15.28:0/383526627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28001a6810 con 0x7f2800101de0 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28059e7640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2800106f10 0x7f2800074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f28051e6640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800101de0 0x7f28000744f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c920 tx=0x7f27fc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f27f67fc640 1 -- 172.21.15.28:0/383526627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc007c70 con 0x7f2800101de0 2024-09-15T21:57:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f27f67fc640 1 -- 172.21.15.28:0/383526627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27fc007e10 con 0x7f2800101de0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28001a6b00 con 0x7f2800101de0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.935+0000 7f27f67fc640 1 -- 172.21.15.28:0/383526627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc018890 con 0x7f2800101de0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.936+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28001a7090 con 0x7f2800101de0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.937+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28001021c0 con 0x7f2800101de0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.938+0000 7f27f67fc640 1 -- 172.21.15.28:0/383526627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f27fc018a70 con 0x7f2800101de0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.939+0000 7f27f67fc640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27ec0a3550 0x7f27ec0a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.939+0000 7f28049e5640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27ec0a3550 0x7f27ec0a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.939+0000 7f27f67fc640 1 -- 172.21.15.28:0/383526627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f27fc0d03f0 con 0x7f2800101de0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.939+0000 7f28049e5640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27ec0a3550 0x7f27ec0a5a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2800075f80 tx=0x7f27f000b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:35.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:35.943+0000 7f27f67fc640 1 -- 172.21.15.28:0/383526627 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f27fc014030 con 0x7f2800101de0 2024-09-15T21:57:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.114+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f28000fec00 con 0x7f27ec0a3550 2024-09-15T21:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.122+0000 7f27f67fc640 1 -- 172.21.15.28:0/383526627 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339473 (secure 0 0 0) 0x7f28000fec00 con 0x7f27ec0a3550 2024-09-15T21:57:36.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-15T21:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.125+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27ec0a3550 msgr2=0x7f27ec0a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.125+0000 7f280744a640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27ec0a3550 0x7f27ec0a5a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2800075f80 tx=0x7f27f000b390 comp rx=0 tx=0).stop 2024-09-15T21:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.125+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800101de0 msgr2=0x7f28000744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.125+0000 7f280744a640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800101de0 0x7f28000744f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c920 tx=0x7f27fc00cde0 comp rx=0 tx=0).stop 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 shutdown_connections 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27ec0a3550 0x7f27ec0a5a10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2800106f10 0x7f2800074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2800102790 0x7f2800074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 --2- 172.21.15.28:0/383526627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2800101de0 0x7f28000744f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 >> 172.21.15.28:0/383526627 conn(0x7f28000fd8c0 msgr2=0x7f28000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 shutdown_connections 2024-09-15T21:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:36.126+0000 7f280744a640 1 -- 172.21.15.28:0/383526627 wait complete. 2024-09-15T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-09-15T21:57:34.204598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1394845,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1746,"num_read_kb":1303,"num_write":718,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1181552,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1044,"ondisk_log_size":1044,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366112,"kb_used_data":16724,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124575712,"statfs":{"total":1151940427776,"available":1151565529088,"internally_reserved":0,"allocated":17125376,"data_stored":10372253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":201076,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":438,"num_read_kb":480,"num_write":146,"num_write_kb":923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.1f","version":"93'14","reported_seq":170,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.857401+0000","last_change":"2024-09-15T21:54:56.346528+0000","last_active":"2024-09-15T21:57:29.857401+0000","last_peered":"2024-09-15T21:57:29.857401+0000","last_clean":"2024-09-15T21:57:29.857401+0000","last_became_active":"2024-09-15T21:54:56.346189+0000","last_became_peered":"2024-09-15T21:54:56.346189+0000","last_unstale":"2024-09-15T21:57:29.857401+0000","last_undegraded":"2024-09-15T21:57:29.857401+0000","last_fullsized":"2024-09-15T21:57:29.857401+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:27:13.526118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280359+0000","last_change":"2024-09-15T21:54:58.444861+0000","last_active":"2024-09-15T21:57:20.280359+0000","last_peered":"2024-09-15T21:57:20.280359+0000","last_clean":"2024-09-15T21:57:20.280359+0000","last_became_active":"2024-09-15T21:54:58.443056+0000","last_became_peered":"2024-09-15T21:54:58.443056+0000","last_unstale":"2024-09-15T21:57:20.280359+0000","last_undegraded":"2024-09-15T21:57:20.280359+0000","last_fullsized":"2024-09-15T21:57:20.280359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:12.622508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395705+0000","last_change":"2024-09-15T21:55:05.471791+0000","last_active":"2024-09-15T21:57:20.395705+0000","last_peered":"2024-09-15T21:57:20.395705+0000","last_clean":"2024-09-15T21:57:20.395705+0000","last_became_active":"2024-09-15T21:55:05.471434+0000","last_became_peered":"2024-09-15T21:55:05.471434+0000","last_unstale":"2024-09-15T21:57:20.395705+0000","last_undegraded":"2024-09-15T21:57:20.395705+0000","last_fullsized":"2024-09-15T21:57:20.395705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:49.923635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280756+0000","last_change":"2024-09-15T21:54:56.482159+0000","last_active":"2024-09-15T21:57:20.280756+0000","last_peered":"2024-09-15T21:57:20.280756+0000","last_clean":"2024-09-15T21:57:20.280756+0000","last_became_active":"2024-09-15T21:54:56.481824+0000","last_became_peered":"2024-09-15T21:54:56.481824+0000","last_unstale":"2024-09-15T21:57:20.280756+0000","last_undegraded":"2024-09-15T21:57:20.280756+0000","last_fullsized":"2024-09-15T21:57:20.280756+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:59:33.556659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467940+0000","last_change":"2024-09-15T21:54:58.446418+0000","last_active":"2024-09-15T21:57:20.467940+0000","last_peered":"2024-09-15T21:57:20.467940+0000","last_clean":"2024-09-15T21:57:20.467940+0000","last_became_active":"2024-09-15T21:54:58.445268+0000","last_became_peered":"2024-09-15T21:54:58.445268+0000","last_unstale":"2024-09-15T21:57:20.467940+0000","last_undegraded":"2024-09-15T21:57:20.467940+0000","last_fullsized":"2024-09-15T21:57:20.467940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:18.428228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190109+0000","last_change":"2024-09-15T21:55:05.470498+0000","last_active":"2024-09-15T21:57:20.190109+0000","last_peered":"2024-09-15T21:57:20.190109+0000","last_clean":"2024-09-15T21:57:20.190109+0000","last_became_active":"2024-09-15T21:55:05.469622+0000","last_became_peered":"2024-09-15T21:55:05.469622+0000","last_unstale":"2024-09-15T21:57:20.190109+0000","last_undegraded":"2024-09-15T21:57:20.190109+0000","last_fullsized":"2024-09-15T21:57:20.190109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:01:15.695554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'13","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.570575+0000","last_change":"2024-09-15T21:55:58.813316+0000","last_active":"2024-09-15T21:57:20.570575+0000","last_peered":"2024-09-15T21:57:20.570575+0000","last_clean":"2024-09-15T21:57:20.570575+0000","last_became_active":"2024-09-15T21:55:58.801588+0000","last_became_peered":"2024-09-15T21:55:58.801588+0000","last_unstale":"2024-09-15T21:57:20.570575+0000","last_undegraded":"2024-09-15T21:57:20.570575+0000","last_fullsized":"2024-09-15T21:57:20.570575+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:00.039839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463433+0000","last_change":"2024-09-15T21:54:58.443688+0000","last_active":"2024-09-15T21:57:20.463433+0000","last_peered":"2024-09-15T21:57:20.463433+0000","last_clean":"2024-09-15T21:57:20.463433+0000","last_became_active":"2024-09-15T21:54:58.443391+0000","last_became_peered":"2024-09-15T21:54:58.443391+0000","last_unstale":"2024-09-15T21:57:20.463433+0000","last_undegraded":"2024-09-15T21:57:20.463433+0000","last_fullsized":"2024-09-15T21:57:20.463433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:51.899145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190387+0000","last_change":"2024-09-15T21:55:05.468665+0000","last_active":"2024-09-15T21:57:20.190387+0000","last_peered":"2024-09-15T21:57:20.190387+0000","last_clean":"2024-09-15T21:57:20.190387+0000","last_became_active":"2024-09-15T21:55:05.468238+0000","last_became_peered":"2024-09-15T21:55:05.468238+0000","last_unstale":"2024-09-15T21:57:20.190387+0000","last_undegraded":"2024-09-15T21:57:20.190387+0000","last_fullsized":"2024-09-15T21:57:20.190387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:54:49.054927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472029+0000","last_change":"2024-09-15T21:55:58.808326+0000","last_active":"2024-09-15T21:57:20.472029+0000","last_peered":"2024-09-15T21:57:20.472029+0000","last_clean":"2024-09-15T21:57:20.472029+0000","last_became_active":"2024-09-15T21:55:58.799762+0000","last_became_peered":"2024-09-15T21:55:58.799762+0000","last_unstale":"2024-09-15T21:57:20.472029+0000","last_undegraded":"2024-09-15T21:57:20.472029+0000","last_fullsized":"2024-09-15T21:57:20.472029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:32:15.456962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188210+0000","last_change":"2024-09-15T21:54:58.450479+0000","last_active":"2024-09-15T21:57:20.188210+0000","last_peered":"2024-09-15T21:57:20.188210+0000","last_clean":"2024-09-15T21:57:20.188210+0000","last_became_active":"2024-09-15T21:54:58.450190+0000","last_became_peered":"2024-09-15T21:54:58.450190+0000","last_unstale":"2024-09-15T21:57:20.188210+0000","last_undegraded":"2024-09-15T21:57:20.188210+0000","last_fullsized":"2024-09-15T21:57:20.188210+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:37.116194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190320+0000","last_change":"2024-09-15T21:55:05.467873+0000","last_active":"2024-09-15T21:57:20.190320+0000","last_peered":"2024-09-15T21:57:20.190320+0000","last_clean":"2024-09-15T21:57:20.190320+0000","last_became_active":"2024-09-15T21:55:05.467435+0000","last_became_peered":"2024-09-15T21:55:05.467435+0000","last_unstale":"2024-09-15T21:57:20.190320+0000","last_undegraded":"2024-09-15T21:57:20.190320+0000","last_fullsized":"2024-09-15T21:57:20.190320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:00.283734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460655+0000","last_change":"2024-09-15T21:54:58.447604+0000","last_active":"2024-09-15T21:57:20.460655+0000","last_peered":"2024-09-15T21:57:20.460655+0000","last_clean":"2024-09-15T21:57:20.460655+0000","last_became_active":"2024-09-15T21:54:58.447261+0000","last_became_peered":"2024-09-15T21:54:58.447261+0000","last_unstale":"2024-09-15T21:57:20.460655+0000","last_undegraded":"2024-09-15T21:57:20.460655+0000","last_fullsized":"2024-09-15T21:57:20.460655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:34.385216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278262+0000","last_change":"2024-09-15T21:54:56.347516+0000","last_active":"2024-09-15T21:57:20.278262+0000","last_peered":"2024-09-15T21:57:20.278262+0000","last_clean":"2024-09-15T21:57:20.278262+0000","last_became_active":"2024-09-15T21:54:56.347200+0000","last_became_peered":"2024-09-15T21:54:56.347200+0000","last_unstale":"2024-09-15T21:57:20.278262+0000","last_undegraded":"2024-09-15T21:57:20.278262+0000","last_fullsized":"2024-09-15T21:57:20.278262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:14.841572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190739+0000","last_change":"2024-09-15T21:55:05.496185+0000","last_active":"2024-09-15T21:57:20.190739+0000","last_peered":"2024-09-15T21:57:20.190739+0000","last_clean":"2024-09-15T21:57:20.190739+0000","last_became_active":"2024-09-15T21:55:05.495727+0000","last_became_peered":"2024-09-15T21:55:05.495727+0000","last_unstale":"2024-09-15T21:57:20.190739+0000","last_undegraded":"2024-09-15T21:57:20.190739+0000","last_fullsized":"2024-09-15T21:57:20.190739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:42.887119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279728+0000","last_change":"2024-09-15T21:54:58.442061+0000","last_active":"2024-09-15T21:57:20.279728+0000","last_peered":"2024-09-15T21:57:20.279728+0000","last_clean":"2024-09-15T21:57:20.279728+0000","last_became_active":"2024-09-15T21:54:58.441435+0000","last_became_peered":"2024-09-15T21:54:58.441435+0000","last_unstale":"2024-09-15T21:57:20.279728+0000","last_undegraded":"2024-09-15T21:57:20.279728+0000","last_fullsized":"2024-09-15T21:57:20.279728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:23:10.764225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"93'5","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471885+0000","last_change":"2024-09-15T21:56:58.926427+0000","last_active":"2024-09-15T21:57:20.471885+0000","last_peered":"2024-09-15T21:57:20.471885+0000","last_clean":"2024-09-15T21:57:20.471885+0000","last_became_active":"2024-09-15T21:56:58.922755+0000","last_became_peered":"2024-09-15T21:56:58.922755+0000","last_unstale":"2024-09-15T21:57:20.471885+0000","last_undegraded":"2024-09-15T21:57:20.471885+0000","last_fullsized":"2024-09-15T21:57:20.471885+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:22.033354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190660+0000","last_change":"2024-09-15T21:55:58.820660+0000","last_active":"2024-09-15T21:57:20.190660+0000","last_peered":"2024-09-15T21:57:20.190660+0000","last_clean":"2024-09-15T21:57:20.190660+0000","last_became_active":"2024-09-15T21:55:58.813634+0000","last_became_peered":"2024-09-15T21:55:58.813634+0000","last_unstale":"2024-09-15T21:57:20.190660+0000","last_undegraded":"2024-09-15T21:57:20.190660+0000","last_fullsized":"2024-09-15T21:57:20.190660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:20.134506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471869+0000","last_change":"2024-09-15T21:54:58.448211+0000","last_active":"2024-09-15T21:57:20.471869+0000","last_peered":"2024-09-15T21:57:20.471869+0000","last_clean":"2024-09-15T21:57:20.471869+0000","last_became_active":"2024-09-15T21:54:58.447666+0000","last_became_peered":"2024-09-15T21:54:58.447666+0000","last_unstale":"2024-09-15T21:57:20.471869+0000","last_undegraded":"2024-09-15T21:57:20.471869+0000","last_fullsized":"2024-09-15T21:57:20.471869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:42.701526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279173+0000","last_change":"2024-09-15T21:54:56.348664+0000","last_active":"2024-09-15T21:57:20.279173+0000","last_peered":"2024-09-15T21:57:20.279173+0000","last_clean":"2024-09-15T21:57:20.279173+0000","last_became_active":"2024-09-15T21:54:56.348356+0000","last_became_peered":"2024-09-15T21:54:56.348356+0000","last_unstale":"2024-09-15T21:57:20.279173+0000","last_undegraded":"2024-09-15T21:57:20.279173+0000","last_fullsized":"2024-09-15T21:57:20.279173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:52.267497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276125+0000","last_change":"2024-09-15T21:55:05.474234+0000","last_active":"2024-09-15T21:57:20.276125+0000","last_peered":"2024-09-15T21:57:20.276125+0000","last_clean":"2024-09-15T21:57:20.276125+0000","last_became_active":"2024-09-15T21:55:05.473883+0000","last_became_peered":"2024-09-15T21:55:05.473883+0000","last_unstale":"2024-09-15T21:57:20.276125+0000","last_undegraded":"2024-09-15T21:57:20.276125+0000","last_fullsized":"2024-09-15T21:57:20.276125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:35:31.519855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278075+0000","last_change":"2024-09-15T21:54:58.446473+0000","last_active":"2024-09-15T21:57:20.278075+0000","last_peered":"2024-09-15T21:57:20.278075+0000","last_clean":"2024-09-15T21:57:20.278075+0000","last_became_active":"2024-09-15T21:54:58.446153+0000","last_became_peered":"2024-09-15T21:54:58.446153+0000","last_unstale":"2024-09-15T21:57:20.278075+0000","last_undegraded":"2024-09-15T21:57:20.278075+0000","last_fullsized":"2024-09-15T21:57:20.278075+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:13.057029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462853+0000","last_change":"2024-09-15T21:54:56.482766+0000","last_active":"2024-09-15T21:57:20.462853+0000","last_peered":"2024-09-15T21:57:20.462853+0000","last_clean":"2024-09-15T21:57:20.462853+0000","last_became_active":"2024-09-15T21:54:56.482484+0000","last_became_peered":"2024-09-15T21:54:56.482484+0000","last_unstale":"2024-09-15T21:57:20.462853+0000","last_undegraded":"2024-09-15T21:57:20.462853+0000","last_fullsized":"2024-09-15T21:57:20.462853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:45.056023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190763+0000","last_change":"2024-09-15T21:55:05.497117+0000","last_active":"2024-09-15T21:57:20.190763+0000","last_peered":"2024-09-15T21:57:20.190763+0000","last_clean":"2024-09-15T21:57:20.190763+0000","last_became_active":"2024-09-15T21:55:05.496711+0000","last_became_peered":"2024-09-15T21:55:05.496711+0000","last_unstale":"2024-09-15T21:57:20.190763+0000","last_undegraded":"2024-09-15T21:57:20.190763+0000","last_fullsized":"2024-09-15T21:57:20.190763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:59:54.743452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467562+0000","last_change":"2024-09-15T21:54:58.439709+0000","last_active":"2024-09-15T21:57:20.467562+0000","last_peered":"2024-09-15T21:57:20.467562+0000","last_clean":"2024-09-15T21:57:20.467562+0000","last_became_active":"2024-09-15T21:54:58.439354+0000","last_became_peered":"2024-09-15T21:54:58.439354+0000","last_unstale":"2024-09-15T21:57:20.467562+0000","last_undegraded":"2024-09-15T21:57:20.467562+0000","last_fullsized":"2024-09-15T21:57:20.467562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:29:47.512917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190932+0000","last_change":"2024-09-15T21:54:56.393727+0000","last_active":"2024-09-15T21:57:20.190932+0000","last_peered":"2024-09-15T21:57:20.190932+0000","last_clean":"2024-09-15T21:57:20.190932+0000","last_became_active":"2024-09-15T21:54:56.393414+0000","last_became_peered":"2024-09-15T21:54:56.393414+0000","last_unstale":"2024-09-15T21:57:20.190932+0000","last_undegraded":"2024-09-15T21:57:20.190932+0000","last_fullsized":"2024-09-15T21:57:20.190932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:22.246066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395988+0000","last_change":"2024-09-15T21:55:05.473604+0000","last_active":"2024-09-15T21:57:20.395988+0000","last_peered":"2024-09-15T21:57:20.395988+0000","last_clean":"2024-09-15T21:57:20.395988+0000","last_became_active":"2024-09-15T21:55:05.473215+0000","last_became_peered":"2024-09-15T21:55:05.473215+0000","last_unstale":"2024-09-15T21:57:20.395988+0000","last_undegraded":"2024-09-15T21:57:20.395988+0000","last_fullsized":"2024-09-15T21:57:20.395988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:51.223605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280166+0000","last_change":"2024-09-15T21:54:58.436836+0000","last_active":"2024-09-15T21:57:20.280166+0000","last_peered":"2024-09-15T21:57:20.280166+0000","last_clean":"2024-09-15T21:57:20.280166+0000","last_became_active":"2024-09-15T21:54:58.436470+0000","last_became_peered":"2024-09-15T21:54:58.436470+0000","last_unstale":"2024-09-15T21:57:20.280166+0000","last_undegraded":"2024-09-15T21:57:20.280166+0000","last_fullsized":"2024-09-15T21:57:20.280166+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:45.310680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'103","reported_seq":230,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.815967+0000","last_change":"2024-09-15T21:54:56.338473+0000","last_active":"2024-09-15T21:57:30.815967+0000","last_peered":"2024-09-15T21:57:30.815967+0000","last_clean":"2024-09-15T21:57:30.815967+0000","last_became_active":"2024-09-15T21:54:56.338113+0000","last_became_peered":"2024-09-15T21:54:56.338113+0000","last_unstale":"2024-09-15T21:57:30.815967+0000","last_undegraded":"2024-09-15T21:57:30.815967+0000","last_fullsized":"2024-09-15T21:57:30.815967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:26.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":292,"num_read_kb":259,"num_write":187,"num_write_kb":1288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.277996+0000","last_change":"2024-09-15T21:55:05.496689+0000","last_active":"2024-09-15T21:57:20.277996+0000","last_peered":"2024-09-15T21:57:20.277996+0000","last_clean":"2024-09-15T21:57:20.277996+0000","last_became_active":"2024-09-15T21:55:05.496275+0000","last_became_peered":"2024-09-15T21:55:05.496275+0000","last_unstale":"2024-09-15T21:57:20.277996+0000","last_undegraded":"2024-09-15T21:57:20.277996+0000","last_fullsized":"2024-09-15T21:57:20.277996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:41.278194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186161+0000","last_change":"2024-09-15T21:54:58.440937+0000","last_active":"2024-09-15T21:57:20.186161+0000","last_peered":"2024-09-15T21:57:20.186161+0000","last_clean":"2024-09-15T21:57:20.186161+0000","last_became_active":"2024-09-15T21:54:58.440596+0000","last_became_peered":"2024-09-15T21:54:58.440596+0000","last_unstale":"2024-09-15T21:57:20.186161+0000","last_undegraded":"2024-09-15T21:57:20.186161+0000","last_fullsized":"2024-09-15T21:57:20.186161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:21.840648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278630+0000","last_change":"2024-09-15T21:54:56.482117+0000","last_active":"2024-09-15T21:57:20.278630+0000","last_peered":"2024-09-15T21:57:20.278630+0000","last_clean":"2024-09-15T21:57:20.278630+0000","last_became_active":"2024-09-15T21:54:56.481714+0000","last_became_peered":"2024-09-15T21:54:56.481714+0000","last_unstale":"2024-09-15T21:57:20.278630+0000","last_undegraded":"2024-09-15T21:57:20.278630+0000","last_fullsized":"2024-09-15T21:57:20.278630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:38.560037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472019+0000","last_change":"2024-09-15T21:55:05.472009+0000","last_active":"2024-09-15T21:57:20.472019+0000","last_peered":"2024-09-15T21:57:20.472019+0000","last_clean":"2024-09-15T21:57:20.472019+0000","last_became_active":"2024-09-15T21:55:05.471280+0000","last_became_peered":"2024-09-15T21:55:05.471280+0000","last_unstale":"2024-09-15T21:57:20.472019+0000","last_undegraded":"2024-09-15T21:57:20.472019+0000","last_fullsized":"2024-09-15T21:57:20.472019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:26:54.668092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285785+0000","last_change":"2024-09-15T21:54:58.436975+0000","last_active":"2024-09-15T21:57:20.285785+0000","last_peered":"2024-09-15T21:57:20.285785+0000","last_clean":"2024-09-15T21:57:20.285785+0000","last_became_active":"2024-09-15T21:54:58.436631+0000","last_became_peered":"2024-09-15T21:54:58.436631+0000","last_unstale":"2024-09-15T21:57:20.285785+0000","last_undegraded":"2024-09-15T21:57:20.285785+0000","last_fullsized":"2024-09-15T21:57:20.285785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:05.439076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276329+0000","last_change":"2024-09-15T21:54:56.483770+0000","last_active":"2024-09-15T21:57:20.276329+0000","last_peered":"2024-09-15T21:57:20.276329+0000","last_clean":"2024-09-15T21:57:20.276329+0000","last_became_active":"2024-09-15T21:54:56.483459+0000","last_became_peered":"2024-09-15T21:54:56.483459+0000","last_unstale":"2024-09-15T21:57:20.276329+0000","last_undegraded":"2024-09-15T21:57:20.276329+0000","last_fullsized":"2024-09-15T21:57:20.276329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:07.109952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278087+0000","last_change":"2024-09-15T21:55:05.495985+0000","last_active":"2024-09-15T21:57:20.278087+0000","last_peered":"2024-09-15T21:57:20.278087+0000","last_clean":"2024-09-15T21:57:20.278087+0000","last_became_active":"2024-09-15T21:55:05.495573+0000","last_became_peered":"2024-09-15T21:55:05.495573+0000","last_unstale":"2024-09-15T21:57:20.278087+0000","last_undegraded":"2024-09-15T21:57:20.278087+0000","last_fullsized":"2024-09-15T21:57:20.278087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:18.181255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285570+0000","last_change":"2024-09-15T21:54:58.434777+0000","last_active":"2024-09-15T21:57:20.285570+0000","last_peered":"2024-09-15T21:57:20.285570+0000","last_clean":"2024-09-15T21:57:20.285570+0000","last_became_active":"2024-09-15T21:54:58.434423+0000","last_became_peered":"2024-09-15T21:54:58.434423+0000","last_unstale":"2024-09-15T21:57:20.285570+0000","last_undegraded":"2024-09-15T21:57:20.285570+0000","last_fullsized":"2024-09-15T21:57:20.285570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:00.864928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.723341+0000","last_change":"2024-09-15T21:54:56.347765+0000","last_active":"2024-09-15T21:57:29.723341+0000","last_peered":"2024-09-15T21:57:29.723341+0000","last_clean":"2024-09-15T21:57:29.723341+0000","last_became_active":"2024-09-15T21:54:56.347090+0000","last_became_peered":"2024-09-15T21:54:56.347090+0000","last_unstale":"2024-09-15T21:57:29.723341+0000","last_undegraded":"2024-09-15T21:57:29.723341+0000","last_fullsized":"2024-09-15T21:57:29.723341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:47.220134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185976+0000","last_change":"2024-09-15T21:55:05.471635+0000","last_active":"2024-09-15T21:57:20.185976+0000","last_peered":"2024-09-15T21:57:20.185976+0000","last_clean":"2024-09-15T21:57:20.185976+0000","last_became_active":"2024-09-15T21:55:05.471248+0000","last_became_peered":"2024-09-15T21:55:05.471248+0000","last_unstale":"2024-09-15T21:57:20.185976+0000","last_undegraded":"2024-09-15T21:57:20.185976+0000","last_fullsized":"2024-09-15T21:57:20.185976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:52:45.836189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187498+0000","last_change":"2024-09-15T21:54:58.443310+0000","last_active":"2024-09-15T21:57:20.187498+0000","last_peered":"2024-09-15T21:57:20.187498+0000","last_clean":"2024-09-15T21:57:20.187498+0000","last_became_active":"2024-09-15T21:54:58.442836+0000","last_became_peered":"2024-09-15T21:54:58.442836+0000","last_unstale":"2024-09-15T21:57:20.187498+0000","last_undegraded":"2024-09-15T21:57:20.187498+0000","last_fullsized":"2024-09-15T21:57:20.187498+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:08.416785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.731968+0000","last_change":"2024-09-15T21:54:56.481851+0000","last_active":"2024-09-15T21:57:29.731968+0000","last_peered":"2024-09-15T21:57:29.731968+0000","last_clean":"2024-09-15T21:57:29.731968+0000","last_became_active":"2024-09-15T21:54:56.481509+0000","last_became_peered":"2024-09-15T21:54:56.481509+0000","last_unstale":"2024-09-15T21:57:29.731968+0000","last_undegraded":"2024-09-15T21:57:29.731968+0000","last_fullsized":"2024-09-15T21:57:29.731968+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:25:05.780865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186270+0000","last_change":"2024-09-15T21:55:58.813389+0000","last_active":"2024-09-15T21:57:20.186270+0000","last_peered":"2024-09-15T21:57:20.186270+0000","last_clean":"2024-09-15T21:57:20.186270+0000","last_became_active":"2024-09-15T21:55:58.812846+0000","last_became_peered":"2024-09-15T21:55:58.812846+0000","last_unstale":"2024-09-15T21:57:20.186270+0000","last_undegraded":"2024-09-15T21:57:20.186270+0000","last_fullsized":"2024-09-15T21:57:20.186270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:01.146782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185590+0000","last_change":"2024-09-15T21:54:58.447285+0000","last_active":"2024-09-15T21:57:20.185590+0000","last_peered":"2024-09-15T21:57:20.185590+0000","last_clean":"2024-09-15T21:57:20.185590+0000","last_became_active":"2024-09-15T21:54:58.446960+0000","last_became_peered":"2024-09-15T21:54:58.446960+0000","last_unstale":"2024-09-15T21:57:20.185590+0000","last_undegraded":"2024-09-15T21:57:20.185590+0000","last_fullsized":"2024-09-15T21:57:20.185590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:57:58.073576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460124+0000","last_change":"2024-09-15T21:54:56.346233+0000","last_active":"2024-09-15T21:57:20.460124+0000","last_peered":"2024-09-15T21:57:20.460124+0000","last_clean":"2024-09-15T21:57:20.460124+0000","last_became_active":"2024-09-15T21:54:56.345737+0000","last_became_peered":"2024-09-15T21:54:56.345737+0000","last_unstale":"2024-09-15T21:57:20.460124+0000","last_undegraded":"2024-09-15T21:57:20.460124+0000","last_fullsized":"2024-09-15T21:57:20.460124+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:11.528845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188088+0000","last_change":"2024-09-15T21:55:05.496846+0000","last_active":"2024-09-15T21:57:20.188088+0000","last_peered":"2024-09-15T21:57:20.188088+0000","last_clean":"2024-09-15T21:57:20.188088+0000","last_became_active":"2024-09-15T21:55:05.496398+0000","last_became_peered":"2024-09-15T21:55:05.496398+0000","last_unstale":"2024-09-15T21:57:20.188088+0000","last_undegraded":"2024-09-15T21:57:20.188088+0000","last_fullsized":"2024-09-15T21:57:20.188088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:53.984712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.279569+0000","last_change":"2024-09-15T21:54:58.440812+0000","last_active":"2024-09-15T21:57:20.279569+0000","last_peered":"2024-09-15T21:57:20.279569+0000","last_clean":"2024-09-15T21:57:20.279569+0000","last_became_active":"2024-09-15T21:54:58.440409+0000","last_became_peered":"2024-09-15T21:54:58.440409+0000","last_unstale":"2024-09-15T21:57:20.279569+0000","last_undegraded":"2024-09-15T21:57:20.279569+0000","last_fullsized":"2024-09-15T21:57:20.279569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:19:05.405332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190561+0000","last_change":"2024-09-15T21:54:56.349945+0000","last_active":"2024-09-15T21:57:20.190561+0000","last_peered":"2024-09-15T21:57:20.190561+0000","last_clean":"2024-09-15T21:57:20.190561+0000","last_became_active":"2024-09-15T21:54:56.349580+0000","last_became_peered":"2024-09-15T21:54:56.349580+0000","last_unstale":"2024-09-15T21:57:20.190561+0000","last_undegraded":"2024-09-15T21:57:20.190561+0000","last_fullsized":"2024-09-15T21:57:20.190561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:45.117378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"93'148","reported_seq":409,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:22.291343+0000","last_change":"2024-09-15T21:53:43.985323+0000","last_active":"2024-09-15T21:57:22.291343+0000","last_peered":"2024-09-15T21:57:22.291343+0000","last_clean":"2024-09-15T21:57:22.291343+0000","last_became_active":"2024-09-15T21:53:43.984723+0000","last_became_peered":"2024-09-15T21:53:43.984723+0000","last_unstale":"2024-09-15T21:57:22.291343+0000","last_undegraded":"2024-09-15T21:57:22.291343+0000","last_fullsized":"2024-09-15T21:57:22.291343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:50:14.286662+0000","last_clean_scrub_stamp":"2024-09-15T21:50:14.286662+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:00:02.916823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460868+0000","last_change":"2024-09-15T21:55:58.818314+0000","last_active":"2024-09-15T21:57:20.460868+0000","last_peered":"2024-09-15T21:57:20.460868+0000","last_clean":"2024-09-15T21:57:20.460868+0000","last_became_active":"2024-09-15T21:55:58.813893+0000","last_became_peered":"2024-09-15T21:55:58.813893+0000","last_unstale":"2024-09-15T21:57:20.460868+0000","last_undegraded":"2024-09-15T21:57:20.460868+0000","last_fullsized":"2024-09-15T21:57:20.460868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:08.779225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461076+0000","last_change":"2024-09-15T21:54:58.448196+0000","last_active":"2024-09-15T21:57:20.461076+0000","last_peered":"2024-09-15T21:57:20.461076+0000","last_clean":"2024-09-15T21:57:20.461076+0000","last_became_active":"2024-09-15T21:54:58.447900+0000","last_became_peered":"2024-09-15T21:54:58.447900+0000","last_unstale":"2024-09-15T21:57:20.461076+0000","last_undegraded":"2024-09-15T21:57:20.461076+0000","last_fullsized":"2024-09-15T21:57:20.461076+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:50.178866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'8","reported_seq":167,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.731921+0000","last_change":"2024-09-15T21:54:56.482381+0000","last_active":"2024-09-15T21:57:29.731921+0000","last_peered":"2024-09-15T21:57:29.731921+0000","last_clean":"2024-09-15T21:57:29.731921+0000","last_became_active":"2024-09-15T21:54:56.482044+0000","last_became_peered":"2024-09-15T21:54:56.482044+0000","last_unstale":"2024-09-15T21:57:29.731921+0000","last_undegraded":"2024-09-15T21:57:29.731921+0000","last_fullsized":"2024-09-15T21:57:29.731921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:15.191675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462918+0000","last_change":"2024-09-15T21:55:05.468597+0000","last_active":"2024-09-15T21:57:20.462918+0000","last_peered":"2024-09-15T21:57:20.462918+0000","last_clean":"2024-09-15T21:57:20.462918+0000","last_became_active":"2024-09-15T21:55:05.468194+0000","last_became_peered":"2024-09-15T21:55:05.468194+0000","last_unstale":"2024-09-15T21:57:20.462918+0000","last_undegraded":"2024-09-15T21:57:20.462918+0000","last_fullsized":"2024-09-15T21:57:20.462918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:47.028027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460824+0000","last_change":"2024-09-15T21:54:58.440228+0000","last_active":"2024-09-15T21:57:20.460824+0000","last_peered":"2024-09-15T21:57:20.460824+0000","last_clean":"2024-09-15T21:57:20.460824+0000","last_became_active":"2024-09-15T21:54:58.439905+0000","last_became_peered":"2024-09-15T21:54:58.439905+0000","last_unstale":"2024-09-15T21:57:20.460824+0000","last_undegraded":"2024-09-15T21:57:20.460824+0000","last_fullsized":"2024-09-15T21:57:20.460824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:42:48.536792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'67","reported_seq":137,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.810429+0000","last_change":"2024-09-15T21:54:56.349551+0000","last_active":"2024-09-15T21:57:30.810429+0000","last_peered":"2024-09-15T21:57:30.810429+0000","last_clean":"2024-09-15T21:57:30.810429+0000","last_became_active":"2024-09-15T21:54:56.349173+0000","last_became_peered":"2024-09-15T21:54:56.349173+0000","last_unstale":"2024-09-15T21:57:30.810429+0000","last_undegraded":"2024-09-15T21:57:30.810429+0000","last_fullsized":"2024-09-15T21:57:30.810429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:44.859965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":31,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466957+0000","last_change":"2024-09-15T21:55:05.471645+0000","last_active":"2024-09-15T21:57:20.466957+0000","last_peered":"2024-09-15T21:57:20.466957+0000","last_clean":"2024-09-15T21:57:20.466957+0000","last_became_active":"2024-09-15T21:55:05.471170+0000","last_became_peered":"2024-09-15T21:55:05.471170+0000","last_unstale":"2024-09-15T21:57:20.466957+0000","last_undegraded":"2024-09-15T21:57:20.466957+0000","last_fullsized":"2024-09-15T21:57:20.466957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:20.552816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460987+0000","last_change":"2024-09-15T21:54:58.445923+0000","last_active":"2024-09-15T21:57:20.460987+0000","last_peered":"2024-09-15T21:57:20.460987+0000","last_clean":"2024-09-15T21:57:20.460987+0000","last_became_active":"2024-09-15T21:54:58.445612+0000","last_became_peered":"2024-09-15T21:54:58.445612+0000","last_unstale":"2024-09-15T21:57:20.460987+0000","last_undegraded":"2024-09-15T21:57:20.460987+0000","last_fullsized":"2024-09-15T21:57:20.460987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:50.661112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186150+0000","last_change":"2024-09-15T21:54:56.341312+0000","last_active":"2024-09-15T21:57:20.186150+0000","last_peered":"2024-09-15T21:57:20.186150+0000","last_clean":"2024-09-15T21:57:20.186150+0000","last_became_active":"2024-09-15T21:54:56.340952+0000","last_became_peered":"2024-09-15T21:54:56.340952+0000","last_unstale":"2024-09-15T21:57:20.186150+0000","last_undegraded":"2024-09-15T21:57:20.186150+0000","last_fullsized":"2024-09-15T21:57:20.186150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:43:56.234633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187377+0000","last_change":"2024-09-15T21:55:05.466798+0000","last_active":"2024-09-15T21:57:20.187377+0000","last_peered":"2024-09-15T21:57:20.187377+0000","last_clean":"2024-09-15T21:57:20.187377+0000","last_became_active":"2024-09-15T21:55:05.466409+0000","last_became_peered":"2024-09-15T21:55:05.466409+0000","last_unstale":"2024-09-15T21:57:20.187377+0000","last_undegraded":"2024-09-15T21:57:20.187377+0000","last_fullsized":"2024-09-15T21:57:20.187377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:52.489156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472329+0000","last_change":"2024-09-15T21:54:58.441990+0000","last_active":"2024-09-15T21:57:20.472329+0000","last_peered":"2024-09-15T21:57:20.472329+0000","last_clean":"2024-09-15T21:57:20.472329+0000","last_became_active":"2024-09-15T21:54:58.441626+0000","last_became_peered":"2024-09-15T21:54:58.441626+0000","last_unstale":"2024-09-15T21:57:20.472329+0000","last_undegraded":"2024-09-15T21:57:20.472329+0000","last_fullsized":"2024-09-15T21:57:20.472329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:18.695855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278620+0000","last_change":"2024-09-15T21:54:56.345239+0000","last_active":"2024-09-15T21:57:20.278620+0000","last_peered":"2024-09-15T21:57:20.278620+0000","last_clean":"2024-09-15T21:57:20.278620+0000","last_became_active":"2024-09-15T21:54:56.344279+0000","last_became_peered":"2024-09-15T21:54:56.344279+0000","last_unstale":"2024-09-15T21:57:20.278620+0000","last_undegraded":"2024-09-15T21:57:20.278620+0000","last_fullsized":"2024-09-15T21:57:20.278620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:49.446467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187448+0000","last_change":"2024-09-15T21:55:05.467738+0000","last_active":"2024-09-15T21:57:20.187448+0000","last_peered":"2024-09-15T21:57:20.187448+0000","last_clean":"2024-09-15T21:57:20.187448+0000","last_became_active":"2024-09-15T21:55:05.467297+0000","last_became_peered":"2024-09-15T21:55:05.467297+0000","last_unstale":"2024-09-15T21:57:20.187448+0000","last_undegraded":"2024-09-15T21:57:20.187448+0000","last_fullsized":"2024-09-15T21:57:20.187448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:48.324287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187552+0000","last_change":"2024-09-15T21:54:58.449662+0000","last_active":"2024-09-15T21:57:20.187552+0000","last_peered":"2024-09-15T21:57:20.187552+0000","last_clean":"2024-09-15T21:57:20.187552+0000","last_became_active":"2024-09-15T21:54:58.449357+0000","last_became_peered":"2024-09-15T21:54:58.449357+0000","last_unstale":"2024-09-15T21:57:20.187552+0000","last_undegraded":"2024-09-15T21:57:20.187552+0000","last_fullsized":"2024-09-15T21:57:20.187552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:32.430508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278144+0000","last_change":"2024-09-15T21:54:56.482256+0000","last_active":"2024-09-15T21:57:20.278144+0000","last_peered":"2024-09-15T21:57:20.278144+0000","last_clean":"2024-09-15T21:57:20.278144+0000","last_became_active":"2024-09-15T21:54:56.481734+0000","last_became_peered":"2024-09-15T21:54:56.481734+0000","last_unstale":"2024-09-15T21:57:20.278144+0000","last_undegraded":"2024-09-15T21:57:20.278144+0000","last_fullsized":"2024-09-15T21:57:20.278144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:08:36.032588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186097+0000","last_change":"2024-09-15T21:55:05.468915+0000","last_active":"2024-09-15T21:57:20.186097+0000","last_peered":"2024-09-15T21:57:20.186097+0000","last_clean":"2024-09-15T21:57:20.186097+0000","last_became_active":"2024-09-15T21:55:05.468533+0000","last_became_peered":"2024-09-15T21:55:05.468533+0000","last_unstale":"2024-09-15T21:57:20.186097+0000","last_undegraded":"2024-09-15T21:57:20.186097+0000","last_fullsized":"2024-09-15T21:57:20.186097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:24.411519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460653+0000","last_change":"2024-09-15T21:54:58.442138+0000","last_active":"2024-09-15T21:57:20.460653+0000","last_peered":"2024-09-15T21:57:20.460653+0000","last_clean":"2024-09-15T21:57:20.460653+0000","last_became_active":"2024-09-15T21:54:58.441798+0000","last_became_peered":"2024-09-15T21:54:58.441798+0000","last_unstale":"2024-09-15T21:57:20.460653+0000","last_undegraded":"2024-09-15T21:57:20.460653+0000","last_fullsized":"2024-09-15T21:57:20.460653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:27:29.085444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'7","reported_seq":164,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.732525+0000","last_change":"2024-09-15T21:54:56.348955+0000","last_active":"2024-09-15T21:57:29.732525+0000","last_peered":"2024-09-15T21:57:29.732525+0000","last_clean":"2024-09-15T21:57:29.732525+0000","last_became_active":"2024-09-15T21:54:56.348625+0000","last_became_peered":"2024-09-15T21:54:56.348625+0000","last_unstale":"2024-09-15T21:57:29.732525+0000","last_undegraded":"2024-09-15T21:57:29.732525+0000","last_fullsized":"2024-09-15T21:57:29.732525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:54.793082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1374,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276283+0000","last_change":"2024-09-15T21:55:05.475553+0000","last_active":"2024-09-15T21:57:20.276283+0000","last_peered":"2024-09-15T21:57:20.276283+0000","last_clean":"2024-09-15T21:57:20.276283+0000","last_became_active":"2024-09-15T21:55:05.475187+0000","last_became_peered":"2024-09-15T21:55:05.475187+0000","last_unstale":"2024-09-15T21:57:20.276283+0000","last_undegraded":"2024-09-15T21:57:20.276283+0000","last_fullsized":"2024-09-15T21:57:20.276283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:09.261860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460781+0000","last_change":"2024-09-15T21:54:58.438602+0000","last_active":"2024-09-15T21:57:20.460781+0000","last_peered":"2024-09-15T21:57:20.460781+0000","last_clean":"2024-09-15T21:57:20.460781+0000","last_became_active":"2024-09-15T21:54:58.437946+0000","last_became_peered":"2024-09-15T21:54:58.437946+0000","last_unstale":"2024-09-15T21:57:20.460781+0000","last_undegraded":"2024-09-15T21:57:20.460781+0000","last_fullsized":"2024-09-15T21:57:20.460781+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:31.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187760+0000","last_change":"2024-09-15T21:54:56.345490+0000","last_active":"2024-09-15T21:57:20.187760+0000","last_peered":"2024-09-15T21:57:20.187760+0000","last_clean":"2024-09-15T21:57:20.187760+0000","last_became_active":"2024-09-15T21:54:56.344704+0000","last_became_peered":"2024-09-15T21:54:56.344704+0000","last_unstale":"2024-09-15T21:57:20.187760+0000","last_undegraded":"2024-09-15T21:57:20.187760+0000","last_fullsized":"2024-09-15T21:57:20.187760+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:11.412245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472098+0000","last_change":"2024-09-15T21:55:58.802722+0000","last_active":"2024-09-15T21:57:20.472098+0000","last_peered":"2024-09-15T21:57:20.472098+0000","last_clean":"2024-09-15T21:57:20.472098+0000","last_became_active":"2024-09-15T21:55:58.802247+0000","last_became_peered":"2024-09-15T21:55:58.802247+0000","last_unstale":"2024-09-15T21:57:20.472098+0000","last_undegraded":"2024-09-15T21:57:20.472098+0000","last_fullsized":"2024-09-15T21:57:20.472098+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:24.447590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185779+0000","last_change":"2024-09-15T21:54:58.435162+0000","last_active":"2024-09-15T21:57:20.185779+0000","last_peered":"2024-09-15T21:57:20.185779+0000","last_clean":"2024-09-15T21:57:20.185779+0000","last_became_active":"2024-09-15T21:54:58.434829+0000","last_became_peered":"2024-09-15T21:54:58.434829+0000","last_unstale":"2024-09-15T21:57:20.185779+0000","last_undegraded":"2024-09-15T21:57:20.185779+0000","last_fullsized":"2024-09-15T21:57:20.185779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:40.345267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395654+0000","last_change":"2024-09-15T21:54:56.348865+0000","last_active":"2024-09-15T21:57:20.395654+0000","last_peered":"2024-09-15T21:57:20.395654+0000","last_clean":"2024-09-15T21:57:20.395654+0000","last_became_active":"2024-09-15T21:54:56.348518+0000","last_became_peered":"2024-09-15T21:54:56.348518+0000","last_unstale":"2024-09-15T21:57:20.395654+0000","last_undegraded":"2024-09-15T21:57:20.395654+0000","last_fullsized":"2024-09-15T21:57:20.395654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:22.013898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188109+0000","last_change":"2024-09-15T21:55:05.473506+0000","last_active":"2024-09-15T21:57:20.188109+0000","last_peered":"2024-09-15T21:57:20.188109+0000","last_clean":"2024-09-15T21:57:20.188109+0000","last_became_active":"2024-09-15T21:55:05.473072+0000","last_became_peered":"2024-09-15T21:55:05.473072+0000","last_unstale":"2024-09-15T21:57:20.188109+0000","last_undegraded":"2024-09-15T21:57:20.188109+0000","last_fullsized":"2024-09-15T21:57:20.188109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:56:47.339287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278543+0000","last_change":"2024-09-15T21:54:58.452180+0000","last_active":"2024-09-15T21:57:20.278543+0000","last_peered":"2024-09-15T21:57:20.278543+0000","last_clean":"2024-09-15T21:57:20.278543+0000","last_became_active":"2024-09-15T21:54:58.451836+0000","last_became_peered":"2024-09-15T21:54:58.451836+0000","last_unstale":"2024-09-15T21:57:20.278543+0000","last_undegraded":"2024-09-15T21:57:20.278543+0000","last_fullsized":"2024-09-15T21:57:20.278543+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:48.273632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'45","reported_seq":328,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.725042+0000","last_change":"2024-09-15T21:54:56.348552+0000","last_active":"2024-09-15T21:57:30.725042+0000","last_peered":"2024-09-15T21:57:30.725042+0000","last_clean":"2024-09-15T21:57:30.725042+0000","last_became_active":"2024-09-15T21:54:56.348208+0000","last_became_peered":"2024-09-15T21:54:56.348208+0000","last_unstale":"2024-09-15T21:57:30.725042+0000","last_undegraded":"2024-09-15T21:57:30.725042+0000","last_fullsized":"2024-09-15T21:57:30.725042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:34.469030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72639,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":192,"num_write":47,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275890+0000","last_change":"2024-09-15T21:55:05.469003+0000","last_active":"2024-09-15T21:57:20.275890+0000","last_peered":"2024-09-15T21:57:20.275890+0000","last_clean":"2024-09-15T21:57:20.275890+0000","last_became_active":"2024-09-15T21:55:05.468638+0000","last_became_peered":"2024-09-15T21:55:05.468638+0000","last_unstale":"2024-09-15T21:57:20.275890+0000","last_undegraded":"2024-09-15T21:57:20.275890+0000","last_fullsized":"2024-09-15T21:57:20.275890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:30:31.168228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463470+0000","last_change":"2024-09-15T21:54:58.451738+0000","last_active":"2024-09-15T21:57:20.463470+0000","last_peered":"2024-09-15T21:57:20.463470+0000","last_clean":"2024-09-15T21:57:20.463470+0000","last_became_active":"2024-09-15T21:54:58.450726+0000","last_became_peered":"2024-09-15T21:54:58.450726+0000","last_unstale":"2024-09-15T21:57:20.463470+0000","last_undegraded":"2024-09-15T21:57:20.463470+0000","last_fullsized":"2024-09-15T21:57:20.463470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:46.520260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.188450+0000","last_change":"2024-09-15T21:54:56.482198+0000","last_active":"2024-09-15T21:57:20.188450+0000","last_peered":"2024-09-15T21:57:20.188450+0000","last_clean":"2024-09-15T21:57:20.188450+0000","last_became_active":"2024-09-15T21:54:56.481737+0000","last_became_peered":"2024-09-15T21:54:56.481737+0000","last_unstale":"2024-09-15T21:57:20.188450+0000","last_undegraded":"2024-09-15T21:57:20.188450+0000","last_fullsized":"2024-09-15T21:57:20.188450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:39:53.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190380+0000","last_change":"2024-09-15T21:55:05.468345+0000","last_active":"2024-09-15T21:57:20.190380+0000","last_peered":"2024-09-15T21:57:20.190380+0000","last_clean":"2024-09-15T21:57:20.190380+0000","last_became_active":"2024-09-15T21:55:05.467967+0000","last_became_peered":"2024-09-15T21:55:05.467967+0000","last_unstale":"2024-09-15T21:57:20.190380+0000","last_undegraded":"2024-09-15T21:57:20.190380+0000","last_fullsized":"2024-09-15T21:57:20.190380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:09:00.866573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467982+0000","last_change":"2024-09-15T21:54:58.443464+0000","last_active":"2024-09-15T21:57:20.467982+0000","last_peered":"2024-09-15T21:57:20.467982+0000","last_clean":"2024-09-15T21:57:20.467982+0000","last_became_active":"2024-09-15T21:54:58.443171+0000","last_became_peered":"2024-09-15T21:54:58.443171+0000","last_unstale":"2024-09-15T21:57:20.467982+0000","last_undegraded":"2024-09-15T21:57:20.467982+0000","last_fullsized":"2024-09-15T21:57:20.467982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:52.985914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461005+0000","last_change":"2024-09-15T21:54:56.347423+0000","last_active":"2024-09-15T21:57:20.461005+0000","last_peered":"2024-09-15T21:57:20.461005+0000","last_clean":"2024-09-15T21:57:20.461005+0000","last_became_active":"2024-09-15T21:54:56.347067+0000","last_became_peered":"2024-09-15T21:54:56.347067+0000","last_unstale":"2024-09-15T21:57:20.461005+0000","last_undegraded":"2024-09-15T21:57:20.461005+0000","last_fullsized":"2024-09-15T21:57:20.461005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:18.902347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275613+0000","last_change":"2024-09-15T21:55:05.468348+0000","last_active":"2024-09-15T21:57:20.275613+0000","last_peered":"2024-09-15T21:57:20.275613+0000","last_clean":"2024-09-15T21:57:20.275613+0000","last_became_active":"2024-09-15T21:55:05.467981+0000","last_became_peered":"2024-09-15T21:55:05.467981+0000","last_unstale":"2024-09-15T21:57:20.275613+0000","last_undegraded":"2024-09-15T21:57:20.275613+0000","last_fullsized":"2024-09-15T21:57:20.275613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:16:55.102622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463309+0000","last_change":"2024-09-15T21:54:58.443324+0000","last_active":"2024-09-15T21:57:20.463309+0000","last_peered":"2024-09-15T21:57:20.463309+0000","last_clean":"2024-09-15T21:57:20.463309+0000","last_became_active":"2024-09-15T21:54:58.442929+0000","last_became_peered":"2024-09-15T21:54:58.442929+0000","last_unstale":"2024-09-15T21:57:20.463309+0000","last_undegraded":"2024-09-15T21:57:20.463309+0000","last_fullsized":"2024-09-15T21:57:20.463309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:30:54.823829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'8","reported_seq":275,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:30.724567+0000","last_change":"2024-09-15T21:54:56.347311+0000","last_active":"2024-09-15T21:57:30.724567+0000","last_peered":"2024-09-15T21:57:30.724567+0000","last_clean":"2024-09-15T21:57:30.724567+0000","last_became_active":"2024-09-15T21:54:56.347006+0000","last_became_peered":"2024-09-15T21:54:56.347006+0000","last_unstale":"2024-09-15T21:57:30.724567+0000","last_undegraded":"2024-09-15T21:57:30.724567+0000","last_fullsized":"2024-09-15T21:57:30.724567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:24:10.567756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278776+0000","last_change":"2024-09-15T21:55:05.467107+0000","last_active":"2024-09-15T21:57:20.278776+0000","last_peered":"2024-09-15T21:57:20.278776+0000","last_clean":"2024-09-15T21:57:20.278776+0000","last_became_active":"2024-09-15T21:55:05.466702+0000","last_became_peered":"2024-09-15T21:55:05.466702+0000","last_unstale":"2024-09-15T21:57:20.278776+0000","last_undegraded":"2024-09-15T21:57:20.278776+0000","last_fullsized":"2024-09-15T21:57:20.278776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:57.585216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467332+0000","last_change":"2024-09-15T21:55:05.472409+0000","last_active":"2024-09-15T21:57:20.467332+0000","last_peered":"2024-09-15T21:57:20.467332+0000","last_clean":"2024-09-15T21:57:20.467332+0000","last_became_active":"2024-09-15T21:55:05.472039+0000","last_became_peered":"2024-09-15T21:55:05.472039+0000","last_unstale":"2024-09-15T21:57:20.467332+0000","last_undegraded":"2024-09-15T21:57:20.467332+0000","last_fullsized":"2024-09-15T21:57:20.467332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:37.515345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.275877+0000","last_change":"2024-09-15T21:54:58.434671+0000","last_active":"2024-09-15T21:57:20.275877+0000","last_peered":"2024-09-15T21:57:20.275877+0000","last_clean":"2024-09-15T21:57:20.275877+0000","last_became_active":"2024-09-15T21:54:58.434385+0000","last_became_peered":"2024-09-15T21:54:58.434385+0000","last_unstale":"2024-09-15T21:57:20.275877+0000","last_undegraded":"2024-09-15T21:57:20.275877+0000","last_fullsized":"2024-09-15T21:57:20.275877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:06:51.162581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187675+0000","last_change":"2024-09-15T21:54:56.346450+0000","last_active":"2024-09-15T21:57:20.187675+0000","last_peered":"2024-09-15T21:57:20.187675+0000","last_clean":"2024-09-15T21:57:20.187675+0000","last_became_active":"2024-09-15T21:54:56.346136+0000","last_became_peered":"2024-09-15T21:54:56.346136+0000","last_unstale":"2024-09-15T21:57:20.187675+0000","last_undegraded":"2024-09-15T21:57:20.187675+0000","last_fullsized":"2024-09-15T21:57:20.187675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:56.692276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472373+0000","last_change":"2024-09-15T21:55:05.470669+0000","last_active":"2024-09-15T21:57:20.472373+0000","last_peered":"2024-09-15T21:57:20.472373+0000","last_clean":"2024-09-15T21:57:20.472373+0000","last_became_active":"2024-09-15T21:55:05.469624+0000","last_became_peered":"2024-09-15T21:55:05.469624+0000","last_unstale":"2024-09-15T21:57:20.472373+0000","last_undegraded":"2024-09-15T21:57:20.472373+0000","last_fullsized":"2024-09-15T21:57:20.472373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:20.124687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278428+0000","last_change":"2024-09-15T21:54:58.437456+0000","last_active":"2024-09-15T21:57:20.278428+0000","last_peered":"2024-09-15T21:57:20.278428+0000","last_clean":"2024-09-15T21:57:20.278428+0000","last_became_active":"2024-09-15T21:54:58.437124+0000","last_became_peered":"2024-09-15T21:54:58.437124+0000","last_unstale":"2024-09-15T21:57:20.278428+0000","last_undegraded":"2024-09-15T21:57:20.278428+0000","last_fullsized":"2024-09-15T21:57:20.278428+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:20.142150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.185660+0000","last_change":"2024-09-15T21:54:56.346061+0000","last_active":"2024-09-15T21:57:20.185660+0000","last_peered":"2024-09-15T21:57:20.185660+0000","last_clean":"2024-09-15T21:57:20.185660+0000","last_became_active":"2024-09-15T21:54:56.345703+0000","last_became_peered":"2024-09-15T21:54:56.345703+0000","last_unstale":"2024-09-15T21:57:20.185660+0000","last_undegraded":"2024-09-15T21:57:20.185660+0000","last_fullsized":"2024-09-15T21:57:20.185660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:57.539220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285631+0000","last_change":"2024-09-15T21:54:56.350849+0000","last_active":"2024-09-15T21:57:20.285631+0000","last_peered":"2024-09-15T21:57:20.285631+0000","last_clean":"2024-09-15T21:57:20.285631+0000","last_became_active":"2024-09-15T21:54:56.349695+0000","last_became_peered":"2024-09-15T21:54:56.349695+0000","last_unstale":"2024-09-15T21:57:20.285631+0000","last_undegraded":"2024-09-15T21:57:20.285631+0000","last_fullsized":"2024-09-15T21:57:20.285631+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:08.915412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187959+0000","last_change":"2024-09-15T21:55:58.812666+0000","last_active":"2024-09-15T21:57:20.187959+0000","last_peered":"2024-09-15T21:57:20.187959+0000","last_clean":"2024-09-15T21:57:20.187959+0000","last_became_active":"2024-09-15T21:55:58.812294+0000","last_became_peered":"2024-09-15T21:55:58.812294+0000","last_unstale":"2024-09-15T21:57:20.187959+0000","last_undegraded":"2024-09-15T21:57:20.187959+0000","last_fullsized":"2024-09-15T21:57:20.187959+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:53.827861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396115+0000","last_change":"2024-09-15T21:55:05.474502+0000","last_active":"2024-09-15T21:57:20.396115+0000","last_peered":"2024-09-15T21:57:20.396115+0000","last_clean":"2024-09-15T21:57:20.396115+0000","last_became_active":"2024-09-15T21:55:05.474131+0000","last_became_peered":"2024-09-15T21:55:05.474131+0000","last_unstale":"2024-09-15T21:57:20.396115+0000","last_undegraded":"2024-09-15T21:57:20.396115+0000","last_fullsized":"2024-09-15T21:57:20.396115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:35.423028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'10","reported_seq":171,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:29.723820+0000","last_change":"2024-09-15T21:54:56.347402+0000","last_active":"2024-09-15T21:57:29.723820+0000","last_peered":"2024-09-15T21:57:29.723820+0000","last_clean":"2024-09-15T21:57:29.723820+0000","last_became_active":"2024-09-15T21:54:56.347050+0000","last_became_peered":"2024-09-15T21:54:56.347050+0000","last_unstale":"2024-09-15T21:57:29.723820+0000","last_undegraded":"2024-09-15T21:57:29.723820+0000","last_fullsized":"2024-09-15T21:57:29.723820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:55.319323+0000","last_clean_scrub_stamp":"2024-09-15T21:54:55.319323+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:37.531719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":7,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278237+0000","last_change":"2024-09-15T21:54:58.446611+0000","last_active":"2024-09-15T21:57:20.278237+0000","last_peered":"2024-09-15T21:57:20.278237+0000","last_clean":"2024-09-15T21:57:20.278237+0000","last_became_active":"2024-09-15T21:54:58.446283+0000","last_became_peered":"2024-09-15T21:54:58.446283+0000","last_unstale":"2024-09-15T21:57:20.278237+0000","last_undegraded":"2024-09-15T21:57:20.278237+0000","last_fullsized":"2024-09-15T21:57:20.278237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:44.910456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396197+0000","last_change":"2024-09-15T21:55:05.474451+0000","last_active":"2024-09-15T21:57:20.396197+0000","last_peered":"2024-09-15T21:57:20.396197+0000","last_clean":"2024-09-15T21:57:20.396197+0000","last_became_active":"2024-09-15T21:55:05.474020+0000","last_became_peered":"2024-09-15T21:55:05.474020+0000","last_unstale":"2024-09-15T21:57:20.396197+0000","last_undegraded":"2024-09-15T21:57:20.396197+0000","last_fullsized":"2024-09-15T21:57:20.396197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:55:04.448297+0000","last_clean_scrub_stamp":"2024-09-15T21:55:04.448297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:22:53.382220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.471807+0000","last_change":"2024-09-15T21:54:58.442771+0000","last_active":"2024-09-15T21:57:20.471807+0000","last_peered":"2024-09-15T21:57:20.471807+0000","last_clean":"2024-09-15T21:57:20.471807+0000","last_became_active":"2024-09-15T21:54:58.442387+0000","last_became_peered":"2024-09-15T21:54:58.442387+0000","last_unstale":"2024-09-15T21:57:20.471807+0000","last_undegraded":"2024-09-15T21:57:20.471807+0000","last_fullsized":"2024-09-15T21:57:20.471807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:09.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396260+0000","last_change":"2024-09-15T21:55:58.799175+0000","last_active":"2024-09-15T21:57:20.396260+0000","last_peered":"2024-09-15T21:57:20.396260+0000","last_clean":"2024-09-15T21:57:20.396260+0000","last_became_active":"2024-09-15T21:55:58.798776+0000","last_became_peered":"2024-09-15T21:55:58.798776+0000","last_unstale":"2024-09-15T21:57:20.396260+0000","last_undegraded":"2024-09-15T21:57:20.396260+0000","last_fullsized":"2024-09-15T21:57:20.396260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:48.497606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.395966+0000","last_change":"2024-09-15T21:54:58.445023+0000","last_active":"2024-09-15T21:57:20.395966+0000","last_peered":"2024-09-15T21:57:20.395966+0000","last_clean":"2024-09-15T21:57:20.395966+0000","last_became_active":"2024-09-15T21:54:58.444662+0000","last_became_peered":"2024-09-15T21:54:58.444662+0000","last_unstale":"2024-09-15T21:57:20.395966+0000","last_undegraded":"2024-09-15T21:57:20.395966+0000","last_fullsized":"2024-09-15T21:57:20.395966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:12.187084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466952+0000","last_change":"2024-09-15T21:54:58.439683+0000","last_active":"2024-09-15T21:57:20.466952+0000","last_peered":"2024-09-15T21:57:20.466952+0000","last_clean":"2024-09-15T21:57:20.466952+0000","last_became_active":"2024-09-15T21:54:58.439327+0000","last_became_peered":"2024-09-15T21:54:58.439327+0000","last_unstale":"2024-09-15T21:57:20.466952+0000","last_undegraded":"2024-09-15T21:57:20.466952+0000","last_fullsized":"2024-09-15T21:57:20.466952+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:25:35.278060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186090+0000","last_change":"2024-09-15T21:54:58.436410+0000","last_active":"2024-09-15T21:57:20.186090+0000","last_peered":"2024-09-15T21:57:20.186090+0000","last_clean":"2024-09-15T21:57:20.186090+0000","last_became_active":"2024-09-15T21:54:58.436097+0000","last_became_peered":"2024-09-15T21:54:58.436097+0000","last_unstale":"2024-09-15T21:57:20.186090+0000","last_undegraded":"2024-09-15T21:57:20.186090+0000","last_fullsized":"2024-09-15T21:57:20.186090+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:02.238471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472240+0000","last_change":"2024-09-15T21:54:58.453519+0000","last_active":"2024-09-15T21:57:20.472240+0000","last_peered":"2024-09-15T21:57:20.472240+0000","last_clean":"2024-09-15T21:57:20.472240+0000","last_became_active":"2024-09-15T21:54:58.453174+0000","last_became_peered":"2024-09-15T21:54:58.453174+0000","last_unstale":"2024-09-15T21:57:20.472240+0000","last_undegraded":"2024-09-15T21:57:20.472240+0000","last_fullsized":"2024-09-15T21:57:20.472240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:50.136653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396338+0000","last_change":"2024-09-15T21:54:58.448024+0000","last_active":"2024-09-15T21:57:20.396338+0000","last_peered":"2024-09-15T21:57:20.396338+0000","last_clean":"2024-09-15T21:57:20.396338+0000","last_became_active":"2024-09-15T21:54:58.447699+0000","last_became_peered":"2024-09-15T21:54:58.447699+0000","last_unstale":"2024-09-15T21:57:20.396338+0000","last_undegraded":"2024-09-15T21:57:20.396338+0000","last_fullsized":"2024-09-15T21:57:20.396338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:55.109401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.280054+0000","last_change":"2024-09-15T21:54:58.445381+0000","last_active":"2024-09-15T21:57:20.280054+0000","last_peered":"2024-09-15T21:57:20.280054+0000","last_clean":"2024-09-15T21:57:20.280054+0000","last_became_active":"2024-09-15T21:54:58.445058+0000","last_became_peered":"2024-09-15T21:54:58.445058+0000","last_unstale":"2024-09-15T21:57:20.280054+0000","last_undegraded":"2024-09-15T21:57:20.280054+0000","last_fullsized":"2024-09-15T21:57:20.280054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:37:04.053487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.187475+0000","last_change":"2024-09-15T21:54:58.444477+0000","last_active":"2024-09-15T21:57:20.187475+0000","last_peered":"2024-09-15T21:57:20.187475+0000","last_clean":"2024-09-15T21:57:20.187475+0000","last_became_active":"2024-09-15T21:54:58.443363+0000","last_became_peered":"2024-09-15T21:54:58.443363+0000","last_unstale":"2024-09-15T21:57:20.187475+0000","last_undegraded":"2024-09-15T21:57:20.187475+0000","last_fullsized":"2024-09-15T21:57:20.187475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:27.363621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278973+0000","last_change":"2024-09-15T21:54:58.448650+0000","last_active":"2024-09-15T21:57:20.278973+0000","last_peered":"2024-09-15T21:57:20.278973+0000","last_clean":"2024-09-15T21:57:20.278973+0000","last_became_active":"2024-09-15T21:54:58.447918+0000","last_became_peered":"2024-09-15T21:54:58.447918+0000","last_unstale":"2024-09-15T21:57:20.278973+0000","last_undegraded":"2024-09-15T21:57:20.278973+0000","last_fullsized":"2024-09-15T21:57:20.278973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:18:23.578626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463016+0000","last_change":"2024-09-15T21:54:58.450769+0000","last_active":"2024-09-15T21:57:20.463016+0000","last_peered":"2024-09-15T21:57:20.463016+0000","last_clean":"2024-09-15T21:57:20.463016+0000","last_became_active":"2024-09-15T21:54:58.449885+0000","last_became_peered":"2024-09-15T21:54:58.449885+0000","last_unstale":"2024-09-15T21:57:20.463016+0000","last_undegraded":"2024-09-15T21:57:20.463016+0000","last_fullsized":"2024-09-15T21:57:20.463016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:51.131293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186044+0000","last_change":"2024-09-15T21:54:58.433896+0000","last_active":"2024-09-15T21:57:20.186044+0000","last_peered":"2024-09-15T21:57:20.186044+0000","last_clean":"2024-09-15T21:57:20.186044+0000","last_became_active":"2024-09-15T21:54:58.433485+0000","last_became_peered":"2024-09-15T21:54:58.433485+0000","last_unstale":"2024-09-15T21:57:20.186044+0000","last_undegraded":"2024-09-15T21:57:20.186044+0000","last_fullsized":"2024-09-15T21:57:20.186044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:28.376524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472203+0000","last_change":"2024-09-15T21:55:58.804422+0000","last_active":"2024-09-15T21:57:20.472203+0000","last_peered":"2024-09-15T21:57:20.472203+0000","last_clean":"2024-09-15T21:57:20.472203+0000","last_became_active":"2024-09-15T21:55:58.803916+0000","last_became_peered":"2024-09-15T21:55:58.803916+0000","last_unstale":"2024-09-15T21:57:20.472203+0000","last_undegraded":"2024-09-15T21:57:20.472203+0000","last_fullsized":"2024-09-15T21:57:20.472203+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:57.153257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.396319+0000","last_change":"2024-09-15T21:54:58.441217+0000","last_active":"2024-09-15T21:57:20.396319+0000","last_peered":"2024-09-15T21:57:20.396319+0000","last_clean":"2024-09-15T21:57:20.396319+0000","last_became_active":"2024-09-15T21:54:58.440533+0000","last_became_peered":"2024-09-15T21:54:58.440533+0000","last_unstale":"2024-09-15T21:57:20.396319+0000","last_undegraded":"2024-09-15T21:57:20.396319+0000","last_fullsized":"2024-09-15T21:57:20.396319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:20.457800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276457+0000","last_change":"2024-09-15T21:54:58.448252+0000","last_active":"2024-09-15T21:57:20.276457+0000","last_peered":"2024-09-15T21:57:20.276457+0000","last_clean":"2024-09-15T21:57:20.276457+0000","last_became_active":"2024-09-15T21:54:58.447698+0000","last_became_peered":"2024-09-15T21:54:58.447698+0000","last_unstale":"2024-09-15T21:57:20.276457+0000","last_undegraded":"2024-09-15T21:57:20.276457+0000","last_fullsized":"2024-09-15T21:57:20.276457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:07.447474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472556+0000","last_change":"2024-09-15T21:54:58.446379+0000","last_active":"2024-09-15T21:57:20.472556+0000","last_peered":"2024-09-15T21:57:20.472556+0000","last_clean":"2024-09-15T21:57:20.472556+0000","last_became_active":"2024-09-15T21:54:58.446018+0000","last_became_peered":"2024-09-15T21:54:58.446018+0000","last_unstale":"2024-09-15T21:57:20.472556+0000","last_undegraded":"2024-09-15T21:57:20.472556+0000","last_fullsized":"2024-09-15T21:57:20.472556+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:19.027638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.462857+0000","last_change":"2024-09-15T21:54:58.450473+0000","last_active":"2024-09-15T21:57:20.462857+0000","last_peered":"2024-09-15T21:57:20.462857+0000","last_clean":"2024-09-15T21:57:20.462857+0000","last_became_active":"2024-09-15T21:54:58.449969+0000","last_became_peered":"2024-09-15T21:54:58.449969+0000","last_unstale":"2024-09-15T21:57:20.462857+0000","last_undegraded":"2024-09-15T21:57:20.462857+0000","last_fullsized":"2024-09-15T21:57:20.462857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:40.297661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466723+0000","last_change":"2024-09-15T21:54:58.441658+0000","last_active":"2024-09-15T21:57:20.466723+0000","last_peered":"2024-09-15T21:57:20.466723+0000","last_clean":"2024-09-15T21:57:20.466723+0000","last_became_active":"2024-09-15T21:54:58.441330+0000","last_became_peered":"2024-09-15T21:54:58.441330+0000","last_unstale":"2024-09-15T21:57:20.466723+0000","last_undegraded":"2024-09-15T21:57:20.466723+0000","last_fullsized":"2024-09-15T21:57:20.466723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:45.048126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.466886+0000","last_change":"2024-09-15T21:54:58.444433+0000","last_active":"2024-09-15T21:57:20.466886+0000","last_peered":"2024-09-15T21:57:20.466886+0000","last_clean":"2024-09-15T21:57:20.466886+0000","last_became_active":"2024-09-15T21:54:58.444129+0000","last_became_peered":"2024-09-15T21:54:58.444129+0000","last_unstale":"2024-09-15T21:57:20.466886+0000","last_undegraded":"2024-09-15T21:57:20.466886+0000","last_fullsized":"2024-09-15T21:57:20.466886+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:50.725724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.186220+0000","last_change":"2024-09-15T21:54:58.441850+0000","last_active":"2024-09-15T21:57:20.186220+0000","last_peered":"2024-09-15T21:57:20.186220+0000","last_clean":"2024-09-15T21:57:20.186220+0000","last_became_active":"2024-09-15T21:54:58.441519+0000","last_became_peered":"2024-09-15T21:54:58.441519+0000","last_unstale":"2024-09-15T21:57:20.186220+0000","last_undegraded":"2024-09-15T21:57:20.186220+0000","last_fullsized":"2024-09-15T21:57:20.186220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:35.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.276015+0000","last_change":"2024-09-15T21:55:58.813704+0000","last_active":"2024-09-15T21:57:20.276015+0000","last_peered":"2024-09-15T21:57:20.276015+0000","last_clean":"2024-09-15T21:57:20.276015+0000","last_became_active":"2024-09-15T21:55:58.813327+0000","last_became_peered":"2024-09-15T21:55:58.813327+0000","last_unstale":"2024-09-15T21:57:20.276015+0000","last_undegraded":"2024-09-15T21:57:20.276015+0000","last_fullsized":"2024-09-15T21:57:20.276015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:05.915940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.461118+0000","last_change":"2024-09-15T21:54:58.449381+0000","last_active":"2024-09-15T21:57:20.461118+0000","last_peered":"2024-09-15T21:57:20.461118+0000","last_clean":"2024-09-15T21:57:20.461118+0000","last_became_active":"2024-09-15T21:54:58.449062+0000","last_became_peered":"2024-09-15T21:54:58.449062+0000","last_unstale":"2024-09-15T21:57:20.461118+0000","last_undegraded":"2024-09-15T21:57:20.461118+0000","last_fullsized":"2024-09-15T21:57:20.461118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:39.712557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.286112+0000","last_change":"2024-09-15T21:54:58.440934+0000","last_active":"2024-09-15T21:57:20.286112+0000","last_peered":"2024-09-15T21:57:20.286112+0000","last_clean":"2024-09-15T21:57:20.286112+0000","last_became_active":"2024-09-15T21:54:58.440545+0000","last_became_peered":"2024-09-15T21:54:58.440545+0000","last_unstale":"2024-09-15T21:57:20.286112+0000","last_undegraded":"2024-09-15T21:57:20.286112+0000","last_fullsized":"2024-09-15T21:57:20.286112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:45.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.463385+0000","last_change":"2024-09-15T21:54:58.452528+0000","last_active":"2024-09-15T21:57:20.463385+0000","last_peered":"2024-09-15T21:57:20.463385+0000","last_clean":"2024-09-15T21:57:20.463385+0000","last_became_active":"2024-09-15T21:54:58.452219+0000","last_became_peered":"2024-09-15T21:54:58.452219+0000","last_unstale":"2024-09-15T21:57:20.463385+0000","last_undegraded":"2024-09-15T21:57:20.463385+0000","last_fullsized":"2024-09-15T21:57:20.463385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:22.423394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278238+0000","last_change":"2024-09-15T21:54:58.443347+0000","last_active":"2024-09-15T21:57:20.278238+0000","last_peered":"2024-09-15T21:57:20.278238+0000","last_clean":"2024-09-15T21:57:20.278238+0000","last_became_active":"2024-09-15T21:54:58.443047+0000","last_became_peered":"2024-09-15T21:54:58.443047+0000","last_unstale":"2024-09-15T21:57:20.278238+0000","last_undegraded":"2024-09-15T21:57:20.278238+0000","last_fullsized":"2024-09-15T21:57:20.278238+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:56.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.460583+0000","last_change":"2024-09-15T21:54:58.441242+0000","last_active":"2024-09-15T21:57:20.460583+0000","last_peered":"2024-09-15T21:57:20.460583+0000","last_clean":"2024-09-15T21:57:20.460583+0000","last_became_active":"2024-09-15T21:54:58.440900+0000","last_became_peered":"2024-09-15T21:54:58.440900+0000","last_unstale":"2024-09-15T21:57:20.460583+0000","last_undegraded":"2024-09-15T21:57:20.460583+0000","last_fullsized":"2024-09-15T21:57:20.460583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:33.767221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.285573+0000","last_change":"2024-09-15T21:54:58.434708+0000","last_active":"2024-09-15T21:57:20.285573+0000","last_peered":"2024-09-15T21:57:20.285573+0000","last_clean":"2024-09-15T21:57:20.285573+0000","last_became_active":"2024-09-15T21:54:58.434106+0000","last_became_peered":"2024-09-15T21:54:58.434106+0000","last_unstale":"2024-09-15T21:57:20.285573+0000","last_undegraded":"2024-09-15T21:57:20.285573+0000","last_fullsized":"2024-09-15T21:57:20.285573+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:14:16.952110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.278231+0000","last_change":"2024-09-15T21:54:58.450392+0000","last_active":"2024-09-15T21:57:20.278231+0000","last_peered":"2024-09-15T21:57:20.278231+0000","last_clean":"2024-09-15T21:57:20.278231+0000","last_became_active":"2024-09-15T21:54:58.450021+0000","last_became_peered":"2024-09-15T21:54:58.450021+0000","last_unstale":"2024-09-15T21:57:20.278231+0000","last_undegraded":"2024-09-15T21:57:20.278231+0000","last_fullsized":"2024-09-15T21:57:20.278231+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:38:59.020869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.190742+0000","last_change":"2024-09-15T21:54:58.448105+0000","last_active":"2024-09-15T21:57:20.190742+0000","last_peered":"2024-09-15T21:57:20.190742+0000","last_clean":"2024-09-15T21:57:20.190742+0000","last_became_active":"2024-09-15T21:54:58.447763+0000","last_became_peered":"2024-09-15T21:54:58.447763+0000","last_unstale":"2024-09-15T21:57:20.190742+0000","last_undegraded":"2024-09-15T21:57:20.190742+0000","last_fullsized":"2024-09-15T21:57:20.190742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:51.612481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467111+0000","last_change":"2024-09-15T21:54:58.445680+0000","last_active":"2024-09-15T21:57:20.467111+0000","last_peered":"2024-09-15T21:57:20.467111+0000","last_clean":"2024-09-15T21:57:20.467111+0000","last_became_active":"2024-09-15T21:54:58.445332+0000","last_became_peered":"2024-09-15T21:54:58.445332+0000","last_unstale":"2024-09-15T21:57:20.467111+0000","last_undegraded":"2024-09-15T21:57:20.467111+0000","last_fullsized":"2024-09-15T21:57:20.467111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:19:14.441344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.467430+0000","last_change":"2024-09-15T21:54:58.435188+0000","last_active":"2024-09-15T21:57:20.467430+0000","last_peered":"2024-09-15T21:57:20.467430+0000","last_clean":"2024-09-15T21:57:20.467430+0000","last_became_active":"2024-09-15T21:54:58.434860+0000","last_became_peered":"2024-09-15T21:54:58.434860+0000","last_unstale":"2024-09-15T21:57:20.467430+0000","last_undegraded":"2024-09-15T21:57:20.467430+0000","last_fullsized":"2024-09-15T21:57:20.467430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:52:10.679058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-15T21:57:20.472483+0000","last_change":"2024-09-15T21:54:58.452944+0000","last_active":"2024-09-15T21:57:20.472483+0000","last_peered":"2024-09-15T21:57:20.472483+0000","last_clean":"2024-09-15T21:57:20.472483+0000","last_became_active":"2024-09-15T21:54:58.452320+0000","last_became_peered":"2024-09-15T21:54:58.452320+0000","last_unstale":"2024-09-15T21:57:20.472483+0000","last_undegraded":"2024-09-15T21:57:20.472483+0000","last_fullsized":"2024-09-15T21:57:20.472483+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:54:57.410645+0000","last_clean_scrub_stamp":"2024-09-15T21:54:57.410645+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:05:47.271945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":804243,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1509,"num_read_kb":973,"num_write":515,"num_write_kb":2400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2777088,"data_stored":2412729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514540,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1081344,"data_stored":536154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":68,"seq":292057776176,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29096,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716056,"statfs":{"total":95995035648,"available":95965241344,"internally_reserved":0,"allocated":2121728,"data_stored":1531310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":63,"seq":270582939703,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1490944,"data_stored":923695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168635,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36796,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708356,"statfs":{"total":95995035648,"available":95957356544,"internally_reserved":0,"allocated":1486848,"data_stored":927587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":49,"seq":210453397569,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37004,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708148,"statfs":{"total":95995035648,"available":95957143552,"internally_reserved":0,"allocated":1695744,"data_stored":1133866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561092,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":530472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790027,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29132,"kb_used_data":1988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716020,"statfs":{"total":95995035648,"available":95965204480,"internally_reserved":0,"allocated":2035712,"data_stored":1456501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953552,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1085440,"data_stored":534939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1617920,"data_stored":1054603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1175552,"data_stored":606523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1155072,"data_stored":604141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":532462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":77166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":753664,"data_stored":721878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":332680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":403393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:57:36.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:57:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 -- 172.21.15.28:0/3673366044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102570 msgr2=0x7f46b01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 --2- 172.21.15.28:0/3673366044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102570 0x7f46b01029b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009900 tx=0x7f46a002a480 comp rx=0 tx=0).stop 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 -- 172.21.15.28:0/3673366044 shutdown_connections 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 --2- 172.21.15.28:0/3673366044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0102ef0 0x7f46b010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 --2- 172.21.15.28:0/3673366044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102570 0x7f46b01029b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 --2- 172.21.15.28:0/3673366044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0100e20 0x7f46b0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 -- 172.21.15.28:0/3673366044 >> 172.21.15.28:0/3673366044 conn(0x7f46b00fc7d0 msgr2=0x7f46b00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 -- 172.21.15.28:0/3673366044 shutdown_connections 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.452+0000 7f46b7a45640 1 -- 172.21.15.28:0/3673366044 wait complete. 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 Processor -- start 2024-09-15T21:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 -- start start 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0100e20 0x7f46b00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0102570 0x7f46b00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102ef0 0x7f46b00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0110f80 con 0x7f46b0102ef0 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0110e00 con 0x7f46b0102570 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b7a45640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0111100 con 0x7f46b0100e20 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102ef0 0x7f46b00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102ef0 0x7f46b00fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54312/0 (socket says 172.21.15.28:54312) 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 -- 172.21.15.28:0/4139655586 learned_addr learned my addr 172.21.15.28:0/4139655586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b4fe0640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0102570 0x7f46b00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b57e1640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0100e20 0x7f46b00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 -- 172.21.15.28:0/4139655586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0100e20 msgr2=0x7f46b00fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0100e20 0x7f46b00fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 -- 172.21.15.28:0/4139655586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0102570 msgr2=0x7f46b00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0102570 0x7f46b00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.453+0000 7f46b5fe2640 1 -- 172.21.15.28:0/4139655586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b01a6990 con 0x7f46b0102ef0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.454+0000 7f46b4fe0640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0102570 0x7f46b00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.454+0000 7f46b5fe2640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102ef0 0x7f46b00fd700 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f46a800eaa0 tx=0x7f46a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.454+0000 7f46a67fc640 1 -- 172.21.15.28:0/4139655586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a800cdb0 con 0x7f46b0102ef0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.454+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b01a6c80 con 0x7f46b0102ef0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.454+0000 7f46a67fc640 1 -- 172.21.15.28:0/4139655586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46a8005020 con 0x7f46b0102ef0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.454+0000 7f46a67fc640 1 -- 172.21.15.28:0/4139655586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a80116b0 con 0x7f46b0102ef0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.454+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b01a71c0 con 0x7f46b0102ef0 2024-09-15T21:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.455+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b0101200 con 0x7f46b0102ef0 2024-09-15T21:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.456+0000 7f46a67fc640 1 -- 172.21.15.28:0/4139655586 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f46a8005560 con 0x7f46b0102ef0 2024-09-15T21:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.457+0000 7f46a67fc640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f469c0a34d0 0x7f469c0a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.457+0000 7f46b57e1640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f469c0a34d0 0x7f469c0a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.457+0000 7f46a67fc640 1 -- 172.21.15.28:0/4139655586 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f46a80d05f0 con 0x7f46b0102ef0 2024-09-15T21:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.457+0000 7f46b57e1640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f469c0a34d0 0x7f469c0a5990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f46ac005fd0 tx=0x7f46ac005850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.461+0000 7f46a67fc640 1 -- 172.21.15.28:0/4139655586 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f46a8016030 con 0x7f46b0102ef0 2024-09-15T21:57:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:37 smithi028 ceph-mon[30969]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:37.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:37 smithi028 ceph-mon[30969]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-09-15T21:57:37.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.672+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46b010a0a0 con 0x7f46b0102ef0 2024-09-15T21:57:37.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.673+0000 7f46a67fc640 1 -- 172.21.15.28:0/4139655586 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f46a809ca20 con 0x7f46b0102ef0 2024-09-15T21:57:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:37.675 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:57:37.675 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:57:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f469c0a34d0 msgr2=0x7f469c0a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f469c0a34d0 0x7f469c0a5990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f46ac005fd0 tx=0x7f46ac005850 comp rx=0 tx=0).stop 2024-09-15T21:57:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102ef0 msgr2=0x7f46b00fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102ef0 0x7f46b00fd700 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f46a800eaa0 tx=0x7f46a800ef60 comp rx=0 tx=0).stop 2024-09-15T21:57:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 shutdown_connections 2024-09-15T21:57:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f469c0a34d0 0x7f469c0a5990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b0102ef0 0x7f46b00fd700 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46b0102570 0x7f46b00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 --2- 172.21.15.28:0/4139655586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0100e20 0x7f46b00fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 >> 172.21.15.28:0/4139655586 conn(0x7f46b00fc7d0 msgr2=0x7f46b006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 shutdown_connections 2024-09-15T21:57:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:37.675+0000 7f46b7a45640 1 -- 172.21.15.28:0/4139655586 wait complete. 2024-09-15T21:57:37.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:37 smithi203 ceph-mon[35546]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:37.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:37 smithi203 ceph-mon[35546]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-09-15T21:57:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:37 smithi174 ceph-mon[35244]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T21:57:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:37 smithi174 ceph-mon[35244]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-09-15T21:57:38.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:38.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:38 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4139655586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:38.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:38 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4139655586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:38.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:38 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4139655586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 -- 172.21.15.28:0/1412982159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 msgr2=0x7f04cc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 --2- 172.21.15.28:0/1412982159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc10ea90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009ca0 tx=0x7f04bc02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 -- 172.21.15.28:0/1412982159 shutdown_connections 2024-09-15T21:57:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 --2- 172.21.15.28:0/1412982159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc10ea90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 --2- 172.21.15.28:0/1412982159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04cc102790 0x7f04cc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 --2- 172.21.15.28:0/1412982159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04cc101de0 0x7f04cc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 -- 172.21.15.28:0/1412982159 >> 172.21.15.28:0/1412982159 conn(0x7f04cc0fd880 msgr2=0x7f04cc0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 -- 172.21.15.28:0/1412982159 shutdown_connections 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.090+0000 7f04d3a60640 1 -- 172.21.15.28:0/1412982159 wait complete. 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 Processor -- start 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 -- start start 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04cc101de0 0x7f04cc1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04cc102790 0x7f04cc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc111f70 con 0x7f04cc106f10 2024-09-15T21:57:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc111df0 con 0x7f04cc102790 2024-09-15T21:57:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.091+0000 7f04d3a60640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc1120f0 con 0x7f04cc101de0 2024-09-15T21:57:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc19cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54330/0 (socket says 172.21.15.28:54330) 2024-09-15T21:57:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 -- 172.21.15.28:0/2042204223 learned_addr learned my addr 172.21.15.28:0/2042204223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d17fc640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04cc101de0 0x7f04cc1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d0ffb640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04cc102790 0x7f04cc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 -- 172.21.15.28:0/2042204223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04cc101de0 msgr2=0x7f04cc1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04cc101de0 0x7f04cc1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 -- 172.21.15.28:0/2042204223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04cc102790 msgr2=0x7f04cc198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04cc102790 0x7f04cc198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 -- 172.21.15.28:0/2042204223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04cc19d380 con 0x7f04cc106f10 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d1ffd640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc19cca0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f04bc031a20 tx=0x7f04bc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04ca7fc640 1 -- 172.21.15.28:0/2042204223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc003b70 con 0x7f04cc106f10 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04cc19d610 con 0x7f04cc106f10 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04ca7fc640 1 -- 172.21.15.28:0/2042204223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04bc003d10 con 0x7f04cc106f10 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04ca7fc640 1 -- 172.21.15.28:0/2042204223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc02fc40 con 0x7f04cc106f10 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.092+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04cc1a4f50 con 0x7f04cc106f10 2024-09-15T21:57:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.093+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04cc1021c0 con 0x7f04cc106f10 2024-09-15T21:57:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.095+0000 7f04ca7fc640 1 -- 172.21.15.28:0/2042204223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f04bc004050 con 0x7f04cc106f10 2024-09-15T21:57:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.096+0000 7f04ca7fc640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f04b40a34d0 0x7f04b40a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.096+0000 7f04ca7fc640 1 -- 172.21.15.28:0/2042204223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f04bc0f15a0 con 0x7f04cc106f10 2024-09-15T21:57:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.096+0000 7f04d17fc640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f04b40a34d0 0x7f04b40a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.097+0000 7f04d17fc640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f04b40a34d0 0x7f04b40a5990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009bd0 tx=0x7f04c4005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.099+0000 7f04ca7fc640 1 -- 172.21.15.28:0/2042204223 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f04bc0bdb70 con 0x7f04cc106f10 2024-09-15T21:57:39.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.311+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04cc10a150 con 0x7f04cc106f10 2024-09-15T21:57:39.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.312+0000 7f04ca7fc640 1 -- 172.21.15.28:0/2042204223 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f04bc0bd470 con 0x7f04cc106f10 2024-09-15T21:57:39.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:39.314 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:57:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:57:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f04b40a34d0 msgr2=0x7f04b40a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f04b40a34d0 0x7f04b40a5990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009bd0 tx=0x7f04c4005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 msgr2=0x7f04cc19cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc19cca0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f04bc031a20 tx=0x7f04bc031a50 comp rx=0 tx=0).stop 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 shutdown_connections 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f04b40a34d0 0x7f04b40a5990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04cc106f10 0x7f04cc19cca0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04cc102790 0x7f04cc198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 --2- 172.21.15.28:0/2042204223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04cc101de0 0x7f04cc1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 >> 172.21.15.28:0/2042204223 conn(0x7f04cc0fd880 msgr2=0x7f04cc0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 shutdown_connections 2024-09-15T21:57:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:39.315+0000 7f04d3a60640 1 -- 172.21.15.28:0/2042204223 wait complete. 2024-09-15T21:57:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:39 smithi028 ceph-mon[30969]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 76 KiB/s wr, 52 op/s 2024-09-15T21:57:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:39 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2042204223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:39 smithi203 ceph-mon[35546]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 76 KiB/s wr, 52 op/s 2024-09-15T21:57:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:39 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2042204223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:39 smithi174 ceph-mon[35244]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 76 KiB/s wr, 52 op/s 2024-09-15T21:57:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:39 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2042204223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:39.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T21:57:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 -- 172.21.15.28:0/3305411741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c101de0 msgr2=0x7fe68c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 --2- 172.21.15.28:0/3305411741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c101de0 0x7fe68c1021c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009900 tx=0x7fe67c02a410 comp rx=0 tx=0).stop 2024-09-15T21:57:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 -- 172.21.15.28:0/3305411741 shutdown_connections 2024-09-15T21:57:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 --2- 172.21.15.28:0/3305411741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c106f10 0x7fe68c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 --2- 172.21.15.28:0/3305411741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe68c102790 0x7fe68c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 --2- 172.21.15.28:0/3305411741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c101de0 0x7fe68c1021c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 -- 172.21.15.28:0/3305411741 >> 172.21.15.28:0/3305411741 conn(0x7fe68c0fd8c0 msgr2=0x7fe68c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 -- 172.21.15.28:0/3305411741 shutdown_connections 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.756+0000 7fe6912d6640 1 -- 172.21.15.28:0/3305411741 wait complete. 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 Processor -- start 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 -- start start 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe68c101de0 0x7fe68c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c102790 0x7fe68c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c106f10 0x7fe68c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c111f90 con 0x7fe68c106f10 2024-09-15T21:57:40.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c111e10 con 0x7fe68c102790 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.757+0000 7fe6912d6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c112110 con 0x7fe68c101de0 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c106f10 0x7fe68c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c106f10 0x7fe68c19cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47452/0 (socket says 172.21.15.28:47452) 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 -- 172.21.15.28:0/454130284 learned_addr learned my addr 172.21.15.28:0/454130284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68affd640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe68c101de0 0x7fe68c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68a7fc640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c102790 0x7fe68c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 -- 172.21.15.28:0/454130284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe68c101de0 msgr2=0x7fe68c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe68c101de0 0x7fe68c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 -- 172.21.15.28:0/454130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c102790 msgr2=0x7fe68c198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c102790 0x7fe68c198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 -- 172.21.15.28:0/454130284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c19d480 con 0x7fe68c106f10 2024-09-15T21:57:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe68b7fe640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c106f10 0x7fe68c19cda0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe684009fd0 tx=0x7fe68400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe673fff640 1 -- 172.21.15.28:0/454130284 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684017070 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe673fff640 1 -- 172.21.15.28:0/454130284 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe684003e50 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c19d770 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe673fff640 1 -- 172.21.15.28:0/454130284 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6840106c0 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.758+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c1a5050 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.759+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c1021c0 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.761+0000 7fe673fff640 1 -- 172.21.15.28:0/454130284 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fe684003970 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.761+0000 7fe673fff640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fe6740a3550 0x7fe6740a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.761+0000 7fe68affd640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fe6740a3550 0x7fe6740a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.761+0000 7fe673fff640 1 -- 172.21.15.28:0/454130284 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fe6840d0060 con 0x7fe68c106f10 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.762+0000 7fe68affd640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fe6740a3550 0x7fe6740a5a10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009230 tx=0x7fe67c0058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.766+0000 7fe673fff640 1 -- 172.21.15.28:0/454130284 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe68409c490 con 0x7fe68c106f10 2024-09-15T21:57:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.943+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe68c109b60 con 0x7fe68c106f10 2024-09-15T21:57:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.944+0000 7fe673fff640 1 -- 172.21.15.28:0/454130284 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+25138 (secure 0 0 0) 0x7fe68409bd90 con 0x7fe68c106f10 2024-09-15T21:57:40.946 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:40.946 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T21:57:20.180529+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/972903625":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/3676950981":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/4057362541":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/955882315":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/1670367809":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6801/1670367809":"2024-09-16T21:57:20.180486+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:57:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fe6740a3550 msgr2=0x7fe6740a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fe6740a3550 0x7fe6740a5a10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009230 tx=0x7fe67c0058d0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c106f10 msgr2=0x7fe68c19cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c106f10 0x7fe68c19cda0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe684009fd0 tx=0x7fe68400b260 comp rx=0 tx=0).stop 2024-09-15T21:57:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 shutdown_connections 2024-09-15T21:57:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fe6740a3550 0x7fe6740a5a10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c106f10 0x7fe68c19cda0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe68c102790 0x7fe68c198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 --2- 172.21.15.28:0/454130284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe68c101de0 0x7fe68c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.946+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 >> 172.21.15.28:0/454130284 conn(0x7fe68c0fd8c0 msgr2=0x7fe68c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.947+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 shutdown_connections 2024-09-15T21:57:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:40.947+0000 7fe6912d6640 1 -- 172.21.15.28:0/454130284 wait complete. 2024-09-15T21:57:41.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:41 smithi028 ceph-mon[30969]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-09-15T21:57:41.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:41 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/454130284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:41.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-09-15T21:57:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:41 smithi203 ceph-mon[35546]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-09-15T21:57:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:41 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/454130284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:41 smithi174 ceph-mon[35244]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 76 KiB/s wr, 40 op/s 2024-09-15T21:57:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:41 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/454130284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T21:57:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:43.087 INFO:teuthology.orchestra.run.smithi028.stdout:{"run_dir":"/var/run/ceph"} 2024-09-15T21:57:43.088 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-15T21:57:43.088 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3393 2024-09-15T21:57:43.088 DEBUG:tasks.quiescer:Yielding 2024-09-15T21:57:43.088 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T21:57:43.104 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-15T21:57:43.105 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T21:57:43.105 INFO:tasks.workunit:timeout=3h 2024-09-15T21:57:43.105 INFO:tasks.workunit:cleanup=True 2024-09-15T21:57:43.106 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:43.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-15 21:55:45.421506200 +0000 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-15 21:55:45.421506200 +0000 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-15 21:57:17.982215652 +0000 2024-09-15T21:57:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-15 21:55:45.421506200 +0000 2024-09-15T21:57:43.126 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-15T21:57:43.126 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-15T21:57:43.201 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-15T21:57:43.258 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T21:57:43.373 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:43.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:43 smithi028 ceph-mon[30969]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-09-15T21:57:43.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:43 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T21:57:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:43 smithi203 ceph-mon[35546]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-09-15T21:57:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:43 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T21:57:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:43 smithi174 ceph-mon[35244]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 76 KiB/s wr, 44 op/s 2024-09-15T21:57:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:43 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T21:57:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 -- 172.21.15.28:0/235831177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90100e20 msgr2=0x7f9b90101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 --2- 172.21.15.28:0/235831177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90100e20 0x7f9b90101200 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009900 tx=0x7f9b8002a480 comp rx=0 tx=0).stop 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 -- 172.21.15.28:0/235831177 shutdown_connections 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 --2- 172.21.15.28:0/235831177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b90102ef0 0x7f9b9010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 --2- 172.21.15.28:0/235831177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90102570 0x7f9b901029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 --2- 172.21.15.28:0/235831177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90100e20 0x7f9b90101200 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 -- 172.21.15.28:0/235831177 >> 172.21.15.28:0/235831177 conn(0x7f9b900fc7d0 msgr2=0x7f9b900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 -- 172.21.15.28:0/235831177 shutdown_connections 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.877+0000 7f9b94dba640 1 -- 172.21.15.28:0/235831177 wait complete. 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 Processor -- start 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 -- start start 2024-09-15T21:57:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90100e20 0x7f9b90199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90102570 0x7f9b90199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b90102ef0 0x7f9b901a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90110f80 con 0x7f9b90102570 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90110e00 con 0x7f9b90102ef0 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b94dba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90111100 con 0x7f9b90100e20 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90102570 0x7f9b90199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90102570 0x7f9b90199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47484/0 (socket says 172.21.15.28:47484) 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b8dd9b640 1 -- 172.21.15.28:0/2873929976 learned_addr learned my addr 172.21.15.28:0/2873929976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.878+0000 7f9b8e59c640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90100e20 0x7f9b90199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8ed9d640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b90102ef0 0x7f9b901a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8e59c640 1 -- 172.21.15.28:0/2873929976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b90102ef0 msgr2=0x7f9b901a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8e59c640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b90102ef0 0x7f9b901a43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8e59c640 1 -- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90102570 msgr2=0x7f9b90199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8e59c640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90102570 0x7f9b90199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8e59c640 1 -- 172.21.15.28:0/2873929976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b901a4900 con 0x7f9b90100e20 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8dd9b640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90102570 0x7f9b90199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b8e59c640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90100e20 0x7f9b90199340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b80005940 tx=0x7f9b80003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b7f7fe640 1 -- 172.21.15.28:0/2873929976 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8003f070 con 0x7f9b90100e20 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b901a4ad0 con 0x7f9b90100e20 2024-09-15T21:57:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b7f7fe640 1 -- 172.21.15.28:0/2873929976 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b80031c30 con 0x7f9b90100e20 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b7f7fe640 1 -- 172.21.15.28:0/2873929976 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80035030 con 0x7f9b90100e20 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.879+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b901a4fb0 con 0x7f9b90100e20 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.880+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b90101200 con 0x7f9b90100e20 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.882+0000 7f9b7f7fe640 1 -- 172.21.15.28:0/2873929976 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f9b80042070 con 0x7f9b90100e20 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.883+0000 7f9b7f7fe640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9b780a34d0 0x7f9b780a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.883+0000 7f9b8dd9b640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9b780a34d0 0x7f9b780a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.883+0000 7f9b7f7fe640 1 -- 172.21.15.28:0/2873929976 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9b800f1260 con 0x7f9b90100e20 2024-09-15T21:57:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.883+0000 7f9b8dd9b640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9b780a34d0 0x7f9b780a5990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b9006ab00 tx=0x7f9b84005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:43.885+0000 7f9b7f7fe640 1 -- 172.21.15.28:0/2873929976 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9b80031de0 con 0x7f9b90100e20 2024-09-15T21:57:44.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.097+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b90109090 con 0x7f9b90100e20 2024-09-15T21:57:44.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.098+0000 7f9b7f7fe640 1 -- 172.21.15.28:0/2873929976 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f9b800bd0b0 con 0x7f9b90100e20 2024-09-15T21:57:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:57:44.100 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:57:44.100 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:57:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.100+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9b780a34d0 msgr2=0x7f9b780a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.100+0000 7f9b94dba640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9b780a34d0 0x7f9b780a5990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b9006ab00 tx=0x7f9b84005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.100+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90100e20 msgr2=0x7f9b90199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.100+0000 7f9b94dba640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90100e20 0x7f9b90199340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b80005940 tx=0x7f9b80003c10 comp rx=0 tx=0).stop 2024-09-15T21:57:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.100+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 shutdown_connections 2024-09-15T21:57:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.101+0000 7f9b94dba640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9b780a34d0 0x7f9b780a5990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.101+0000 7f9b94dba640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b90102ef0 0x7f9b901a43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.101+0000 7f9b94dba640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b90102570 0x7f9b90199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.101+0000 7f9b94dba640 1 --2- 172.21.15.28:0/2873929976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90100e20 0x7f9b90199340 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.101+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 >> 172.21.15.28:0/2873929976 conn(0x7f9b900fc7d0 msgr2=0x7f9b900fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.101+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 shutdown_connections 2024-09-15T21:57:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:44.101+0000 7f9b94dba640 1 -- 172.21.15.28:0/2873929976 wait complete. 2024-09-15T21:57:44.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:44 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2873929976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:44.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:44 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2873929976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:44.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:44 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2873929976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:57:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 -- 172.21.15.28:0/2271564537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 msgr2=0x7f50fc10deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 --2- 172.21.15.28:0/2271564537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc10deb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009900 tx=0x7f50ec02a410 comp rx=0 tx=0).stop 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 -- 172.21.15.28:0/2271564537 shutdown_connections 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 --2- 172.21.15.28:0/2271564537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc10e480 0x7f50fc110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 --2- 172.21.15.28:0/2271564537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc10deb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 --2- 172.21.15.28:0/2271564537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1008d0 0x7f50fc100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 -- 172.21.15.28:0/2271564537 >> 172.21.15.28:0/2271564537 conn(0x7f50fc0fc6b0 msgr2=0x7f50fc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 -- 172.21.15.28:0/2271564537 shutdown_connections 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.795+0000 7f51033e8640 1 -- 172.21.15.28:0/2271564537 wait complete. 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 Processor -- start 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 -- start start 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1008d0 0x7f50fc19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc10e480 0x7f50fc19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1140e0 con 0x7f50fc101280 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc113f60 con 0x7f50fc1008d0 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.796+0000 7f51033e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc114260 con 0x7f50fc10e480 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc19ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47498/0 (socket says 172.21.15.28:47498) 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5101184640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1008d0 0x7f50fc19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 -- 172.21.15.28:0/1143785543 learned_addr learned my addr 172.21.15.28:0/1143785543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5101985640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc10e480 0x7f50fc19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 -- 172.21.15.28:0/1143785543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc10e480 msgr2=0x7f50fc19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc10e480 0x7f50fc19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 -- 172.21.15.28:0/1143785543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1008d0 msgr2=0x7f50fc19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1008d0 0x7f50fc19a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 -- 172.21.15.28:0/1143785543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50fc19f5c0 con 0x7f50fc101280 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5101184640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1008d0 0x7f50fc19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f5100983640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc19ab10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f50ec006010 tx=0x7f50ec004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f50f27fc640 1 -- 172.21.15.28:0/1143785543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec03f030 con 0x7f50fc101280 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f50f27fc640 1 -- 172.21.15.28:0/1143785543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ec004440 con 0x7f50fc101280 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50fc19f850 con 0x7f50fc101280 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f50f27fc640 1 -- 172.21.15.28:0/1143785543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec035070 con 0x7f50fc101280 2024-09-15T21:57:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.797+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50fc1a7140 con 0x7f50fc101280 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f50f27fc640 1 -- 172.21.15.28:0/1143785543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f50ec042030 con 0x7f50fc101280 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 Processor -- start 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 -- start start 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2baf0 0x55c95ea2bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea36770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea36cb0 0x55c95ea390a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c95e9902f0 con 0x55c95ea2baf0 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c95e938270 con 0x55c95ea2c410 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.800+0000 7f5106262740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c95e98c2b0 con 0x55c95ea36cb0 2024-09-15T21:57:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2baf0 0x55c95ea2bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f50f27fc640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e80a3600 0x7f50e80a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2baf0 0x55c95ea2bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47506/0 (socket says 172.21.15.28:47506) 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 -- 172.21.15.28:0/3757457951 learned_addr learned my addr 172.21.15.28:0/3757457951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101184640 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea36770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5100983640 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea36cb0 0x55c95ea390a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e80a3600 0x7f50e80a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f50f27fc640 1 -- 172.21.15.28:0/1143785543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f50ec0f0270 con 0x7f50fc101280 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 -- 172.21.15.28:0/3757457951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea36cb0 msgr2=0x55c95ea390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea36cb0 0x55c95ea390a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 -- 172.21.15.28:0/3757457951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 msgr2=0x55c95ea36770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea36770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 -- 172.21.15.28:0/3757457951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c95e937340 con 0x55c95ea2baf0 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101184640 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea36770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f5101985640 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2baf0 0x55c95ea2bed0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f50f40129b0 tx=0x7f50f4012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53f7a85600e8232 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f50e77fe640 1 -- 172.21.15.28:0/3757457951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4004030 con 0x55c95ea2baf0 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f50e77fe640 1 -- 172.21.15.28:0/3757457951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f40046b0 con 0x55c95ea2baf0 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.801+0000 7f50e77fe640 1 -- 172.21.15.28:0/3757457951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f401fbb0 con 0x55c95ea2baf0 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 -- 172.21.15.28:0/3757457951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2baf0 msgr2=0x55c95ea2bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2baf0 0x55c95ea2bed0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f50f40129b0 tx=0x7f50f4012e70 comp rx=0 tx=0).stop 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5101985640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e80a3600 0x7f50e80a5ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50f4023000 tx=0x7f50f4021000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 -- 172.21.15.28:0/3757457951 shutdown_connections 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea36cb0 0x55c95ea390a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea36770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 --2- 172.21.15.28:0/3757457951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2baf0 0x55c95ea2bed0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 -- 172.21.15.28:0/3757457951 >> 172.21.15.28:0/3757457951 conn(0x55c95ea28060 msgr2=0x55c95ea28440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 -- 172.21.15.28:0/3757457951 shutdown_connections 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.802+0000 7f5106262740 1 -- 172.21.15.28:0/3757457951 wait complete. 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 -- start start 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 Processor -- start 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea33b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2c410 0x55c95ea340a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea36cb0 0x55c95ea345e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c95ea39880 con 0x55c95ea2c410 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c95e919290 con 0x55c95ea36cb0 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c95ea39a00 con 0x55c95ea2baf0 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5106262740 10 client.0 _ll_register_callbacks cb 0x55c95e94a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5100983640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2c410 0x55c95ea340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5100983640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2c410 0x55c95ea340a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47516/0 (socket says 172.21.15.28:47516) 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5100983640 1 -- 172.21.15.28:0/3505510445 learned_addr learned my addr 172.21.15.28:0/3505510445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5101985640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea36cb0 0x55c95ea345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.803+0000 7f5101184640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea33b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5100983640 1 -- 172.21.15.28:0/3505510445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 msgr2=0x55c95ea33b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5100983640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea33b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5100983640 1 -- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea36cb0 msgr2=0x55c95ea345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5100983640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea36cb0 0x55c95ea345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5100983640 1 -- 172.21.15.28:0/3505510445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c95e6a3210 con 0x55c95ea2c410 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5101184640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea33b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5100983640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2c410 0x55c95ea340a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0c47b0 tx=0x7f50ec0424f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0266484f08c93db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f50e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f50e4ff9640 1 -- 172.21.15.28:0/3505510445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec049030 con 0x55c95ea2c410 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f50e4ff9640 1 -- 172.21.15.28:0/3505510445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f50ec0b86e0 con 0x55c95ea2c410 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50f40046b0 con 0x55c95ea2c410 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f50e4ff9640 1 -- 172.21.15.28:0/3505510445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50ec042320 con 0x55c95ea2c410 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5106262740 10 client.15594 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c95e76f620 con 0x55c95ea2c410 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.804+0000 7f50e4ff9640 1 -- 172.21.15.28:0/3505510445 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f40046b0 con 0x55c95ea2c410 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f50e4ff9640 1 -- 172.21.15.28:0/3505510445 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f50ec036ca0 con 0x55c95ea2c410 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5106262740 10 client.15594 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5106262740 10 client.15594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5106262740 20 client.15594 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5106262740 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95ea48f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5106262740 4 client.15594 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c95e99da20 con 0x55c95ea46af0 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5101184640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95ea48f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f5101184640 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95ea48f40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e429f88b6e7546c server_cookie=8c3c81937fce4a87 in_seq=0 out_seq=0 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.805+0000 7f50e4ff9640 10 client.15594 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f50e4ff9640 1 -- 172.21.15.28:0/3505510445 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c95e99da20 con 0x55c95ea46af0 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f50e4ff9640 10 client.15594 handle_command_reply: tid=0 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 client.15594 shutdown 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2c410 msgr2=0x55c95ea340a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2c410 0x55c95ea340a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0c47b0 tx=0x7f50ec0424f0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 shutdown_connections 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95ea48f40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f50e4ff9640 0 client.15594 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea36cb0 0x55c95ea345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea2c410 0x55c95ea340a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 --2- 172.21.15.28:0/3505510445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea33b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 >> 172.21.15.28:0/3505510445 conn(0x55c95ea28060 msgr2=0x55c95ea2ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 shutdown_connections 2024-09-15T21:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.806+0000 7f5106262740 1 -- 172.21.15.28:0/3505510445 wait complete. 2024-09-15T21:57:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.807+0000 7f5106262740 20 client.15594 trim_cache size 0 max 16384 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 Processor -- start 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 -- start start 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2baf0 0x55c95ea35e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2c410 0x55c95ea34410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea4e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1b10a0 con 0x55c95ea36cb0 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1b2be0 con 0x55c95ea2baf0 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5106262740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1b1770 con 0x55c95ea2c410 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5101985640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea4e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5101184640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2baf0 0x55c95ea35e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5101985640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea4e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47524/0 (socket says 172.21.15.28:47524) 2024-09-15T21:57:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5101985640 1 -- 172.21.15.28:0/4263750059 learned_addr learned my addr 172.21.15.28:0/4263750059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5100983640 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2c410 0x55c95ea34410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.822+0000 7f5101184640 1 -- 172.21.15.28:0/4263750059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2c410 msgr2=0x55c95ea34410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f5101184640 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2c410 0x55c95ea34410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f5101184640 1 -- 172.21.15.28:0/4263750059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 msgr2=0x55c95ea4e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f5101184640 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea4e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f5101184640 1 -- 172.21.15.28:0/4263750059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c95ea2f700 con 0x55c95ea2baf0 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f5100983640 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2c410 0x55c95ea34410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f5101985640 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea4e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f5101184640 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2baf0 0x55c95ea35e90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55c95ea49790 tx=0x7f50f80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6330679270ff5430 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f50e6ffd640 1 -- 172.21.15.28:0/4263750059 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f80044e0 con 0x55c95ea2baf0 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f50e6ffd640 1 -- 172.21.15.28:0/4263750059 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f80125f0 con 0x55c95ea2baf0 2024-09-15T21:57:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.823+0000 7f50e6ffd640 1 -- 172.21.15.28:0/4263750059 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f801e650 con 0x55c95ea2baf0 2024-09-15T21:57:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 -- 172.21.15.28:0/4263750059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2baf0 msgr2=0x55c95ea35e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2baf0 0x55c95ea35e90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55c95ea49790 tx=0x7f50f80046c0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 -- 172.21.15.28:0/4263750059 shutdown_connections 2024-09-15T21:57:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea4e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2c410 0x55c95ea34410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 --2- 172.21.15.28:0/4263750059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2baf0 0x55c95ea35e90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 -- 172.21.15.28:0/4263750059 >> 172.21.15.28:0/4263750059 conn(0x55c95ea28060 msgr2=0x55c95ea271c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 -- 172.21.15.28:0/4263750059 shutdown_connections 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.824+0000 7f5106262740 1 -- 172.21.15.28:0/4263750059 wait complete. 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 -- start start 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 Processor -- start 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea3acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea3b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea3b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1b2510 con 0x55c95ea36cb0 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1b1e40 con 0x55c95ea2c410 2024-09-15T21:57:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1af080 con 0x55c95ea2baf0 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5106262740 10 client.0 _ll_register_callbacks cb 0x7f50ec005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.825+0000 7f5101985640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea3b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5100983640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea3b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea3b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47534/0 (socket says 172.21.15.28:47534) 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 -- 172.21.15.28:0/2244000928 learned_addr learned my addr 172.21.15.28:0/2244000928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101184640 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea3acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 -- 172.21.15.28:0/2244000928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 msgr2=0x55c95ea3acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea3acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 -- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 msgr2=0x55c95ea3b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea3b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 -- 172.21.15.28:0/2244000928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c95eab0ac0 con 0x55c95ea36cb0 2024-09-15T21:57:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5100983640 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea3b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5101985640 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea3b740 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f50f4005fd0 tx=0x7f50f401f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8a8fbc539ed51e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f50e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f50e4ff9640 1 -- 172.21.15.28:0/2244000928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4004a00 con 0x55c95ea36cb0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f50e4ff9640 1 -- 172.21.15.28:0/2244000928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f50f4004ba0 con 0x55c95ea36cb0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.826+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c95e997d70 con 0x55c95ea36cb0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f50e4ff9640 1 -- 172.21.15.28:0/2244000928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50f403f050 con 0x55c95ea36cb0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 10 client.15600 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c95eab1db0 con 0x55c95ea36cb0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f50e4ff9640 1 -- 172.21.15.28:0/2244000928 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c95e997d70 con 0x55c95ea36cb0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f50e4ff9640 1 -- 172.21.15.28:0/2244000928 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f50f401e070 con 0x55c95ea36cb0 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 10 client.15600 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 10 client.15600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 20 client.15600 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95eab4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 4 client.15600 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:57:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c95e914900 con 0x55c95ea46af0 2024-09-15T21:57:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.827+0000 7f5101184640 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95eab4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.828+0000 7f5101184640 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95eab4480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f738477342d8919c server_cookie=7342ea0e0f1e9085 in_seq=0 out_seq=0 2024-09-15T21:57:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.828+0000 7f50e4ff9640 10 client.15600 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f50e4ff9640 1 -- 172.21.15.28:0/2244000928 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c95e914900 con 0x55c95ea46af0 2024-09-15T21:57:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f50e4ff9640 10 client.15600 handle_command_reply: tid=0 2024-09-15T21:57:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 client.15600 shutdown 2024-09-15T21:57:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 msgr2=0x55c95ea3b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea3b740 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f50f4005fd0 tx=0x7f50f401f810 comp rx=0 tx=0).stop 2024-09-15T21:57:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 shutdown_connections 2024-09-15T21:57:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c95ea46af0 0x55c95eab4480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c95ea36cb0 0x55c95ea3b740 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f50e4ff9640 0 client.15600 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95ea2c410 0x55c95ea3b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 --2- 172.21.15.28:0/2244000928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c95ea2baf0 0x55c95ea3acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 >> 172.21.15.28:0/2244000928 conn(0x55c95ea28060 msgr2=0x55c95ea41b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 shutdown_connections 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 1 -- 172.21.15.28:0/2244000928 wait complete. 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.839+0000 7f5106262740 20 client.15600 trim_cache size 0 max 16384 2024-09-15T21:57:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T21:57:44.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T21:57:44.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T21:57:44.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e80a3600 msgr2=0x7f50e80a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e80a3600 0x7f50e80a5ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50f4023000 tx=0x7f50f4021000 comp rx=0 tx=0).stop 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 msgr2=0x7f50fc19ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc19ab10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f50ec006010 tx=0x7f50ec004040 comp rx=0 tx=0).stop 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 shutdown_connections 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e80a3600 0x7f50e80a5ac0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc10e480 0x7f50fc19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc101280 0x7f50fc19ab10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 --2- 172.21.15.28:0/1143785543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc1008d0 0x7f50fc19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 >> 172.21.15.28:0/1143785543 conn(0x7f50fc0fc6b0 msgr2=0x7f50fc0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 shutdown_connections 2024-09-15T21:57:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:44.840+0000 7f51033e8640 1 -- 172.21.15.28:0/1143785543 wait complete. 2024-09-15T21:57:44.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T21:54:59.048835+0000', 'modified': '2024-09-15T21:57:18.674828+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24485, 'mds_1': 24481, 'mds_2': 24460, 'mds_3': 24458, 'mds_4': 24454}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/3176985638', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 3176985638}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 3176985638}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/2641693590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 2641693590}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 2641693590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/2800102614', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 2800102614}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 2800102614}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6833/1988552520', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 1988552520}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 1988552520}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/280309786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 280309786}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 280309786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.203:6835/1524895518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 1524895518}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 1524895518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.174:6837/3327498469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3327498469}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3327498469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.203:6837/803844131', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 803844131}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 803844131}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.203:6839/3242294678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 3242294678}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 3242294678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24485': {'gid': 24485, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.174:6839/3735159621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3735159621}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3735159621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24485, 'qdb_cluster': [24454, 24458, 24460, 24481, 24485]} 2024-09-15T21:57:44.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-15T21:57:44.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-15T21:57:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:45.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:45 smithi028 ceph-mon[30969]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 -- 172.21.15.28:0/3091788257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb478106f20 msgr2=0x7fb47810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 --2- 172.21.15.28:0/3091788257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb478106f20 0x7fb47810eaa0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb470009ca0 tx=0x7fb47002a8c0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 -- 172.21.15.28:0/3091788257 shutdown_connections 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 --2- 172.21.15.28:0/3091788257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb478106f20 0x7fb47810eaa0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 --2- 172.21.15.28:0/3091788257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4781027a0 0x7fb4781067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 --2- 172.21.15.28:0/3091788257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb478101df0 0x7fb4781021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 -- 172.21.15.28:0/3091788257 >> 172.21.15.28:0/3091788257 conn(0x7fb4780fd8b0 msgr2=0x7fb4780ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 -- 172.21.15.28:0/3091788257 shutdown_connections 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.585+0000 7fb47f25f640 1 -- 172.21.15.28:0/3091788257 wait complete. 2024-09-15T21:57:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 Processor -- start 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 -- start start 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb478101df0 0x7fb478198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4781027a0 0x7fb4781988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb478106f20 0x7fb47819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb478111f80 con 0x7fb4781027a0 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb478111e00 con 0x7fb478101df0 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47f25f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb478112100 con 0x7fb478106f20 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4781027a0 0x7fb4781988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4781027a0 0x7fb4781988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47550/0 (socket says 172.21.15.28:47550) 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.586+0000 7fb47cffb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb478101df0 0x7fb478198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb477fff640 1 -- 172.21.15.28:0/1621834117 learned_addr learned my addr 172.21.15.28:0/1621834117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb478106f20 0x7fb47819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb477fff640 1 -- 172.21.15.28:0/1621834117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb478106f20 msgr2=0x7fb47819cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb477fff640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb478106f20 0x7fb47819cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb477fff640 1 -- 172.21.15.28:0/1621834117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb478101df0 msgr2=0x7fb478198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb477fff640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb478101df0 0x7fb478198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb477fff640 1 -- 172.21.15.28:0/1621834117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47819d340 con 0x7fb4781027a0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb47cffb640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb478101df0 0x7fb478198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb477fff640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4781027a0 0x7fb4781988d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb46400d950 tx=0x7fb46400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb475ffb640 1 -- 172.21.15.28:0/1621834117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464017070 con 0x7fb4781027a0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb475ffb640 1 -- 172.21.15.28:0/1621834117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb464009c30 con 0x7fb4781027a0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb47819d630 con 0x7fb4781027a0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb475ffb640 1 -- 172.21.15.28:0/1621834117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46400f6b0 con 0x7fb4781027a0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.587+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4781a4f10 con 0x7fb4781027a0 2024-09-15T21:57:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.590+0000 7fb475ffb640 1 -- 172.21.15.28:0/1621834117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fb4640109d0 con 0x7fb4781027a0 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.590+0000 7fb481638740 1 Processor -- start 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.590+0000 7fb481638740 1 -- start start 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb481638740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951d9d0 0x55b14951ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb481638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149528670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb481638740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14952afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb481638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b149484090 con 0x55b14951e2f0 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb481638740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b14942ba70 con 0x55b149528bb0 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb481638740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b149480050 con 0x55b14951d9d0 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb475ffb640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb4680a34d0 0x7fb4680a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47cffb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149528670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47d7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951d9d0 0x55b14951ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47d7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951d9d0 0x55b14951ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:41758/0 (socket says 172.21.15.28:41758) 2024-09-15T21:57:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47d7fc640 1 -- 172.21.15.28:0/3314474575 learned_addr learned my addr 172.21.15.28:0/3314474575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb477fff640 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14952afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb4680a34d0 0x7fb4680a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb475ffb640 1 -- 172.21.15.28:0/1621834117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb4640cfe70 con 0x7fb4781027a0 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47cffb640 1 -- 172.21.15.28:0/3314474575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951d9d0 msgr2=0x55b14951ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47cffb640 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951d9d0 0x55b14951ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47cffb640 1 -- 172.21.15.28:0/3314474575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 msgr2=0x55b14952afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47cffb640 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14952afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47cffb640 1 -- 172.21.15.28:0/3314474575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b149438130 con 0x55b14951e2f0 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951d9d0 0x55b14951ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47cffb640 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149528670 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009fd0 tx=0x7fb46c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f46d498c94420f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.591+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb4680a34d0 0x7fb4680a5990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb470009a60 tx=0x7fb470004a80 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb462ffd640 1 -- 172.21.15.28:0/3314474575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c004030 con 0x55b14951e2f0 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb462ffd640 1 -- 172.21.15.28:0/3314474575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb46c005500 con 0x55b14951e2f0 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb462ffd640 1 -- 172.21.15.28:0/3314474575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c004dd0 con 0x55b14951e2f0 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 -- 172.21.15.28:0/3314474575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 msgr2=0x55b149528670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149528670 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009fd0 tx=0x7fb46c0118d0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 -- 172.21.15.28:0/3314474575 shutdown_connections 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14952afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149528670 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 --2- 172.21.15.28:0/3314474575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951d9d0 0x55b14951ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 -- 172.21.15.28:0/3314474575 >> 172.21.15.28:0/3314474575 conn(0x55b149519f60 msgr2=0x55b14951a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 -- 172.21.15.28:0/3314474575 shutdown_connections 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.592+0000 7fb481638740 1 -- 172.21.15.28:0/3314474575 wait complete. 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 -- start start 2024-09-15T21:57:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 Processor -- start 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b1495283c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b149525710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b149525c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b14952b780 con 0x55b14951d9d0 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b14943ef20 con 0x55b149528bb0 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.593+0000 7fb481638740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b14952b900 con 0x55b14951e2f0 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb481638740 10 client.0 _ll_register_callbacks cb 0x55b149408030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b1495283c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b1495283c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47568/0 (socket says 172.21.15.28:47568) 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 -- 172.21.15.28:0/177933972 learned_addr learned my addr 172.21.15.28:0/177933972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb477fff640 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b149525710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b149525c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 -- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 msgr2=0x55b149525710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b149525710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 -- 172.21.15.28:0/177933972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 msgr2=0x55b149525c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b149525c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 -- 172.21.15.28:0/177933972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1491f52b0 con 0x55b14951d9d0 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb47cffb640 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b1495283c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009700 tx=0x7fb46c005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2fa34ab057f7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb457fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb457fff640 1 -- 172.21.15.28:0/177933972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c028020 con 0x55b14951d9d0 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb46c005500 con 0x55b14951d9d0 2024-09-15T21:57:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.594+0000 7fb457fff640 1 -- 172.21.15.28:0/177933972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb46c016040 con 0x55b14951d9d0 2024-09-15T21:57:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb457fff640 1 -- 172.21.15.28:0/177933972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb46c033050 con 0x55b14951d9d0 2024-09-15T21:57:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 10 client.15612 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b14953b420 con 0x55b14951d9d0 2024-09-15T21:57:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb457fff640 1 -- 172.21.15.28:0/177933972 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb46c005500 con 0x55b14951d9d0 2024-09-15T21:57:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb457fff640 1 -- 172.21.15.28:0/177933972 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb46c003b80 con 0x55b14951d9d0 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 10 client.15612 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 10 client.15612 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 20 client.15612 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b14953d860 0x55b14953ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 4 client.15612 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b149491600 con 0x55b14953d860 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.595+0000 7fb477fff640 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b14953d860 0x55b14953ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.596+0000 7fb477fff640 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b14953d860 0x55b14953ecd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=db0531bbf7f7057e server_cookie=ae8a072e95959f2a in_seq=0 out_seq=0 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.596+0000 7fb457fff640 10 client.15612 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.596+0000 7fb457fff640 1 -- 172.21.15.28:0/177933972 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b149491600 con 0x55b14953d860 2024-09-15T21:57:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.596+0000 7fb457fff640 10 client.15612 handle_command_reply: tid=0 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.596+0000 7fb481638740 1 client.15612 shutdown 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 msgr2=0x55b1495283c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b1495283c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009700 tx=0x7fb46c005730 comp rx=0 tx=0).stop 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 shutdown_connections 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b14953d860 0x55b14953ecd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b149525c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b149525710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 --2- 172.21.15.28:0/177933972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b1495283c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 >> 172.21.15.28:0/177933972 conn(0x55b149519f60 msgr2=0x55b14951fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb457fff640 0 client.15612 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 shutdown_connections 2024-09-15T21:57:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 1 -- 172.21.15.28:0/177933972 wait complete. 2024-09-15T21:57:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.597+0000 7fb481638740 20 client.15612 trim_cache size 0 max 16384 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 Processor -- start 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 -- start start 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b14953d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b14953e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14951eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b14953d010 con 0x55b14951d9d0 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b14953d6e0 con 0x55b149528bb0 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.615+0000 7fb481638740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b14953ddb0 con 0x55b14951e2f0 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47cffb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b14953d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47cffb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b14953d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47572/0 (socket says 172.21.15.28:47572) 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47cffb640 1 -- 172.21.15.28:0/901154654 learned_addr learned my addr 172.21.15.28:0/901154654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14951eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb477fff640 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b14953e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47d7fc640 1 -- 172.21.15.28:0/901154654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 msgr2=0x55b14953e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b14953e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47d7fc640 1 -- 172.21.15.28:0/901154654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 msgr2=0x55b14953d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b14953d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47d7fc640 1 -- 172.21.15.28:0/901154654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1495197f0 con 0x55b149528bb0 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47cffb640 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b14953d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14951eb70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb470006010 tx=0x7fb470036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b602e1921645b217 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb4627fc640 1 -- 172.21.15.28:0/901154654 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47002fc60 con 0x55b149528bb0 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb4627fc640 1 -- 172.21.15.28:0/901154654 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb470036070 con 0x55b149528bb0 2024-09-15T21:57:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.616+0000 7fb4627fc640 1 -- 172.21.15.28:0/901154654 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47004b700 con 0x55b149528bb0 2024-09-15T21:57:45.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 -- 172.21.15.28:0/901154654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 msgr2=0x55b14951eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14951eb70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb470006010 tx=0x7fb470036000 comp rx=0 tx=0).stop 2024-09-15T21:57:45.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 -- 172.21.15.28:0/901154654 shutdown_connections 2024-09-15T21:57:45.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b149528bb0 0x55b14951eb70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b14951e2f0 0x55b14953e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 --2- 172.21.15.28:0/901154654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951d9d0 0x55b14953d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 -- 172.21.15.28:0/901154654 >> 172.21.15.28:0/901154654 conn(0x55b149519f60 msgr2=0x55b14951fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 -- 172.21.15.28:0/901154654 shutdown_connections 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 -- 172.21.15.28:0/901154654 wait complete. 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.617+0000 7fb481638740 1 -- start start 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 1 Processor -- start 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b14951d9d0 0x55b14953fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149540410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b149528bb0 0x55b149540950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b14953eb50 con 0x55b14951e2f0 2024-09-15T21:57:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b14953e480 con 0x55b14951d9d0 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b14953f220 con 0x55b149528bb0 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb481638740 10 client.0 _ll_register_callbacks cb 0x7fb46400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149540410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149540410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47588/0 (socket says 172.21.15.28:47588) 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb477fff640 1 -- 172.21.15.28:0/4043230229 learned_addr learned my addr 172.21.15.28:0/4043230229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb47cffb640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b14951d9d0 0x55b14953fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.618+0000 7fb47d7fc640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b149528bb0 0x55b149540950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb47cffb640 1 -- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b149528bb0 msgr2=0x55b149540950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb47cffb640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b149528bb0 0x55b149540950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb47cffb640 1 -- 172.21.15.28:0/4043230229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 msgr2=0x55b149540410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb47cffb640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149540410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb47cffb640 1 -- 172.21.15.28:0/4043230229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1495a5f60 con 0x55b14951d9d0 2024-09-15T21:57:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb477fff640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149540410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb47cffb640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b14951d9d0 0x55b14953fed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00a010 tx=0x7fb46c011dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e2c543c87ec01a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb457fff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb457fff640 1 -- 172.21.15.28:0/4043230229 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c028070 con 0x55b14951d9d0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb457fff640 1 -- 172.21.15.28:0/4043230229 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb46c00f930 con 0x55b14951d9d0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb470036070 con 0x55b14951d9d0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb457fff640 1 -- 172.21.15.28:0/4043230229 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb46c00fb30 con 0x55b14951d9d0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb481638740 10 client.25256 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b1495a7240 con 0x55b14951d9d0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb457fff640 1 -- 172.21.15.28:0/4043230229 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb470036070 con 0x55b14951d9d0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb457fff640 1 -- 172.21.15.28:0/4043230229 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb46c004d20 con 0x55b14951d9d0 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.619+0000 7fb481638740 10 client.25256 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 10 client.25256 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 20 client.25256 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b1495abcb0 0x55b1495aa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 4 client.25256 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "status"}) -- 0x55b14948bb10 con 0x55b1495abcb0 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb477fff640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b1495abcb0 0x55b1495aa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb477fff640 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b1495abcb0 0x55b1495aa9c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eb790ea58380b003 server_cookie=48d1823ac234e5b3 in_seq=0 out_seq=0 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb457fff640 10 client.25256 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb457fff640 1 -- 172.21.15.28:0/4043230229 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b14948bb10 con 0x55b1495abcb0 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb457fff640 10 client.25256 handle_command_reply: tid=0 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 1 client.25256 shutdown 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b14951d9d0 msgr2=0x55b14953fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.620+0000 7fb481638740 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b14951d9d0 0x55b14953fed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00a010 tx=0x7fb46c011dd0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 shutdown_connections 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55b1495abcb0 0x55b1495aa9c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b149528bb0 0x55b149540950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b14951e2f0 0x55b149540410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 --2- 172.21.15.28:0/4043230229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b14951d9d0 0x55b14953fed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb457fff640 0 client.25256 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 >> 172.21.15.28:0/4043230229 conn(0x55b149519f60 msgr2=0x55b14951fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 shutdown_connections 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 1 -- 172.21.15.28:0/4043230229 wait complete. 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb481638740 20 client.25256 trim_cache size 0 max 16384 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:45.628 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24454, 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 128.069086613, 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 29, 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 194.42807625200001 2024-09-15T21:57:45.629 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb4680a34d0 msgr2=0x7fb4680a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb47f25f640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb4680a34d0 0x7fb4680a5990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb470009a60 tx=0x7fb470004a80 comp rx=0 tx=0).stop 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4781027a0 msgr2=0x7fb4781988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb47f25f640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4781027a0 0x7fb4781988d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb46400d950 tx=0x7fb46400de10 comp rx=0 tx=0).stop 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 shutdown_connections 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb47f25f640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb4680a34d0 0x7fb4680a5990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.621+0000 7fb47f25f640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb478106f20 0x7fb47819cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.622+0000 7fb47f25f640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4781027a0 0x7fb4781988d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.622+0000 7fb47f25f640 1 --2- 172.21.15.28:0/1621834117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb478101df0 0x7fb478198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.622+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 >> 172.21.15.28:0/1621834117 conn(0x7fb4780fd8b0 msgr2=0x7fb4780fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.622+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 shutdown_connections 2024-09-15T21:57:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:45.622+0000 7fb47f25f640 1 -- 172.21.15.28:0/1621834117 wait complete. 2024-09-15T21:57:45.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:45 smithi203 ceph-mon[35546]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-15T21:57:45.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:45 smithi174 ceph-mon[35244]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-15T21:57:46.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T21:57:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 -- 172.21.15.28:0/2048809806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 msgr2=0x7f261410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 --2- 172.21.15.28:0/2048809806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f261410ea90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f260c009ca0 tx=0x7f260c02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 -- 172.21.15.28:0/2048809806 shutdown_connections 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 --2- 172.21.15.28:0/2048809806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f261410ea90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 --2- 172.21.15.28:0/2048809806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2614102790 0x7f26141067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 --2- 172.21.15.28:0/2048809806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614101de0 0x7f26141021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 -- 172.21.15.28:0/2048809806 >> 172.21.15.28:0/2048809806 conn(0x7f26140fd8c0 msgr2=0x7f26140ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 -- 172.21.15.28:0/2048809806 shutdown_connections 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 -- 172.21.15.28:0/2048809806 wait complete. 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 Processor -- start 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.137+0000 7f261a5f3640 1 -- start start 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f261a5f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2614101de0 0x7f26140744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f261a5f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614102790 0x7f2614074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f261a5f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f2614074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f261a5f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2614111fc0 con 0x7f2614106f10 2024-09-15T21:57:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f261a5f3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2614111e40 con 0x7f2614102790 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f261a5f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2614112140 con 0x7f2614101de0 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f2614074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f2614074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47610/0 (socket says 172.21.15.28:47610) 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 -- 172.21.15.28:0/622203897 learned_addr learned my addr 172.21.15.28:0/622203897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2613fff640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2614101de0 0x7f26140744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f260bfff640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614102790 0x7f2614074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 -- 172.21.15.28:0/622203897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2614101de0 msgr2=0x7f26140744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2614101de0 0x7f26140744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 -- 172.21.15.28:0/622203897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614102790 msgr2=0x7f2614074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614102790 0x7f2614074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 -- 172.21.15.28:0/622203897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26141a67e0 con 0x7f2614106f10 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2618b90640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f2614074f70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f260c003f10 tx=0x7f260c003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:47.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2611ffb640 1 -- 172.21.15.28:0/622203897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c031650 con 0x7f2614106f10 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f2611ffb640 1 -- 172.21.15.28:0/622203897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f260c0317f0 con 0x7f2614106f10 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.138+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26141a6a70 con 0x7f2614106f10 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.139+0000 7f2611ffb640 1 -- 172.21.15.28:0/622203897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260c02fa90 con 0x7f2614106f10 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.139+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26141a7000 con 0x7f2614106f10 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.141+0000 7f2611ffb640 1 -- 172.21.15.28:0/622203897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f260c02fcb0 con 0x7f2614106f10 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.141+0000 7f261c9cc740 1 Processor -- start 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.141+0000 7f261c9cc740 1 -- start start 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.141+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cbe9d0 0x55a594cbedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.141+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.141+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cc9bb0 0x55a594ccbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.141+0000 7f261c9cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a594c25090 con 0x55a594cc9bb0 2024-09-15T21:57:47.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f261c9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a594bcca70 con 0x55a594cbe9d0 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f261c9cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a594c21050 con 0x55a594cbf2f0 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cc9bb0 0x55a594ccbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cc9bb0 0x55a594ccbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47626/0 (socket says 172.21.15.28:47626) 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 -- 172.21.15.28:0/3467503165 learned_addr learned my addr 172.21.15.28:0/3467503165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f2613fff640 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cbe9d0 0x55a594cbedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 -- 172.21.15.28:0/3467503165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 msgr2=0x55a594cc9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f2611ffb640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f26000a3550 0x7f26000a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 -- 172.21.15.28:0/3467503165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cbe9d0 msgr2=0x55a594cbedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cbe9d0 0x55a594cbedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 -- 172.21.15.28:0/3467503165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a594bd9130 con 0x55a594cc9bb0 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f2613fff640 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f2618b90640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f26000a3550 0x7f26000a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260bfff640 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cc9bb0 0x55a594ccbfa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f25fc012730 tx=0x7f25fc00b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6fc3f155935ba23 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f2611ffb640 1 -- 172.21.15.28:0/622203897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f260c0f15d0 con 0x7f2614106f10 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.142+0000 7f260a7fc640 1 -- 172.21.15.28:0/3467503165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc012d70 con 0x55a594cc9bb0 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f260a7fc640 1 -- 172.21.15.28:0/3467503165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25fc005020 con 0x55a594cc9bb0 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f260a7fc640 1 -- 172.21.15.28:0/3467503165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc004a00 con 0x55a594cc9bb0 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 -- 172.21.15.28:0/3467503165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cc9bb0 msgr2=0x55a594ccbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f2618b90640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f26000a3550 0x7f26000a5a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f260c0ee9b0 tx=0x7f260c08d150 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cc9bb0 0x55a594ccbfa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f25fc012730 tx=0x7f25fc00b6e0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 -- 172.21.15.28:0/3467503165 shutdown_connections 2024-09-15T21:57:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cc9bb0 0x55a594ccbfa0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 --2- 172.21.15.28:0/3467503165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cbe9d0 0x55a594cbedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 -- 172.21.15.28:0/3467503165 >> 172.21.15.28:0/3467503165 conn(0x55a594cbaf60 msgr2=0x55a594cbb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 -- 172.21.15.28:0/3467503165 shutdown_connections 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.143+0000 7f261c9cc740 1 -- 172.21.15.28:0/3467503165 wait complete. 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 -- start start 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 Processor -- start 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbe9d0 0x55a594cc93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cc6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a594ccc780 con 0x55a594cbe9d0 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a594bdff20 con 0x55a594cc9bb0 2024-09-15T21:57:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a594ccc900 con 0x55a594cbf2f0 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f261c9cc740 10 client.0 _ll_register_callbacks cb 0x55a594ba9030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f260bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbe9d0 0x55a594cc93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f260bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbe9d0 0x55a594cc93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47642/0 (socket says 172.21.15.28:47642) 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f260bfff640 1 -- 172.21.15.28:0/2008555420 learned_addr learned my addr 172.21.15.28:0/2008555420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f260bfff640 1 -- 172.21.15.28:0/2008555420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 msgr2=0x55a594cc6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.144+0000 7f2613fff640 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f260bfff640 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f260bfff640 1 -- 172.21.15.28:0/2008555420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 msgr2=0x55a594cc6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f260bfff640 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cc6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f260bfff640 1 -- 172.21.15.28:0/2008555420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5949962b0 con 0x55a594cbe9d0 2024-09-15T21:57:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f2613fff640 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f260bfff640 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbe9d0 0x55a594cc93f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f25fc006000 tx=0x7f25fc0051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13be29200438ae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f25f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f25f37fe640 1 -- 172.21.15.28:0/2008555420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc010830 con 0x55a594cbe9d0 2024-09-15T21:57:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f25f37fe640 1 -- 172.21.15.28:0/2008555420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f25fc0109d0 con 0x55a594cbe9d0 2024-09-15T21:57:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25fc005020 con 0x55a594cbe9d0 2024-09-15T21:57:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f25f37fe640 1 -- 172.21.15.28:0/2008555420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f25fc010bd0 con 0x55a594cbe9d0 2024-09-15T21:57:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f261c9cc740 10 client.15630 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a594cd8070 con 0x55a594cbe9d0 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.145+0000 7f25f37fe640 1 -- 172.21.15.28:0/2008555420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25fc005020 con 0x55a594cbe9d0 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f25f37fe640 1 -- 172.21.15.28:0/2008555420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f25fc01f540 con 0x55a594cbe9d0 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f261c9cc740 10 client.15630 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f261c9cc740 10 client.15630 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f261c9cc740 20 client.15630 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594cda430 0x55a594cdb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f261c9cc740 4 client.15630 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a594c32600 con 0x55a594cda430 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f2613fff640 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594cda430 0x55a594cdb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f25f37fe640 10 client.15630 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.146+0000 7f2613fff640 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594cda430 0x55a594cdb8a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=19e37b79e3f17b79 server_cookie=cb99fd22270ef329 in_seq=0 out_seq=0 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.147+0000 7f25f37fe640 1 -- 172.21.15.28:0/2008555420 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a594c32600 con 0x55a594cda430 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.147+0000 7f25f37fe640 10 client.15630 handle_command_reply: tid=0 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.147+0000 7f261c9cc740 1 client.15630 shutdown 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbe9d0 msgr2=0x55a594cc93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbe9d0 0x55a594cc93f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f25fc006000 tx=0x7f25fc0051d0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 shutdown_connections 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cc6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbf2f0 0x55a594cc6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594cda430 0x55a594cdb8a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2008555420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbe9d0 0x55a594cc93f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 >> 172.21.15.28:0/2008555420 conn(0x55a594cbaf60 msgr2=0x55a594cc0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f25f37fe640 0 client.15630 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 shutdown_connections 2024-09-15T21:57:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 1 -- 172.21.15.28:0/2008555420 wait complete. 2024-09-15T21:57:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.148+0000 7f261c9cc740 20 client.15630 trim_cache size 0 max 16384 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 Processor -- start 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 -- start start 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cda430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cd9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a594cd9be0 con 0x55a594cbf2f0 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a594cda2b0 con 0x55a594cc9bb0 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f261c9cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a594cda980 con 0x55a594cbe9d0 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f2613fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f2613fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47654/0 (socket says 172.21.15.28:47654) 2024-09-15T21:57:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.166+0000 7f2613fff640 1 -- 172.21.15.28:0/2815631742 learned_addr learned my addr 172.21.15.28:0/2815631742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f260bfff640 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cda430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2613fff640 1 -- 172.21.15.28:0/2815631742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 msgr2=0x55a594cda430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2613fff640 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cda430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2613fff640 1 -- 172.21.15.28:0/2815631742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 msgr2=0x55a594cd9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2613fff640 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cd9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2613fff640 1 -- 172.21.15.28:0/2815631742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a594cc6fb0 con 0x55a594cbf2f0 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f260bfff640 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cda430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2613fff640 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdb1d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x55a594cc5fd0 tx=0x7f2604010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c337bfeff56b9e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2609ffb640 1 -- 172.21.15.28:0/2815631742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604005020 con 0x55a594cbf2f0 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2609ffb640 1 -- 172.21.15.28:0/2815631742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26040047a0 con 0x55a594cbf2f0 2024-09-15T21:57:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.167+0000 7f2609ffb640 1 -- 172.21.15.28:0/2815631742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260401c840 con 0x55a594cbf2f0 2024-09-15T21:57:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 -- 172.21.15.28:0/2815631742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 msgr2=0x55a594cdb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdb1d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x55a594cc5fd0 tx=0x7f2604010800 comp rx=0 tx=0).stop 2024-09-15T21:57:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 -- 172.21.15.28:0/2815631742 shutdown_connections 2024-09-15T21:57:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cd9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdb1d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2815631742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cda430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 -- 172.21.15.28:0/2815631742 >> 172.21.15.28:0/2815631742 conn(0x55a594cbaf60 msgr2=0x55a594cbb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 -- 172.21.15.28:0/2815631742 shutdown_connections 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 -- 172.21.15.28:0/2815631742 wait complete. 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 -- start start 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.168+0000 7f261c9cc740 1 Processor -- start 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cdca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f261c9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cdd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f261c9cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a594cdb720 con 0x55a594cbf2f0 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f261c9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a594cdb050 con 0x55a594cc9bb0 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f261c9cc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a594cdbdf0 con 0x55a594cbe9d0 2024-09-15T21:57:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f261c9cc740 10 client.0 _ll_register_callbacks cb 0x7f25fc00bda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdcf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47666/0 (socket says 172.21.15.28:47666) 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 -- 172.21.15.28:0/2855416040 learned_addr learned my addr 172.21.15.28:0/2855416040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f260bfff640 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cdca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f260bfff640 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cdd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 -- 172.21.15.28:0/2855416040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 msgr2=0x55a594cdca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cdca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 -- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 msgr2=0x55a594cdd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cdd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 -- 172.21.15.28:0/2855416040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a594d42b30 con 0x55a594cbf2f0 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f2613fff640 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdcf80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f260400af50 tx=0x7f2604004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4beda49b0dee3a49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.169+0000 7f25f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f25f37fe640 1 -- 172.21.15.28:0/2855416040 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604005430 con 0x55a594cbf2f0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f25f37fe640 1 -- 172.21.15.28:0/2855416040 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f260401c5c0 con 0x55a594cbf2f0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26040047a0 con 0x55a594cbf2f0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f25f37fe640 1 -- 172.21.15.28:0/2855416040 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2604037050 con 0x55a594cbf2f0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a594d43d40 con 0x55a594cbf2f0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f25f37fe640 1 -- 172.21.15.28:0/2855416040 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26040047a0 con 0x55a594cbf2f0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f25f37fe640 1 -- 172.21.15.28:0/2855416040 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f26040276a0 con 0x55a594cbf2f0 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 10 client.15636 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 20 client.15636 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594d48580 0x55a594d47290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 4 client.15636 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.170+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "status"}) -- 0x55a594c2cb10 con 0x55a594d48580 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.171+0000 7f260bfff640 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594d48580 0x55a594d47290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.171+0000 7f260bfff640 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594d48580 0x55a594d47290 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=23de61cbaa540f6a server_cookie=7519521fce59dd9c in_seq=0 out_seq=0 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.171+0000 7f25f37fe640 10 client.15636 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.171+0000 7f25f37fe640 1 -- 172.21.15.28:0/2855416040 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55a594c2cb10 con 0x55a594d48580 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.171+0000 7f25f37fe640 10 client.15636 handle_command_reply: tid=0 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 client.15636 shutdown 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 msgr2=0x55a594cdcf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdcf80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f260400af50 tx=0x7f2604004b80 comp rx=0 tx=0).stop 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 shutdown_connections 2024-09-15T21:57:47.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a594cc9bb0 0x55a594cdd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a594d48580 0x55a594d47290 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a594cbf2f0 0x55a594cdcf80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 --2- 172.21.15.28:0/2855416040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a594cbe9d0 0x55a594cdca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f25f37fe640 0 client.15636 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 >> 172.21.15.28:0/2855416040 conn(0x55a594cbaf60 msgr2=0x55a594cc4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 shutdown_connections 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 1 -- 172.21.15.28:0/2855416040 wait complete. 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261c9cc740 20 client.15636 trim_cache size 0 max 16384 2024-09-15T21:57:47.179 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24458, 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 130.625460621, 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 29, 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-09-15T21:57:47.180 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T21:57:47.181 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 190.477673565 2024-09-15T21:57:47.181 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f26000a3550 msgr2=0x7f26000a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261a5f3640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f26000a3550 0x7f26000a5a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f260c0ee9b0 tx=0x7f260c08d150 comp rx=0 tx=0).stop 2024-09-15T21:57:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 msgr2=0x7f2614074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.172+0000 7f261a5f3640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f2614074f70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f260c003f10 tx=0x7f260c003f40 comp rx=0 tx=0).stop 2024-09-15T21:57:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 shutdown_connections 2024-09-15T21:57:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f26000a3550 0x7f26000a5a10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2614106f10 0x7f2614074f70 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2614102790 0x7f2614074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 --2- 172.21.15.28:0/622203897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2614101de0 0x7f26140744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 >> 172.21.15.28:0/622203897 conn(0x7f26140fd8c0 msgr2=0x7f26140ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 shutdown_connections 2024-09-15T21:57:47.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:47.173+0000 7f261a5f3640 1 -- 172.21.15.28:0/622203897 wait complete. 2024-09-15T21:57:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:47 smithi028 ceph-mon[30969]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-09-15T21:57:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:47 smithi203 ceph-mon[35546]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-09-15T21:57:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:47 smithi174 ceph-mon[35244]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-09-15T21:57:47.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-15T21:57:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 -- 172.21.15.28:0/61272784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4101de0 msgr2=0x7f76a41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 --2- 172.21.15.28:0/61272784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4101de0 0x7f76a41021c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7694009900 tx=0x7f769402a3f0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 -- 172.21.15.28:0/61272784 shutdown_connections 2024-09-15T21:57:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 --2- 172.21.15.28:0/61272784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a4106f10 0x7f76a410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 --2- 172.21.15.28:0/61272784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76a4102790 0x7f76a41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 --2- 172.21.15.28:0/61272784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4101de0 0x7f76a41021c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 -- 172.21.15.28:0/61272784 >> 172.21.15.28:0/61272784 conn(0x7f76a40fd8c0 msgr2=0x7f76a40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 -- 172.21.15.28:0/61272784 shutdown_connections 2024-09-15T21:57:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.588+0000 7f76a9138640 1 -- 172.21.15.28:0/61272784 wait complete. 2024-09-15T21:57:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.589+0000 7f76a9138640 1 Processor -- start 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.589+0000 7f76a9138640 1 -- start start 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.589+0000 7f76a9138640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76a4101de0 0x7f76a41983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.589+0000 7f76a9138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4102790 0x7f76a41988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.589+0000 7f76a9138640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a4106f10 0x7f76a419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a9138640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76a4111f80 con 0x7f76a4102790 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a9138640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f76a4111e00 con 0x7f76a4106f10 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a9138640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f76a4112100 con 0x7f76a4101de0 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4102790 0x7f76a41988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4102790 0x7f76a41988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47688/0 (socket says 172.21.15.28:47688) 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 -- 172.21.15.28:0/2846286444 learned_addr learned my addr 172.21.15.28:0/2846286444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76a4101de0 0x7f76a41983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a359e640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a4106f10 0x7f76a419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 -- 172.21.15.28:0/2846286444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76a4101de0 msgr2=0x7f76a41983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76a4101de0 0x7f76a41983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 -- 172.21.15.28:0/2846286444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a4106f10 msgr2=0x7f76a419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a4106f10 0x7f76a419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 -- 172.21.15.28:0/2846286444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a419d350 con 0x7f76a4102790 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f76a259c640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4102790 0x7f76a41988e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f769800c920 tx=0x7f769800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f768bfff640 1 -- 172.21.15.28:0/2846286444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698007c70 con 0x7f76a4102790 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.590+0000 7f768bfff640 1 -- 172.21.15.28:0/2846286444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7698007e10 con 0x7f76a4102790 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.591+0000 7f768bfff640 1 -- 172.21.15.28:0/2846286444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698018890 con 0x7f76a4102790 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.591+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a419d640 con 0x7f76a4102790 2024-09-15T21:57:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.591+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a41a4f20 con 0x7f76a4102790 2024-09-15T21:57:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.593+0000 7f768bfff640 1 -- 172.21.15.28:0/2846286444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f7698018a70 con 0x7f76a4102790 2024-09-15T21:57:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.593+0000 7f76ab511740 1 Processor -- start 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.593+0000 7f76ab511740 1 -- start start 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76ab511740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76ab511740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76ab511740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76ab511740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce68b33090 con 0x55ce68bcd2f0 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76ab511740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce68adaa70 con 0x55ce68bcc9d0 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76ab511740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ce68b2f050 con 0x55ce68bd7bb0 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47700/0 (socket says 172.21.15.28:47700) 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 -- 172.21.15.28:0/3822043542 learned_addr learned my addr 172.21.15.28:0/3822043542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f768bfff640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f768c0a34d0 0x7f768c0a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a359e640 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a359e640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f768c0a34d0 0x7f768c0a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f768bfff640 1 -- 172.21.15.28:0/2846286444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f76980d0480 con 0x7f76a4102790 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a259c640 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 -- 172.21.15.28:0/3822043542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 msgr2=0x55ce68bd9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 -- 172.21.15.28:0/3822043542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 msgr2=0x55ce68bccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 -- 172.21.15.28:0/3822043542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce68ae7130 con 0x55ce68bcd2f0 2024-09-15T21:57:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd7670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7694031550 tx=0x7f76940041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b0321dee97425a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.594+0000 7f76a359e640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f768c0a34d0 0x7f768c0a5990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f769c005c40 tx=0x7f769c005bd0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76897fa640 1 -- 172.21.15.28:0/3822043542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694035040 con 0x55ce68bcd2f0 2024-09-15T21:57:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76897fa640 1 -- 172.21.15.28:0/3822043542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7694005520 con 0x55ce68bcd2f0 2024-09-15T21:57:48.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76897fa640 1 -- 172.21.15.28:0/3822043542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76940037f0 con 0x55ce68bcd2f0 2024-09-15T21:57:48.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 -- 172.21.15.28:0/3822043542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 msgr2=0x55ce68bd7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd7670 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7694031550 tx=0x7f76940041a0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 -- 172.21.15.28:0/3822043542 shutdown_connections 2024-09-15T21:57:48.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd7670 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 --2- 172.21.15.28:0/3822043542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 -- 172.21.15.28:0/3822043542 >> 172.21.15.28:0/3822043542 conn(0x55ce68bc8f60 msgr2=0x55ce68bc9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.595+0000 7f76ab511740 1 -- 172.21.15.28:0/3822043542 shutdown_connections 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 -- 172.21.15.28:0/3822043542 wait complete. 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 -- start start 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 Processor -- start 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bd7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bdeb90 con 0x55ce68bcd2f0 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce68aedf20 con 0x55ce68bcc9d0 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bded10 con 0x55ce68bd7bb0 2024-09-15T21:57:48.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76ab511740 10 client.0 _ll_register_callbacks cb 0x55ce68ab7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76a259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47714/0 (socket says 172.21.15.28:47714) 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76a259c640 1 -- 172.21.15.28:0/4238992538 learned_addr learned my addr 172.21.15.28:0/4238992538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.596+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bd7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a359e640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a259c640 1 -- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 msgr2=0x55ce68bd4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a259c640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a259c640 1 -- 172.21.15.28:0/4238992538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 msgr2=0x55ce68bd7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a259c640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bd7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a259c640 1 -- 172.21.15.28:0/4238992538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce688a42b0 con 0x55ce68bcd2f0 2024-09-15T21:57:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bd7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76a259c640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd4760 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7698098610 tx=0x7f769809c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4ce29d3231dfb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f7686ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f7686ff5640 1 -- 172.21.15.28:0/4238992538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76980a4e20 con 0x55ce68bcd2f0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f7686ff5640 1 -- 172.21.15.28:0/4238992538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f769800fb30 con 0x55ce68bcd2f0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7694005520 con 0x55ce68bcd2f0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f7686ff5640 1 -- 172.21.15.28:0/4238992538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76980d0170 con 0x55ce68bcd2f0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76ab511740 10 client.15648 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce68bea4e0 con 0x55ce68bcd2f0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.597+0000 7f7686ff5640 1 -- 172.21.15.28:0/4238992538 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7694005520 con 0x55ce68bcd2f0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f7686ff5640 1 -- 172.21.15.28:0/4238992538 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f76980a4990 con 0x55ce68bcd2f0 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76ab511740 10 client.15648 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:48.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76ab511740 10 client.15648 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76ab511740 20 client.15648 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76ab511740 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68becab0 0x55ce68bedf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76ab511740 4 client.15648 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce68b40600 con 0x55ce68becab0 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68becab0 0x55ce68bedf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68becab0 0x55ce68bedf20 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c85cbbc35eeb8e6 server_cookie=adf6561d5748bca9 in_seq=0 out_seq=0 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.598+0000 7f7686ff5640 10 client.15648 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f7686ff5640 1 -- 172.21.15.28:0/4238992538 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ce68b40600 con 0x55ce68becab0 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f7686ff5640 10 client.15648 handle_command_reply: tid=0 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 client.15648 shutdown 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 msgr2=0x55ce68bd4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd4760 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7698098610 tx=0x7f769809c970 comp rx=0 tx=0).stop 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 shutdown_connections 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68becab0 0x55ce68bedf20 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcd2f0 0x55ce68bd4760 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 --2- 172.21.15.28:0/4238992538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcc9d0 0x55ce68bd7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f7686ff5640 0 client.15648 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 >> 172.21.15.28:0/4238992538 conn(0x55ce68bc8f60 msgr2=0x55ce68bcec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 shutdown_connections 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.599+0000 7f76ab511740 1 -- 172.21.15.28:0/4238992538 wait complete. 2024-09-15T21:57:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.600+0000 7f76ab511740 20 client.15648 trim_cache size 0 max 16384 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 Processor -- start 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 -- start start 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcd2f0 0x55ce68bedf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bec260 con 0x55ce68bcc9d0 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bec930 con 0x55ce68bcd2f0 2024-09-15T21:57:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.617+0000 7f76ab511740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bed000 con 0x55ce68bd7bb0 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bec3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bec3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47724/0 (socket says 172.21.15.28:47724) 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 -- 172.21.15.28:0/2557686669 learned_addr learned my addr 172.21.15.28:0/2557686669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a359e640 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a259c640 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcd2f0 0x55ce68bedf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 -- 172.21.15.28:0/2557686669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 msgr2=0x55ce68bd4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 -- 172.21.15.28:0/2557686669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcd2f0 msgr2=0x55ce68bedf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcd2f0 0x55ce68bedf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 -- 172.21.15.28:0/2557686669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce68bcd120 con 0x55ce68bcc9d0 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a359e640 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bec3e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7694031550 tx=0x7f7694000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b2e93a2112db34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f7688ff9640 1 -- 172.21.15.28:0/2557686669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694035040 con 0x55ce68bcc9d0 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f7688ff9640 1 -- 172.21.15.28:0/2557686669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7694054040 con 0x55ce68bcc9d0 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.618+0000 7f7688ff9640 1 -- 172.21.15.28:0/2557686669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694031930 con 0x55ce68bcc9d0 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 -- 172.21.15.28:0/2557686669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 msgr2=0x55ce68bec3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bec3e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7694031550 tx=0x7f7694000bc0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 -- 172.21.15.28:0/2557686669 shutdown_connections 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bd7bb0 0x55ce68bd4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bcd2f0 0x55ce68bedf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 --2- 172.21.15.28:0/2557686669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bec3e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 -- 172.21.15.28:0/2557686669 >> 172.21.15.28:0/2557686669 conn(0x55ce68bc8f60 msgr2=0x55ce68bd4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 -- 172.21.15.28:0/2557686669 shutdown_connections 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 -- 172.21.15.28:0/2557686669 wait complete. 2024-09-15T21:57:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 -- start start 2024-09-15T21:57:48.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.619+0000 7f76ab511740 1 Processor -- start 2024-09-15T21:57:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76ab511740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bf10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76ab511740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bcd2f0 0x55ce68bf15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76ab511740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bd7bb0 0x55ce68bf1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76ab511740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bedda0 con 0x55ce68bcc9d0 2024-09-15T21:57:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bf10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76ab511740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bed6d0 con 0x55ce68bd7bb0 2024-09-15T21:57:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bf10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47740/0 (socket says 172.21.15.28:47740) 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76ab511740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ce68bee470 con 0x55ce68bcd2f0 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 -- 172.21.15.28:0/675274194 learned_addr learned my addr 172.21.15.28:0/675274194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a259c640 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bcd2f0 0x55ce68bf15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76ab511740 10 client.0 _ll_register_callbacks cb 0x7f769800cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a359e640 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bd7bb0 0x55ce68bf1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 -- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bcd2f0 msgr2=0x55ce68bf15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bcd2f0 0x55ce68bf15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 -- 172.21.15.28:0/675274194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bd7bb0 msgr2=0x55ce68bf1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bd7bb0 0x55ce68bf1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 -- 172.21.15.28:0/675274194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce68c55350 con 0x55ce68bcc9d0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f76a2d9d640 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bf10b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7694009900 tx=0x7f7694004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c899dd90b90076d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.620+0000 7f7686ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f7686ff5640 1 -- 172.21.15.28:0/675274194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694035070 con 0x55ce68bcc9d0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f7686ff5640 1 -- 172.21.15.28:0/675274194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f76940046c0 con 0x55ce68bcc9d0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7694054040 con 0x55ce68bcc9d0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f7686ff5640 1 -- 172.21.15.28:0/675274194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f769406a050 con 0x55ce68bcc9d0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 10 client.15654 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce68c56560 con 0x55ce68bcc9d0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f7686ff5640 1 -- 172.21.15.28:0/675274194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7694054040 con 0x55ce68bcc9d0 2024-09-15T21:57:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f7686ff5640 1 -- 172.21.15.28:0/675274194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7694004a50 con 0x55ce68bcc9d0 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 10 client.15654 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 10 client.15654 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 20 client.15654 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68c5a910 0x55ce68c59620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 4 client.15654 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.621+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "status"}) -- 0x55ce68b3ab10 con 0x55ce68c5a910 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76a259c640 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68c5a910 0x55ce68c59620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76a259c640 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68c5a910 0x55ce68c59620 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef94ab3054971568 server_cookie=1d126617d1902a69 in_seq=0 out_seq=0 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f7686ff5640 10 client.15654 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f7686ff5640 1 -- 172.21.15.28:0/675274194 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ce68b3ab10 con 0x55ce68c5a910 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f7686ff5640 10 client.15654 handle_command_reply: tid=0 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 client.15654 shutdown 2024-09-15T21:57:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 msgr2=0x55ce68bf10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bf10b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7694009900 tx=0x7f7694004270 comp rx=0 tx=0).stop 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 shutdown_connections 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55ce68c5a910 0x55ce68c59620 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ce68bd7bb0 0x55ce68bf1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ce68bcd2f0 0x55ce68bf15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f7686ff5640 0 client.15654 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 --2- 172.21.15.28:0/675274194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce68bcc9d0 0x55ce68bf10b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.622+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 >> 172.21.15.28:0/675274194 conn(0x55ce68bc8f60 msgr2=0x55ce68bc9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 shutdown_connections 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76ab511740 1 -- 172.21.15.28:0/675274194 wait complete. 2024-09-15T21:57:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76ab511740 20 client.15654 trim_cache size 0 max 16384 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24460, 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 133.088844959, 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 29, 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-09-15T21:57:48.632 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T21:57:48.633 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 194.79894750899999 2024-09-15T21:57:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f768c0a34d0 msgr2=0x7f768c0a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f768c0a34d0 0x7f768c0a5990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f769c005c40 tx=0x7f769c005bd0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4102790 msgr2=0x7f76a41988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4102790 0x7f76a41988e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f769800c920 tx=0x7f769800cde0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 shutdown_connections 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f768c0a34d0 0x7f768c0a5990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a4106f10 0x7f76a419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a4102790 0x7f76a41988e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 --2- 172.21.15.28:0/2846286444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76a4101de0 0x7f76a41983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 >> 172.21.15.28:0/2846286444 conn(0x7f76a40fd8c0 msgr2=0x7f76a40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 shutdown_connections 2024-09-15T21:57:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:48.623+0000 7f76a9138640 1 -- 172.21.15.28:0/2846286444 wait complete. 2024-09-15T21:57:49.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T21:57:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:49 smithi028 ceph-mon[30969]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-09-15T21:57:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:49 smithi203 ceph-mon[35546]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-09-15T21:57:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:49 smithi174 ceph-mon[35244]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-09-15T21:57:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 -- 172.21.15.28:0/2223179000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1036e0 msgr2=0x7f5aac107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2223179000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1036e0 0x7f5aac107b90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009930 tx=0x7f5aa002a420 comp rx=0 tx=0).stop 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 -- 172.21.15.28:0/2223179000 shutdown_connections 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2223179000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aac1082a0 0x7f5aac10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2223179000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1036e0 0x7f5aac107b90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2223179000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5aac102d30 0x7f5aac103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 -- 172.21.15.28:0/2223179000 >> 172.21.15.28:0/2223179000 conn(0x7f5aac0fc7d0 msgr2=0x7f5aac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.195+0000 7f5ab2218640 1 -- 172.21.15.28:0/2223179000 shutdown_connections 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.196+0000 7f5ab2218640 1 -- 172.21.15.28:0/2223179000 wait complete. 2024-09-15T21:57:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.196+0000 7f5ab2218640 1 Processor -- start 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.196+0000 7f5ab2218640 1 -- start start 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.196+0000 7f5ab2218640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aac102d30 0x7f5aac198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5ab2218640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5aac1036e0 0x7f5aac198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5ab2218640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1082a0 0x7f5aac19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5ab2218640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac1120a0 con 0x7f5aac1082a0 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5ab2218640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac111f20 con 0x7f5aac1036e0 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5ab2218640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac112220 con 0x7f5aac102d30 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1082a0 0x7f5aac19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1082a0 0x7f5aac19cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47764/0 (socket says 172.21.15.28:47764) 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 -- 172.21.15.28:0/2225176512 learned_addr learned my addr 172.21.15.28:0/2225176512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5aac1036e0 0x7f5aac198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aac102d30 0x7f5aac198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 -- 172.21.15.28:0/2225176512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aac102d30 msgr2=0x7f5aac198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aac102d30 0x7f5aac198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 -- 172.21.15.28:0/2225176512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5aac1036e0 msgr2=0x7f5aac198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5aac1036e0 0x7f5aac198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 -- 172.21.15.28:0/2225176512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac19d5f0 con 0x7f5aac1082a0 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5aac1036e0 0x7f5aac198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aabfff640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1082a0 0x7f5aac19cf10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400eaa0 tx=0x7f5aa400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aa8ff9640 1 -- 172.21.15.28:0/2225176512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa400cdb0 con 0x7f5aac1082a0 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aa8ff9640 1 -- 172.21.15.28:0/2225176512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa4005020 con 0x7f5aac1082a0 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.197+0000 7f5aa8ff9640 1 -- 172.21.15.28:0/2225176512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa40116b0 con 0x7f5aac1082a0 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.198+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aac19d8e0 con 0x7f5aac1082a0 2024-09-15T21:57:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.198+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aac1a51c0 con 0x7f5aac1082a0 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.200+0000 7f5aa8ff9640 1 -- 172.21.15.28:0/2225176512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f5aa4005560 con 0x7f5aac1082a0 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.200+0000 7f5ab45f1740 1 Processor -- start 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.200+0000 7f5ab45f1740 1 -- start start 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.200+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e21cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.200+0000 7f5aa8ff9640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a940a34d0 0x7f5a940a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.200+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e30a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e333b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559451d88090 con 0x559451e221f0 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559451d2fa70 con 0x559451e30fc0 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559451d84050 con 0x559451e218d0 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a940a34d0 0x7f5a940a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aa8ff9640 1 -- 172.21.15.28:0/2225176512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f5aa40d0440 con 0x7f5aac1082a0 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e30a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e30a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47766/0 (socket says 172.21.15.28:47766) 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 -- 172.21.15.28:0/733092004 learned_addr learned my addr 172.21.15.28:0/733092004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aabfff640 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e21cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 -- 172.21.15.28:0/733092004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 msgr2=0x559451e21cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e21cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 -- 172.21.15.28:0/733092004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 msgr2=0x559451e333b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e333b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.201+0000 7f5aaaffd640 1 -- 172.21.15.28:0/733092004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559451d3c130 con 0x559451e221f0 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.202+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e30a80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009930 tx=0x7f5aa0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a25e54173473318 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.202+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a940a34d0 0x7f5a940a5990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009bd0 tx=0x7f5a9c005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.202+0000 7f5a91ffb640 1 -- 172.21.15.28:0/733092004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0035040 con 0x559451e221f0 2024-09-15T21:57:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.202+0000 7f5a91ffb640 1 -- 172.21.15.28:0/733092004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa00035f0 con 0x559451e221f0 2024-09-15T21:57:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.202+0000 7f5a91ffb640 1 -- 172.21.15.28:0/733092004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa003d6e0 con 0x559451e221f0 2024-09-15T21:57:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 -- 172.21.15.28:0/733092004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 msgr2=0x559451e30a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e30a80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009930 tx=0x7f5aa0005b50 comp rx=0 tx=0).stop 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 -- 172.21.15.28:0/733092004 shutdown_connections 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e333b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e30a80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/733092004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e21cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 -- 172.21.15.28:0/733092004 >> 172.21.15.28:0/733092004 conn(0x559451e1df60 msgr2=0x559451e1e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 -- 172.21.15.28:0/733092004 shutdown_connections 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 -- 172.21.15.28:0/733092004 wait complete. 2024-09-15T21:57:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 -- start start 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.203+0000 7f5ab45f1740 1 Processor -- start 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e218d0 0x559451e30910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e221f0 0x559451e2dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e30fc0 0x559451e2e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559451e33b90 con 0x559451e218d0 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559451d42f20 con 0x559451e221f0 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559451e33d10 con 0x559451e30fc0 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5ab45f1740 10 client.0 _ll_register_callbacks cb 0x559451d0c030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e218d0 0x559451e30910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e218d0 0x559451e30910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47774/0 (socket says 172.21.15.28:47774) 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 -- 172.21.15.28:0/3370183634 learned_addr learned my addr 172.21.15.28:0/3370183634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aabfff640 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e30fc0 0x559451e2e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e221f0 0x559451e2dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 -- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e30fc0 msgr2=0x559451e2e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e30fc0 0x559451e2e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 -- 172.21.15.28:0/3370183634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e221f0 msgr2=0x559451e2dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e221f0 0x559451e2dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 -- 172.21.15.28:0/3370183634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559451af92b0 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e218d0 0x559451e30910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00ecd0 tx=0x7f5a9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5583ee9f3c2f39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.204+0000 7f5a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5a8affd640 1 -- 172.21.15.28:0/3370183634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c01ccb0 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5a8affd640 1 -- 172.21.15.28:0/3370183634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f5a9c004020 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa00035f0 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5a8affd640 1 -- 172.21.15.28:0/3370183634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5a9c030050 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 10 client.15666 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559451e3f600 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5a8affd640 1 -- 172.21.15.28:0/3370183634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa00035f0 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5a8affd640 1 -- 172.21.15.28:0/3370183634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5a9c027430 con 0x559451e218d0 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 10 client.15666 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 10 client.15666 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T21:57:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 20 client.15666 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451e43080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 4 client.15666 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.205+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559451d95600 con 0x559451e41c10 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.206+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451e43080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.206+0000 7f5a8affd640 10 client.15666 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.206+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451e43080 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6677c154668dd147 server_cookie=9aaef8733fb95619 in_seq=0 out_seq=0 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5a8affd640 1 -- 172.21.15.28:0/3370183634 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559451d95600 con 0x559451e41c10 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5a8affd640 10 client.15666 handle_command_reply: tid=0 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 client.15666 shutdown 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e218d0 msgr2=0x559451e30910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e218d0 0x559451e30910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00ecd0 tx=0x7f5a9c004730 comp rx=0 tx=0).stop 2024-09-15T21:57:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 shutdown_connections 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451e43080 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e30fc0 0x559451e2e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e221f0 0x559451e2dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5a8affd640 0 client.15666 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/3370183634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e218d0 0x559451e30910 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 >> 172.21.15.28:0/3370183634 conn(0x559451e1df60 msgr2=0x559451e23c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 shutdown_connections 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.207+0000 7f5ab45f1740 1 -- 172.21.15.28:0/3370183634 wait complete. 2024-09-15T21:57:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.208+0000 7f5ab45f1740 20 client.15666 trim_cache size 0 max 16384 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 Processor -- start 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 -- start start 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e2e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e43080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e2fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559451e413c0 con 0x559451e221f0 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559451e41a90 con 0x559451e30fc0 2024-09-15T21:57:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559451e42160 con 0x559451e218d0 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5aaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e43080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5aaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e43080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47776/0 (socket says 172.21.15.28:47776) 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5aaaffd640 1 -- 172.21.15.28:0/2719523168 learned_addr learned my addr 172.21.15.28:0/2719523168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5aabfff640 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e2fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.226+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e2e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5aaaffd640 1 -- 172.21.15.28:0/2719523168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 msgr2=0x559451e2e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e2e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5aaaffd640 1 -- 172.21.15.28:0/2719523168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 msgr2=0x559451e2fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e2fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5aaaffd640 1 -- 172.21.15.28:0/2719523168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559451e22020 con 0x559451e221f0 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e2e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e43080 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x559451e245c0 tx=0x7f5aa0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8d25c8c5242da9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5a917fa640 1 -- 172.21.15.28:0/2719523168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0035070 con 0x559451e221f0 2024-09-15T21:57:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5a917fa640 1 -- 172.21.15.28:0/2719523168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa0031a60 con 0x559451e221f0 2024-09-15T21:57:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.227+0000 7f5a917fa640 1 -- 172.21.15.28:0/2719523168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0003c70 con 0x559451e221f0 2024-09-15T21:57:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 -- 172.21.15.28:0/2719523168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 msgr2=0x559451e43080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e43080 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x559451e245c0 tx=0x7f5aa0005b50 comp rx=0 tx=0).stop 2024-09-15T21:57:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 -- 172.21.15.28:0/2719523168 shutdown_connections 2024-09-15T21:57:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451e2fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e43080 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/2719523168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e2e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 -- 172.21.15.28:0/2719523168 >> 172.21.15.28:0/2719523168 conn(0x559451e1df60 msgr2=0x559451e2f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 -- 172.21.15.28:0/2719523168 shutdown_connections 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 -- 172.21.15.28:0/2719523168 wait complete. 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.228+0000 7f5ab45f1740 1 -- start start 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 1 Processor -- start 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e44730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e44c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451eaa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559451e42f00 con 0x559451e221f0 2024-09-15T21:57:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559451e42830 con 0x559451e30fc0 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559451e435d0 con 0x559451e218d0 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5ab45f1740 10 client.0 _ll_register_callbacks cb 0x7f5a9c009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e44c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aab7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e44730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aab7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e44730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:41962/0 (socket says 172.21.15.28:41962) 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aab7fe640 1 -- 172.21.15.28:0/658593537 learned_addr learned my addr 172.21.15.28:0/658593537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aabfff640 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451eaa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aaaffd640 1 -- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 msgr2=0x559451e44730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e44730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aaaffd640 1 -- 172.21.15.28:0/658593537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 msgr2=0x559451eaa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451eaa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aaaffd640 1 -- 172.21.15.28:0/658593537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559451eaa850 con 0x559451e221f0 2024-09-15T21:57:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.229+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e44730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5aaaffd640 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e44c70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x559451e244c0 tx=0x7f5aa0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b2475fac0f386e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5a8affd640 1 -- 172.21.15.28:0/658593537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0035070 con 0x559451e221f0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559451d8fb10 con 0x559451e221f0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5a8affd640 1 -- 172.21.15.28:0/658593537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f5aa003d480 con 0x559451e221f0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5a8affd640 1 -- 172.21.15.28:0/658593537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5aa0068050 con 0x559451e221f0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5ab45f1740 10 client.15672 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559451eaba90 con 0x559451e221f0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.230+0000 7f5a8affd640 1 -- 172.21.15.28:0/658593537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559451d8fb10 con 0x559451e221f0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5a8affd640 1 -- 172.21.15.28:0/658593537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5aa003a040 con 0x559451e221f0 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5ab45f1740 10 client.15672 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5ab45f1740 10 client.15672 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T21:57:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5ab45f1740 20 client.15672 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451eaec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5ab45f1740 4 client.15672 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "status"}) -- 0x559451d0c0e0 con 0x559451e41c10 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451eaec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5aab7fe640 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451eaec30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=898f480f154cd9fb server_cookie=7d0d917e9761d43b in_seq=0 out_seq=0 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.231+0000 7f5a8affd640 10 client.15672 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5a8affd640 1 -- 172.21.15.28:0/658593537 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559451d0c0e0 con 0x559451e41c10 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5a8affd640 10 client.15672 handle_command_reply: tid=0 2024-09-15T21:57:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 client.15672 shutdown 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 msgr2=0x559451e44c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e44c70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x559451e244c0 tx=0x7f5aa0005a20 comp rx=0 tx=0).stop 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 shutdown_connections 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559451e41c10 0x559451eaec30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559451e30fc0 0x559451eaa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559451e221f0 0x559451e44c70 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 --2- 172.21.15.28:0/658593537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559451e218d0 0x559451e44730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 >> 172.21.15.28:0/658593537 conn(0x559451e1df60 msgr2=0x559451e1d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5a8affd640 0 client.15672 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:57:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 shutdown_connections 2024-09-15T21:57:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 1 -- 172.21.15.28:0/658593537 wait complete. 2024-09-15T21:57:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.232+0000 7f5ab45f1740 20 client.15672 trim_cache size 0 max 16384 2024-09-15T21:57:50.240 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24481, 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 135.70361906900001, 2024-09-15T21:57:50.241 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 29, 2024-09-15T21:57:50.242 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-09-15T21:57:50.242 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T21:57:50.242 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 190.936574705 2024-09-15T21:57:50.242 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a940a34d0 msgr2=0x7f5a940a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a940a34d0 0x7f5a940a5990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009bd0 tx=0x7f5a9c005f90 comp rx=0 tx=0).stop 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1082a0 msgr2=0x7f5aac19cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1082a0 0x7f5aac19cf10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400eaa0 tx=0x7f5aa400ef60 comp rx=0 tx=0).stop 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 shutdown_connections 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5a940a34d0 0x7f5a940a5990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aac1082a0 0x7f5aac19cf10 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5aac1036e0 0x7f5aac198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 --2- 172.21.15.28:0/2225176512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5aac102d30 0x7f5aac198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 >> 172.21.15.28:0/2225176512 conn(0x7f5aac0fc7d0 msgr2=0x7f5aac106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 shutdown_connections 2024-09-15T21:57:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:50.233+0000 7f5ab2218640 1 -- 172.21.15.28:0/2225176512 wait complete. 2024-09-15T21:57:51.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T21:57:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:57:51.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:51 smithi028 ceph-mon[30969]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T21:57:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:51 smithi203 ceph-mon[35546]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T21:57:51.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:51 smithi174 ceph-mon[35244]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T21:57:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 -- 172.21.15.28:0/3307922936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8100e20 msgr2=0x7f6ec8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/3307922936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8100e20 0x7f6ec8101200 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009900 tx=0x7f6eb802a480 comp rx=0 tx=0).stop 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 -- 172.21.15.28:0/3307922936 shutdown_connections 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/3307922936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ec8102ef0 0x7f6ec810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/3307922936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec8102570 0x7f6ec81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/3307922936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8100e20 0x7f6ec8101200 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 -- 172.21.15.28:0/3307922936 >> 172.21.15.28:0/3307922936 conn(0x7f6ec80fc7d0 msgr2=0x7f6ec80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 -- 172.21.15.28:0/3307922936 shutdown_connections 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.813+0000 7f6ece8a1640 1 -- 172.21.15.28:0/3307922936 wait complete. 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 Processor -- start 2024-09-15T21:57:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 -- start start 2024-09-15T21:57:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec8100e20 0x7f6ec8199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ec8102570 0x7f6ec81998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8102ef0 0x7f6ec81a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec8110fa0 con 0x7f6ec8102ef0 2024-09-15T21:57:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec8110e20 con 0x7f6ec8102570 2024-09-15T21:57:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ece8a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec8111120 con 0x7f6ec8100e20 2024-09-15T21:57:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ecce3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8102ef0 0x7f6ec81a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ecce3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8102ef0 0x7f6ec81a43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49440/0 (socket says 172.21.15.28:49440) 2024-09-15T21:57:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.814+0000 7f6ecce3e640 1 -- 172.21.15.28:0/1113557128 learned_addr learned my addr 172.21.15.28:0/1113557128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ebffff640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ec8102570 0x7f6ec81998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec8100e20 0x7f6ec8199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ecce3e640 1 -- 172.21.15.28:0/1113557128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec8100e20 msgr2=0x7f6ec8199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ecce3e640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec8100e20 0x7f6ec8199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ecce3e640 1 -- 172.21.15.28:0/1113557128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ec8102570 msgr2=0x7f6ec81998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ecce3e640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ec8102570 0x7f6ec81998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ecce3e640 1 -- 172.21.15.28:0/1113557128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec81a4930 con 0x7f6ec8102ef0 2024-09-15T21:57:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ebffff640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ec8102570 0x7f6ec81998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ecce3e640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8102ef0 0x7f6ec81a43f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6ec000cce0 tx=0x7f6ec000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ec5ffb640 1 -- 172.21.15.28:0/1113557128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec000f450 con 0x7f6ec8102ef0 2024-09-15T21:57:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ec5ffb640 1 -- 172.21.15.28:0/1113557128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ec000fad0 con 0x7f6ec8102ef0 2024-09-15T21:57:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec81a4b00 con 0x7f6ec8102ef0 2024-09-15T21:57:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ec5ffb640 1 -- 172.21.15.28:0/1113557128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0018c10 con 0x7f6ec8102ef0 2024-09-15T21:57:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.815+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec81a4fc0 con 0x7f6ec8102ef0 2024-09-15T21:57:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.817+0000 7f6ec5ffb640 1 -- 172.21.15.28:0/1113557128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7f6ec000f5f0 con 0x7f6ec8102ef0 2024-09-15T21:57:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 Processor -- start 2024-09-15T21:57:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 -- start start 2024-09-15T21:57:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x5635552fedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552ff2f0 0x563555309670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355530bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563555265090 con 0x5635552ff2f0 2024-09-15T21:57:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56355520ca70 con 0x563555309bb0 2024-09-15T21:57:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563555261050 con 0x5635552fe9d0 2024-09-15T21:57:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552ff2f0 0x563555309670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552ff2f0 0x563555309670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49444/0 (socket says 172.21.15.28:49444) 2024-09-15T21:57:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 -- 172.21.15.28:0/867640699 learned_addr learned my addr 172.21.15.28:0/867640699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ebffff640 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355530bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ebffff640 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x5635552fedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec5ffb640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6eb40a34d0 0x7f6eb40a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 -- 172.21.15.28:0/867640699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 msgr2=0x5635552fedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x5635552fedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 -- 172.21.15.28:0/867640699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 msgr2=0x56355530bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355530bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ec7fff640 1 -- 172.21.15.28:0/867640699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563555219130 con 0x5635552ff2f0 2024-09-15T21:57:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.818+0000 7f6ebffff640 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x5635552fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.819+0000 7f6ecce3e640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6eb40a34d0 0x7f6eb40a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.819+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552ff2f0 0x563555309670 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80314e0 tx=0x7f6eb8004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ac9904b6333956 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.819+0000 7f6ec5ffb640 1 -- 172.21.15.28:0/1113557128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f6ec00d0b20 con 0x7f6ec8102ef0 2024-09-15T21:57:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.819+0000 7f6ebe7fc640 1 -- 172.21.15.28:0/867640699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8035030 con 0x5635552ff2f0 2024-09-15T21:57:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.819+0000 7f6ebe7fc640 1 -- 172.21.15.28:0/867640699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb8003ad0 con 0x5635552ff2f0 2024-09-15T21:57:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.819+0000 7f6ebe7fc640 1 -- 172.21.15.28:0/867640699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8031c60 con 0x5635552ff2f0 2024-09-15T21:57:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/867640699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552ff2f0 msgr2=0x563555309670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ecce3e640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6eb40a34d0 0x7f6eb40a5990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00cdd90 tx=0x7f6ec00186b0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552ff2f0 0x563555309670 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80314e0 tx=0x7f6eb8004170 comp rx=0 tx=0).stop 2024-09-15T21:57:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/867640699 shutdown_connections 2024-09-15T21:57:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355530bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552ff2f0 0x563555309670 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/867640699 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x5635552fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/867640699 >> 172.21.15.28:0/867640699 conn(0x5635552faf60 msgr2=0x5635552fb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/867640699 shutdown_connections 2024-09-15T21:57:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/867640699 wait complete. 2024-09-15T21:57:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 -- start start 2024-09-15T21:57:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.820+0000 7f6ed0c7a740 1 Processor -- start 2024-09-15T21:57:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x563555309420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x563555306770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56355530c780 con 0x563555309bb0 2024-09-15T21:57:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56355521ff20 con 0x5635552ff2f0 2024-09-15T21:57:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56355530c900 con 0x5635552fe9d0 2024-09-15T21:57:51.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ed0c7a740 10 client.0 _ll_register_callbacks cb 0x5635551e9030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49460/0 (socket says 172.21.15.28:49460) 2024-09-15T21:57:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 -- 172.21.15.28:0/2446826193 learned_addr learned my addr 172.21.15.28:0/2446826193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x563555306770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x563555309420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 -- 172.21.15.28:0/2446826193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 msgr2=0x563555309420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x563555309420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 -- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 msgr2=0x563555306770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x563555306770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 -- 172.21.15.28:0/2446826193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563554fd62b0 con 0x563555309bb0 2024-09-15T21:57:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x563555306770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0017a70 tx=0x7f6eb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f1b2ee886f0fb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.821+0000 7f6ea77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2446826193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0012b20 con 0x563555309bb0 2024-09-15T21:57:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb8003ad0 con 0x563555309bb0 2024-09-15T21:57:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2446826193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f6eb0012cc0 con 0x563555309bb0 2024-09-15T21:57:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2446826193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6eb0026050 con 0x563555309bb0 2024-09-15T21:57:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 10 client.15684 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563555318070 con 0x563555309bb0 2024-09-15T21:57:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2446826193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb8003ad0 con 0x563555309bb0 2024-09-15T21:57:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2446826193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6eb0015450 con 0x563555309bb0 2024-09-15T21:57:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 10 client.15684 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 10 client.15684 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:57:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 20 client.15684 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355531a690 0x56355531bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 4 client.15684 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.822+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563555272600 con 0x56355531a690 2024-09-15T21:57:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.823+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355531a690 0x56355531bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.823+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355531a690 0x56355531bb00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=726e81462019eddb server_cookie=c1de92d36f7bc157 in_seq=0 out_seq=0 2024-09-15T21:57:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.823+0000 7f6ea77fe640 10 client.15684 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2446826193 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563555272600 con 0x56355531a690 2024-09-15T21:57:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ea77fe640 10 client.15684 handle_command_reply: tid=0 2024-09-15T21:57:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 client.15684 shutdown 2024-09-15T21:57:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 msgr2=0x563555306cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0017a70 tx=0x7f6eb00046c0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 shutdown_connections 2024-09-15T21:57:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355531a690 0x56355531bb00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306cb0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x563555306770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2446826193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x563555309420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ea77fe640 0 client.15684 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 >> 172.21.15.28:0/2446826193 conn(0x5635552faf60 msgr2=0x563555300c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 shutdown_connections 2024-09-15T21:57:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2446826193 wait complete. 2024-09-15T21:57:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.824+0000 7f6ed0c7a740 20 client.15684 trim_cache size 0 max 16384 2024-09-15T21:57:51.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.842+0000 7f6ed0c7a740 1 Processor -- start 2024-09-15T21:57:51.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.842+0000 7f6ed0c7a740 1 -- start start 2024-09-15T21:57:51.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.842+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x56355531bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x5635553241d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563555319e40 con 0x563555309bb0 2024-09-15T21:57:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56355531a510 con 0x5635552ff2f0 2024-09-15T21:57:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56355531abe0 con 0x5635552fe9d0 2024-09-15T21:57:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49462/0 (socket says 172.21.15.28:49462) 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 -- 172.21.15.28:0/2970785118 learned_addr learned my addr 172.21.15.28:0/2970785118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x56355531bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x5635553241d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 -- 172.21.15.28:0/2970785118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 msgr2=0x56355531bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x56355531bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 -- 172.21.15.28:0/2970785118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 msgr2=0x5635553241d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x5635553241d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 -- 172.21.15.28:0/2970785118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56355531b090 con 0x563555309bb0 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.843+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306730 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00175c0 tx=0x7f6eb000a1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449862a34e873913 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ebdffb640 1 -- 172.21.15.28:0/2970785118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb000a270 con 0x563555309bb0 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ebdffb640 1 -- 172.21.15.28:0/2970785118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb001d070 con 0x563555309bb0 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ebdffb640 1 -- 172.21.15.28:0/2970785118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0018e40 con 0x563555309bb0 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2970785118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 msgr2=0x563555306730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306730 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00175c0 tx=0x7f6eb000a1b0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2970785118 shutdown_connections 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563555309bb0 0x563555306730 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635552ff2f0 0x5635553241d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2970785118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552fe9d0 0x56355531bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2970785118 >> 172.21.15.28:0/2970785118 conn(0x5635552faf60 msgr2=0x5635552fb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2970785118 shutdown_connections 2024-09-15T21:57:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.844+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2970785118 wait complete. 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 -- start start 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 Processor -- start 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552fe9d0 0x5635553236d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552ff2f0 0x563555323c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355537fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56355531b980 con 0x5635552fe9d0 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56355531b2b0 con 0x563555309bb0 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56355531c050 con 0x5635552ff2f0 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ed0c7a740 10 client.0 _ll_register_callbacks cb 0x7f6eb000a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552fe9d0 0x5635553236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552fe9d0 0x5635553236d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49470/0 (socket says 172.21.15.28:49470) 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.845+0000 7f6ebffff640 1 -- 172.21.15.28:0/2129839576 learned_addr learned my addr 172.21.15.28:0/2129839576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552ff2f0 0x563555323c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355537fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ebffff640 1 -- 172.21.15.28:0/2129839576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552ff2f0 msgr2=0x563555323c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552ff2f0 0x563555323c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ebffff640 1 -- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 msgr2=0x56355537fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355537fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ebffff640 1 -- 172.21.15.28:0/2129839576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635553804b0 con 0x5635552fe9d0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ebffff640 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552fe9d0 0x5635553236d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0004700 tx=0x7f6eb0007fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f094db53b96d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ea77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2129839576 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb003e020 con 0x5635552fe9d0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2129839576 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f6eb000a640 con 0x5635552fe9d0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb001d070 con 0x5635552fe9d0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2129839576 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6eb0049050 con 0x5635552fe9d0 2024-09-15T21:57:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ed0c7a740 10 client.15690 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.846+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563555388a40 con 0x5635552fe9d0 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2129839576 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb001d070 con 0x5635552fe9d0 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2129839576 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6eb0018450 con 0x5635552fe9d0 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ed0c7a740 10 client.15690 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ed0c7a740 10 client.15690 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ed0c7a740 20 client.15690 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355538d280 0x56355538bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ed0c7a740 4 client.15690 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "status"}) -- 0x56355526cb10 con 0x56355538d280 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.847+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355538d280 0x56355538bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ea77fe640 10 client.15690 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355538d280 0x56355538bf90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76fd4ff00522adac server_cookie=2b38b93b852fad55 in_seq=0 out_seq=0 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ea77fe640 1 -- 172.21.15.28:0/2129839576 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56355526cb10 con 0x56355538d280 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ea77fe640 10 client.15690 handle_command_reply: tid=0 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 client.15690 shutdown 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552fe9d0 msgr2=0x5635553236d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552fe9d0 0x5635553236d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0004700 tx=0x7f6eb0007fd0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 shutdown_connections 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56355538d280 0x56355538bf90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ea77fe640 0 client.15690 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563555309bb0 0x56355537fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635552ff2f0 0x563555323c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 --2- 172.21.15.28:0/2129839576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635552fe9d0 0x5635553236d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 >> 172.21.15.28:0/2129839576 conn(0x5635552faf60 msgr2=0x563555306730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 shutdown_connections 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.848+0000 7f6ed0c7a740 1 -- 172.21.15.28:0/2129839576 wait complete. 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ed0c7a740 20 client.15690 trim_cache size 0 max 16384 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:57:51.867 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "e78b7764-73ab-11ef-bceb-c7b262605968", 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24485, 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 172.42383357400001, 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 29, 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 183.57257599900001 2024-09-15T21:57:51.868 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6eb40a34d0 msgr2=0x7f6eb40a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6eb40a34d0 0x7f6eb40a5990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00cdd90 tx=0x7f6ec00186b0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8102ef0 msgr2=0x7f6ec81a43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8102ef0 0x7f6ec81a43f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6ec000cce0 tx=0x7f6ec000a620 comp rx=0 tx=0).stop 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 shutdown_connections 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6eb40a34d0 0x7f6eb40a5990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8102ef0 0x7f6ec81a43f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ec8102570 0x7f6ec81998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 --2- 172.21.15.28:0/1113557128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec8100e20 0x7f6ec8199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 >> 172.21.15.28:0/1113557128 conn(0x7f6ec80fc7d0 msgr2=0x7f6ec80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 shutdown_connections 2024-09-15T21:57:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:57:51.849+0000 7f6ece8a1640 1 -- 172.21.15.28:0/1113557128 wait complete. 2024-09-15T21:57:52.499 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-15T21:57:53.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:53 smithi028 ceph-mon[30969]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 17 op/s 2024-09-15T21:57:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:53 smithi203 ceph-mon[35546]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 17 op/s 2024-09-15T21:57:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:53 smithi174 ceph-mon[35244]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 17 op/s 2024-09-15T21:57:55.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:55 smithi028 ceph-mon[30969]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T21:57:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:55 smithi203 ceph-mon[35546]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T21:57:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:55 smithi174 ceph-mon[35244]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T21:57:57.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:57 smithi028 ceph-mon[30969]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 17 op/s 2024-09-15T21:57:57.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:57 smithi203 ceph-mon[35546]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 17 op/s 2024-09-15T21:57:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:57 smithi174 ceph-mon[35244]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 17 op/s 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 -- 172.21.15.28:0/759343315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb944100830 msgr2=0x7fb944100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 --2- 172.21.15.28:0/759343315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb944100830 0x7fb944100c10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb934009930 tx=0x7fb93402a4d0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 -- 172.21.15.28:0/759343315 shutdown_connections 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 --2- 172.21.15.28:0/759343315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9441080f0 0x7fb94410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 --2- 172.21.15.28:0/759343315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944101150 0x7fb944107990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 --2- 172.21.15.28:0/759343315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb944100830 0x7fb944100c10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 -- 172.21.15.28:0/759343315 >> 172.21.15.28:0/759343315 conn(0x7fb944075ec0 msgr2=0x7fb9440762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 -- 172.21.15.28:0/759343315 shutdown_connections 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 -- 172.21.15.28:0/759343315 wait complete. 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 Processor -- start 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.941+0000 7fb94b6af640 1 -- start start 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb94b6af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb944100830 0x7fb94419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb94b6af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944101150 0x7fb94419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb94b6af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9441080f0 0x7fb94419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb94b6af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb944111f20 con 0x7fb9441080f0 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb94b6af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb944111da0 con 0x7fb944101150 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb94b6af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441120a0 con 0x7fb944100830 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9441080f0 0x7fb94419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9441080f0 0x7fb94419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49498/0 (socket says 172.21.15.28:49498) 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 -- 172.21.15.28:0/3454753790 learned_addr learned my addr 172.21.15.28:0/3454753790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb94944b640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb944100830 0x7fb94419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb948c4a640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944101150 0x7fb94419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 -- 172.21.15.28:0/3454753790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb944100830 msgr2=0x7fb94419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb944100830 0x7fb94419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 -- 172.21.15.28:0/3454753790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944101150 msgr2=0x7fb94419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944101150 0x7fb94419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 -- 172.21.15.28:0/3454753790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94419f540 con 0x7fb9441080f0 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb948c4a640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944101150 0x7fb94419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.942+0000 7fb949c4c640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9441080f0 0x7fb94419ee10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00eaa0 tx=0x7fb93c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.943+0000 7fb93a7fc640 1 -- 172.21.15.28:0/3454753790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c00cdb0 con 0x7fb9441080f0 2024-09-15T21:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.943+0000 7fb93a7fc640 1 -- 172.21.15.28:0/3454753790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93c005020 con 0x7fb9441080f0 2024-09-15T21:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.943+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94419f830 con 0x7fb9441080f0 2024-09-15T21:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.943+0000 7fb93a7fc640 1 -- 172.21.15.28:0/3454753790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0116b0 con 0x7fb9441080f0 2024-09-15T21:57:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.943+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9441a70c0 con 0x7fb9441080f0 2024-09-15T21:57:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.945+0000 7fb93a7fc640 1 -- 172.21.15.28:0/3454753790 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138582+0+0 (secure 0 0 0) 0x7fb93c005560 con 0x7fb9441080f0 2024-09-15T21:57:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 Processor -- start 2024-09-15T21:57:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 -- start start 2024-09-15T21:57:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abdbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdc410 0x558d1abe6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d1ab402f0 con 0x558d1abe6cb0 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558d1aae8270 con 0x558d1abdbaf0 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94e440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d1ab3c2b0 con 0x558d1abdc410 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb948c4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb93a7fc640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb9300a3600 0x7fb9300a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb948c4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49500/0 (socket says 172.21.15.28:49500) 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb948c4a640 1 -- 172.21.15.28:0/513717480 learned_addr learned my addr 172.21.15.28:0/513717480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb94944b640 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdc410 0x558d1abe6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb949c4c640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb9300a3600 0x7fb9300a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb948c4a640 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abdbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.946+0000 7fb93a7fc640 1 -- 172.21.15.28:0/3454753790 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb93c0d04c0 con 0x7fb9441080f0 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb948c4a640 1 -- 172.21.15.28:0/513717480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdc410 msgr2=0x558d1abe6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb948c4a640 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdc410 0x558d1abe6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb948c4a640 1 -- 172.21.15.28:0/513717480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 msgr2=0x558d1abdbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb948c4a640 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abdbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb948c4a640 1 -- 172.21.15.28:0/513717480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d1aae7340 con 0x558d1abe6cb0 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb94944b640 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdc410 0x558d1abe6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb948c4a640 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe90a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb94000cfd0 tx=0x7fb94000c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c686280b0948904e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb92f7fe640 1 -- 172.21.15.28:0/513717480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940019030 con 0x558d1abe6cb0 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb949c4c640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb9300a3600 0x7fb9300a5ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0cd790 tx=0x7fb93c06df70 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb92f7fe640 1 -- 172.21.15.28:0/513717480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb940011a80 con 0x558d1abe6cb0 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.947+0000 7fb92f7fe640 1 -- 172.21.15.28:0/513717480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94000fb40 con 0x558d1abe6cb0 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 -- 172.21.15.28:0/513717480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 msgr2=0x558d1abe90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe90a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb94000cfd0 tx=0x7fb94000c720 comp rx=0 tx=0).stop 2024-09-15T21:57:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 -- 172.21.15.28:0/513717480 shutdown_connections 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe90a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdc410 0x558d1abe6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 --2- 172.21.15.28:0/513717480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abdbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 -- 172.21.15.28:0/513717480 >> 172.21.15.28:0/513717480 conn(0x558d1abd8060 msgr2=0x558d1abd8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 -- 172.21.15.28:0/513717480 shutdown_connections 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 -- 172.21.15.28:0/513717480 wait complete. 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 -- start start 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.948+0000 7fb94e440740 1 Processor -- start 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94e440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdbaf0 0x558d1abe3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94e440740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdc410 0x558d1abe40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94e440740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94e440740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d1abe9880 con 0x558d1abe6cb0 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94e440740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558d1aac9290 con 0x558d1abdc410 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94e440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d1abe9a00 con 0x558d1abdbaf0 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94e440740 10 client.0 _ll_register_callbacks cb 0x558d1aafa140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49514/0 (socket says 172.21.15.28:49514) 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 -- 172.21.15.28:0/2608320599 learned_addr learned my addr 172.21.15.28:0/2608320599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdbaf0 0x558d1abe3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb94944b640 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdc410 0x558d1abe40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 -- 172.21.15.28:0/2608320599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdbaf0 msgr2=0x558d1abe3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdbaf0 0x558d1abe3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 -- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdc410 msgr2=0x558d1abe40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdc410 0x558d1abe40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.949+0000 7fb948c4a640 1 -- 172.21.15.28:0/2608320599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d1a853210 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb948c4a640 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe4620 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb94000c2f0 tx=0x7fb940015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c482bb12b387e4e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb92cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb92cff9640 1 -- 172.21.15.28:0/2608320599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940019030 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb92cff9640 1 -- 172.21.15.28:0/2608320599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb940027040 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb940011a80 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb92cff9640 1 -- 172.21.15.28:0/2608320599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb9400117b0 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb94e440740 10 client.15702 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558d1a91f620 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb92cff9640 1 -- 172.21.15.28:0/2608320599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb940011a80 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb92cff9640 1 -- 172.21.15.28:0/2608320599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb9400035f0 con 0x558d1abe6cb0 2024-09-15T21:57:57.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb94e440740 10 client.15702 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb94e440740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.950+0000 7fb94e440740 20 client.15702 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.951+0000 7fb94e440740 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1abf6b90 0x558d1abf8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.951+0000 7fb94e440740 4 client.15702 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.951+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d1ab4da20 con 0x558d1abf6b90 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.951+0000 7fb94944b640 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1abf6b90 0x558d1abf8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.951+0000 7fb92cff9640 10 client.15702 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.951+0000 7fb94944b640 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1abf6b90 0x558d1abf8f80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=679ec91d34d1abea server_cookie=fa71baa4976635ba in_seq=0 out_seq=0 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb92cff9640 1 -- 172.21.15.28:0/2608320599 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558d1ab4da20 con 0x558d1abf6b90 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb92cff9640 10 client.15702 handle_command_reply: tid=0 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 client.15702 shutdown 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 msgr2=0x558d1abe4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe4620 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb94000c2f0 tx=0x7fb940015040 comp rx=0 tx=0).stop 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 shutdown_connections 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1abf6b90 0x558d1abf8f80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abe6cb0 0x558d1abe4620 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdc410 0x558d1abe40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb92cff9640 0 client.15702 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 --2- 172.21.15.28:0/2608320599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abdbaf0 0x558d1abe3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 >> 172.21.15.28:0/2608320599 conn(0x558d1abd8060 msgr2=0x558d1abddd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 shutdown_connections 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.952+0000 7fb94e440740 1 -- 172.21.15.28:0/2608320599 wait complete. 2024-09-15T21:57:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.953+0000 7fb94e440740 20 client.15702 trim_cache size 0 max 16384 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.967+0000 7fb94e440740 1 Processor -- start 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.967+0000 7fb94e440740 1 -- start start 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94e440740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abde560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94e440740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abe4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94e440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 0x558d1abfe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94e440740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441b10d0 con 0x558d1abdc410 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94e440740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441b2c10 con 0x558d1abdbaf0 2024-09-15T21:57:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94e440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441b17a0 con 0x558d1abe6cb0 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abe4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abe4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49530/0 (socket says 172.21.15.28:49530) 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 -- 172.21.15.28:0/2611732504 learned_addr learned my addr 172.21.15.28:0/2611732504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb948c4a640 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abde560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 -- 172.21.15.28:0/2611732504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 msgr2=0x558d1abfe6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 0x558d1abfe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 -- 172.21.15.28:0/2611732504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 msgr2=0x558d1abde560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abde560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 -- 172.21.15.28:0/2611732504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d1abd7e90 con 0x558d1abdc410 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb948c4a640 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abde560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:57:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb94944b640 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abe4450 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb9340096c0 tx=0x7fb934031980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9234205eefde903 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb92effd640 1 -- 172.21.15.28:0/2611732504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934035040 con 0x558d1abdc410 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb92effd640 1 -- 172.21.15.28:0/2611732504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb934005500 con 0x558d1abdc410 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.968+0000 7fb92effd640 1 -- 172.21.15.28:0/2611732504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934004ca0 con 0x558d1abdc410 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 -- 172.21.15.28:0/2611732504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 msgr2=0x558d1abe4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abe4450 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb9340096c0 tx=0x7fb934031980 comp rx=0 tx=0).stop 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 -- 172.21.15.28:0/2611732504 shutdown_connections 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 0x558d1abfe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abe4450 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 --2- 172.21.15.28:0/2611732504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abde560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 -- 172.21.15.28:0/2611732504 >> 172.21.15.28:0/2611732504 conn(0x558d1abd8060 msgr2=0x558d1abe5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 -- 172.21.15.28:0/2611732504 shutdown_connections 2024-09-15T21:57:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 -- 172.21.15.28:0/2611732504 wait complete. 2024-09-15T21:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 -- start start 2024-09-15T21:57:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.969+0000 7fb94e440740 1 Processor -- start 2024-09-15T21:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94e440740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abeee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94e440740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94e440740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 0x558d1abea230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94e440740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441b2540 con 0x558d1abdc410 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94e440740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441b1e70 con 0x558d1abdbaf0 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94e440740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb9441af0b0 con 0x558d1abe6cb0 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94e440740 10 client.0 _ll_register_callbacks cb 0x7fb9400134f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abef340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49534/0 (socket says 172.21.15.28:49534) 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 -- 172.21.15.28:0/1944213457 learned_addr learned my addr 172.21.15.28:0/1944213457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb948c4a640 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 0x558d1abea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb948c4a640 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abeee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 -- 172.21.15.28:0/1944213457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 msgr2=0x558d1abea230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 0x558d1abea230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 -- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 msgr2=0x558d1abeee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abeee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.970+0000 7fb94944b640 1 -- 172.21.15.28:0/1944213457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d1abeaab0 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94944b640 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abef340 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb934000c00 tx=0x7fb934009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1a969110bc686d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb92cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb92cff9640 1 -- 172.21.15.28:0/1944213457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934035040 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb92cff9640 1 -- 172.21.15.28:0/1944213457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb934004040 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb934005500 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb92cff9640 1 -- 172.21.15.28:0/1944213457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb934062050 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 10 client.15708 fetch_fsmap learned FSMap version 29 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558d1ac61130 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb92cff9640 1 -- 172.21.15.28:0/1944213457 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb934005500 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb92cff9640 1 -- 172.21.15.28:0/1944213457 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb934003990 con 0x558d1abdc410 2024-09-15T21:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 10 client.15708 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 20 client.15708 populate_metadata read hostname 'smithi028' 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1ac61790 0x558d1ac63bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 4 client.15708 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.971+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d1ab47d70 con 0x558d1ac61790 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.972+0000 7fb948c4a640 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1ac61790 0x558d1ac63bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.972+0000 7fb948c4a640 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1ac61790 0x558d1ac63bb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4acce97ffab95253 server_cookie=90eff54dafe5f11d in_seq=0 out_seq=0 2024-09-15T21:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.972+0000 7fb92cff9640 10 client.15708 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb92cff9640 1 -- 172.21.15.28:0/1944213457 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d1ab47d70 con 0x558d1ac61790 2024-09-15T21:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb92cff9640 10 client.15708 handle_command_reply: tid=0 2024-09-15T21:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 client.15708 shutdown 2024-09-15T21:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 msgr2=0x558d1abef340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abef340 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb934000c00 tx=0x7fb934009230 comp rx=0 tx=0).stop 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 shutdown_connections 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558d1ac61790 0x558d1ac63bb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d1abe6cb0 0x558d1abea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d1abdc410 0x558d1abef340 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb92cff9640 0 client.15708 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 --2- 172.21.15.28:0/1944213457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558d1abdbaf0 0x558d1abeee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 >> 172.21.15.28:0/1944213457 conn(0x558d1abd8060 msgr2=0x558d1abd69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 shutdown_connections 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 1 -- 172.21.15.28:0/1944213457 wait complete. 2024-09-15T21:57:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.976+0000 7fb94e440740 20 client.15708 trim_cache size 0 max 16384 2024-09-15T21:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T21:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T21:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T21:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb9300a3600 msgr2=0x7fb9300a5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb9300a3600 0x7fb9300a5ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0cd790 tx=0x7fb93c06df70 comp rx=0 tx=0).stop 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9441080f0 msgr2=0x7fb94419ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9441080f0 0x7fb94419ee10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00eaa0 tx=0x7fb93c00ef60 comp rx=0 tx=0).stop 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 shutdown_connections 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb9300a3600 0x7fb9300a5ac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9441080f0 0x7fb94419ee10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944101150 0x7fb94419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 --2- 172.21.15.28:0/3454753790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb944100830 0x7fb94419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 >> 172.21.15.28:0/3454753790 conn(0x7fb944075ec0 msgr2=0x7fb9441020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 shutdown_connections 2024-09-15T21:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:57:57.977+0000 7fb94b6af640 1 -- 172.21.15.28:0/3454753790 wait complete. 2024-09-15T21:57:59.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:57:59 smithi028 ceph-mon[30969]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-15T21:57:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:57:59 smithi203 ceph-mon[35546]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-15T21:57:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:57:59 smithi174 ceph-mon[35244]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-15T21:58:01.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:58:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:58:00.827+0000 7fb8e7494640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:58:01.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:58:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:58:00.827+0000 7fb8e7494640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:58:01.172 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:58:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:58:00.827+0000 7fb8e7494640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:58:01.172 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:58:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:58:00.827+0000 7fb8e7494640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:58:01.172 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 21:58:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T21:58:00.827+0000 7fb8e7494640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T21:58:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:01 smithi028 ceph-mon[30969]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:01 smithi203 ceph-mon[35546]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:01 smithi174 ceph-mon[35244]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:02.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T21:58:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 -- 172.21.15.28:0/3710309924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc102570 msgr2=0x7f50fc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 --2- 172.21.15.28:0/3710309924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc102570 0x7f50fc1029b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009900 tx=0x7f50f002a480 comp rx=0 tx=0).stop 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 -- 172.21.15.28:0/3710309924 shutdown_connections 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 --2- 172.21.15.28:0/3710309924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc102ef0 0x7f50fc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 --2- 172.21.15.28:0/3710309924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc102570 0x7f50fc1029b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 --2- 172.21.15.28:0/3710309924 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc100e20 0x7f50fc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 -- 172.21.15.28:0/3710309924 >> 172.21.15.28:0/3710309924 conn(0x7f50fc0fc7d0 msgr2=0x7f50fc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 -- 172.21.15.28:0/3710309924 shutdown_connections 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.892+0000 7f5101878640 1 -- 172.21.15.28:0/3710309924 wait complete. 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 Processor -- start 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 -- start start 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc100e20 0x7f50fc0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc102570 0x7f50fc0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc102ef0 0x7f50fc0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc110f80 con 0x7f50fc100e20 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc110e00 con 0x7f50fc102570 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.893+0000 7f5101878640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc111100 con 0x7f50fc102ef0 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc100e20 0x7f50fc0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc100e20 0x7f50fc0fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58210/0 (socket says 172.21.15.28:58210) 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 -- 172.21.15.28:0/2648640652 learned_addr learned my addr 172.21.15.28:0/2648640652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc102570 0x7f50fc0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50fb7fe640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc102ef0 0x7f50fc0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 -- 172.21.15.28:0/2648640652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc102ef0 msgr2=0x7f50fc0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc102ef0 0x7f50fc0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 -- 172.21.15.28:0/2648640652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc102570 msgr2=0x7f50fc0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc102570 0x7f50fc0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 -- 172.21.15.28:0/2648640652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50fc1a69c0 con 0x7f50fc100e20 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50fb7fe640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc102ef0 0x7f50fc0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f50faffd640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc100e20 0x7f50fc0fcc80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c890 tx=0x7f50ec00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f5100876640 1 -- 172.21.15.28:0/2648640652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec004220 con 0x7f50fc100e20 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50fc1a6cb0 con 0x7f50fc100e20 2024-09-15T21:58:02.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.894+0000 7f5100876640 1 -- 172.21.15.28:0/2648640652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ec0043c0 con 0x7f50fc100e20 2024-09-15T21:58:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.895+0000 7f5100876640 1 -- 172.21.15.28:0/2648640652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec004a00 con 0x7f50fc100e20 2024-09-15T21:58:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.895+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50fc1a71c0 con 0x7f50fc100e20 2024-09-15T21:58:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.897+0000 7f5100876640 1 -- 172.21.15.28:0/2648640652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f50ec0079b0 con 0x7f50fc100e20 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f5100876640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e40a35e0 0x7f50e40a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f5103c51740 1 Processor -- start 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e40a35e0 0x7f50e40a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f5103c51740 1 -- start start 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f5100876640 1 -- 172.21.15.28:0/2648640652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f50ec0d0370 con 0x7f50fc100e20 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f5103c51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a5f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f5103c51740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a72950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.898+0000 7f5103c51740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a75280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f5103c51740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bb59c52e0 con 0x560bb5a5fcc0 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f5103c51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560bb596cce0 con 0x560bb5a5f3a0 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f5103c51740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560bb59c12a0 con 0x560bb5a72e90 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a72950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a72950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58222/0 (socket says 172.21.15.28:58222) 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 -- 172.21.15.28:0/2070330507 learned_addr learned my addr 172.21.15.28:0/2070330507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50faffd640 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a5f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e40a35e0 0x7f50e40a5aa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f50f0000c00 tx=0x7f50f0004120 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fb7fe640 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a75280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 -- 172.21.15.28:0/2070330507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 msgr2=0x560bb5a75280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a75280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 -- 172.21.15.28:0/2070330507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 msgr2=0x560bb5a5f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a5f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 -- 172.21.15.28:0/2070330507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb5951440 con 0x560bb5a5fcc0 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.899+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a72950 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009900 tx=0x7f50f0046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9a581c2bb57146 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.900+0000 7f50e1ffb640 1 -- 172.21.15.28:0/2070330507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0035050 con 0x560bb5a5fcc0 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.900+0000 7f50e1ffb640 1 -- 172.21.15.28:0/2070330507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f0031a20 con 0x560bb5a5fcc0 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.900+0000 7f50e1ffb640 1 -- 172.21.15.28:0/2070330507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f003cb50 con 0x560bb5a5fcc0 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.900+0000 7f5103c51740 1 -- 172.21.15.28:0/2070330507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 msgr2=0x560bb5a72950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.900+0000 7f5103c51740 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a72950 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009900 tx=0x7f50f0046000 comp rx=0 tx=0).stop 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.900+0000 7f5103c51740 1 -- 172.21.15.28:0/2070330507 shutdown_connections 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a75280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a72950 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 --2- 172.21.15.28:0/2070330507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a5f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 -- 172.21.15.28:0/2070330507 >> 172.21.15.28:0/2070330507 conn(0x560bb5a5ba30 msgr2=0x560bb5a5be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 -- 172.21.15.28:0/2070330507 shutdown_connections 2024-09-15T21:58:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 -- 172.21.15.28:0/2070330507 wait complete. 2024-09-15T21:58:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 -- start start 2024-09-15T21:58:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.901+0000 7f5103c51740 1 Processor -- start 2024-09-15T21:58:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f5103c51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a72730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f5103c51740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a6f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f5103c51740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a6ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f5103c51740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bb5a75a60 con 0x560bb5a5fcc0 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f5103c51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560bb5980050 con 0x560bb5a5f3a0 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a6f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f5103c51740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560bb5a75be0 con 0x560bb5a72e90 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a6f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58236/0 (socket says 172.21.15.28:58236) 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3178883443 learned_addr learned my addr 172.21.15.28:0/3178883443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50faffd640 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a72730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fb7fe640 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a6ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f5103c51740 10 client.0 _ll_register_callbacks cb 0x560bb59c0160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3178883443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 msgr2=0x560bb5a6ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a6ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 msgr2=0x560bb5a72730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a72730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3178883443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb57363c0 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a6f9e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f50f0003f50 tx=0x7f50f003c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2206d32b5408b1a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.902+0000 7f50daffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.903+0000 7f50daffd640 1 -- 172.21.15.28:0/3178883443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0055070 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.903+0000 7f50daffd640 1 -- 172.21.15.28:0/3178883443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f50f0031540 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.903+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50f0031a20 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.903+0000 7f50daffd640 1 -- 172.21.15.28:0/3178883443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50f0031740 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.903+0000 7f5103c51740 10 client.15726 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.903+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560bb5a75a60 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.903+0000 7f50daffd640 1 -- 172.21.15.28:0/3178883443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f0031a20 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f50daffd640 1 -- 172.21.15.28:0/3178883443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f50f005a890 con 0x560bb5a5fcc0 2024-09-15T21:58:02.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f5103c51740 10 client.15726 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f5103c51740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f5103c51740 20 client.15726 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f5103c51740 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a82560 0x560bb5a84950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f5103c51740 4 client.15726 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bb59d2850 con 0x560bb5a82560 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f50faffd640 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a82560 0x560bb5a84950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f50faffd640 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a82560 0x560bb5a84950 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e55cd0ce4bb182f3 server_cookie=760c4b806a3255fa in_seq=0 out_seq=0 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.904+0000 7f50daffd640 10 client.15726 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f50daffd640 1 -- 172.21.15.28:0/3178883443 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560bb59d2850 con 0x560bb5a82560 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f50daffd640 10 client.15726 handle_command_reply: tid=0 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 client.15726 shutdown 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 msgr2=0x560bb5a6f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a6f9e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f50f0003f50 tx=0x7f50f003c6f0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 shutdown_connections 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a82560 0x560bb5a84950 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f50daffd640 0 client.15726 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a72e90 0x560bb5a6ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5fcc0 0x560bb5a6f9e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 --2- 172.21.15.28:0/3178883443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a72730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.905+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 >> 172.21.15.28:0/3178883443 conn(0x560bb5a5ba30 msgr2=0x560bb5a61790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.906+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 shutdown_connections 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.906+0000 7f5103c51740 1 -- 172.21.15.28:0/3178883443 wait complete. 2024-09-15T21:58:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.906+0000 7f5103c51740 20 client.15726 trim_cache size 0 max 16384 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 Processor -- start 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 -- start start 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5ba30 0x560bb5a6ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a8a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5fcc0 0x560bb5a8a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1acdb0 con 0x560bb5a5ba30 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1ae8f0 con 0x560bb5a5f3a0 2024-09-15T21:58:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f5103c51740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1ad480 con 0x560bb5a5fcc0 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f50faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5ba30 0x560bb5a6ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a8a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a8a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:46228/0 (socket says 172.21.15.28:46228) 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f50faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5ba30 0x560bb5a6ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58238/0 (socket says 172.21.15.28:58238) 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3939326202 learned_addr learned my addr 172.21.15.28:0/3939326202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.923+0000 7f50fb7fe640 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5fcc0 0x560bb5a8a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3939326202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5fcc0 msgr2=0x560bb5a8a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5fcc0 0x560bb5a8a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3939326202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5ba30 msgr2=0x560bb5a6ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5ba30 0x560bb5a6ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50fa7fc640 1 -- 172.21.15.28:0/3939326202 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bb5a72aa0 con 0x560bb5a5f3a0 2024-09-15T21:58:02.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50faffd640 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5ba30 0x560bb5a6ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:02.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a8a1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50f005ae20 tx=0x7f50f0005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fa3c3091225b65c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:02.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50e17fa640 1 -- 172.21.15.28:0/3939326202 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0055070 con 0x560bb5a5f3a0 2024-09-15T21:58:02.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50e17fa640 1 -- 172.21.15.28:0/3939326202 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f0031540 con 0x560bb5a5f3a0 2024-09-15T21:58:02.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.924+0000 7f50e17fa640 1 -- 172.21.15.28:0/3939326202 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0035040 con 0x560bb5a5f3a0 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 -- 172.21.15.28:0/3939326202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 msgr2=0x560bb5a8a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a8a1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50f005ae20 tx=0x7f50f0005df0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 -- 172.21.15.28:0/3939326202 shutdown_connections 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5fcc0 0x560bb5a8a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5f3a0 0x560bb5a8a1b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 --2- 172.21.15.28:0/3939326202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5ba30 0x560bb5a6ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 -- 172.21.15.28:0/3939326202 >> 172.21.15.28:0/3939326202 conn(0x560bb5a70080 msgr2=0x560bb5a61790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 -- 172.21.15.28:0/3939326202 shutdown_connections 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 -- 172.21.15.28:0/3939326202 wait complete. 2024-09-15T21:58:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.925+0000 7f5103c51740 1 -- start start 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 1 Processor -- start 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5ba30 0x560bb5a7aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5f3a0 0x560bb5a7b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5fcc0 0x560bb5a77ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1ae220 con 0x560bb5a5f3a0 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1adb50 con 0x560bb5a5fcc0 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50fc1aad90 con 0x560bb5a5ba30 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5f3a0 0x560bb5a7b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f5103c51740 10 client.0 _ll_register_callbacks cb 0x7f50ec0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f50fa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5f3a0 0x560bb5a7b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58240/0 (socket says 172.21.15.28:58240) 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f50fa7fc640 1 -- 172.21.15.28:0/1775390402 learned_addr learned my addr 172.21.15.28:0/1775390402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f50faffd640 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5ba30 0x560bb5a7aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.926+0000 7f50fb7fe640 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5fcc0 0x560bb5a77ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50fa7fc640 1 -- 172.21.15.28:0/1775390402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5ba30 msgr2=0x560bb5a7aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5ba30 0x560bb5a7aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50fa7fc640 1 -- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5fcc0 msgr2=0x560bb5a77ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5fcc0 0x560bb5a77ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50fa7fc640 1 -- 172.21.15.28:0/1775390402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bb5a783e0 con 0x560bb5a5f3a0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50fa7fc640 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5f3a0 0x560bb5a7b410 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f50f002a9a0 tx=0x7f50f003c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bef92060b20d1e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50daffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50daffd640 1 -- 172.21.15.28:0/1775390402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0035030 con 0x560bb5a5f3a0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50daffd640 1 -- 172.21.15.28:0/1775390402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f50f003cab0 con 0x560bb5a5f3a0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50f0031540 con 0x560bb5a5f3a0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50daffd640 1 -- 172.21.15.28:0/1775390402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50f0055040 con 0x560bb5a5f3a0 2024-09-15T21:58:02.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f5103c51740 10 client.15732 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560bb5aed830 con 0x560bb5a5f3a0 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.927+0000 7f50daffd640 1 -- 172.21.15.28:0/1775390402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f0031540 con 0x560bb5a5f3a0 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f50daffd640 1 -- 172.21.15.28:0/1775390402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f50f0049680 con 0x560bb5a5f3a0 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f5103c51740 10 client.15732 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f5103c51740 10 client.15732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f5103c51740 20 client.15732 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f5103c51740 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a72e90 0x560bb5aefc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f5103c51740 4 client.15732 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560bb59ccd60 con 0x560bb5a72e90 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f50faffd640 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a72e90 0x560bb5aefc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f50daffd640 10 client.15732 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.928+0000 7f50faffd640 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a72e90 0x560bb5aefc80 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33321b0cf0543092 server_cookie=6ffa3aa56a595f8 in_seq=0 out_seq=0 2024-09-15T21:58:02.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f50daffd640 1 -- 172.21.15.28:0/1775390402 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560bb59ccd60 con 0x560bb5a72e90 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f50daffd640 10 client.15732 handle_command_reply: tid=0 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 client.15732 shutdown 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5f3a0 msgr2=0x560bb5a7b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5f3a0 0x560bb5a7b410 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f50f002a9a0 tx=0x7f50f003c830 comp rx=0 tx=0).stop 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 shutdown_connections 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560bb5a72e90 0x560bb5aefc80 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560bb5a5fcc0 0x560bb5a77ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bb5a5f3a0 0x560bb5a7b410 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f50daffd640 0 client.15732 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 --2- 172.21.15.28:0/1775390402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560bb5a5ba30 0x560bb5a7aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 >> 172.21.15.28:0/1775390402 conn(0x560bb5a70080 msgr2=0x560bb5a5a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 shutdown_connections 2024-09-15T21:58:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 1 -- 172.21.15.28:0/1775390402 wait complete. 2024-09-15T21:58:02.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.929+0000 7f5103c51740 20 client.15732 trim_cache size 0 max 16384 2024-09-15T21:58:02.938 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:58:02.938 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T21:58:02.938 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T21:58:02.938 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e40a35e0 msgr2=0x7f50e40a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e40a35e0 0x7f50e40a5aa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f50f0000c00 tx=0x7f50f0004120 comp rx=0 tx=0).stop 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc100e20 msgr2=0x7f50fc0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc100e20 0x7f50fc0fcc80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00c890 tx=0x7f50ec00cd50 comp rx=0 tx=0).stop 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 shutdown_connections 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f50e40a35e0 0x7f50e40a5aa0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc102ef0 0x7f50fc0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50fc102570 0x7f50fc0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 --2- 172.21.15.28:0/2648640652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50fc100e20 0x7f50fc0fcc80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 >> 172.21.15.28:0/2648640652 conn(0x7f50fc0fc7d0 msgr2=0x7f50fc06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 shutdown_connections 2024-09-15T21:58:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:02.930+0000 7f5101878640 1 -- 172.21.15.28:0/2648640652 wait complete. 2024-09-15T21:58:03.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:03 smithi028 ceph-mon[30969]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-15T21:58:03.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:03 smithi028 ceph-mon[30969]: mgrmap e24: x(active, since 42s), standbys: z, y 2024-09-15T21:58:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:03 smithi203 ceph-mon[35546]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-15T21:58:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:03 smithi203 ceph-mon[35546]: mgrmap e24: x(active, since 42s), standbys: z, y 2024-09-15T21:58:03.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:03 smithi174 ceph-mon[35244]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-15T21:58:03.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:03 smithi174 ceph-mon[35244]: mgrmap e24: x(active, since 42s), standbys: z, y 2024-09-15T21:58:03.783 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T21:58:03.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:58:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/1622614916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 msgr2=0x7fb7bc107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/1622614916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc107ba0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009930 tx=0x7fb7a402a420 comp rx=0 tx=0).stop 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/1622614916 shutdown_connections 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/1622614916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc1082e0 0x7fb7bc10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/1622614916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc107ba0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/1622614916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7bc102db0 0x7fb7bc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/1622614916 >> 172.21.15.28:0/1622614916 conn(0x7fb7bc0fc7d0 msgr2=0x7fb7bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/1622614916 shutdown_connections 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.600+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/1622614916 wait complete. 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 Processor -- start 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 -- start start 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7bc102db0 0x7fb7bc1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc1082e0 0x7fb7bc19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc112050 con 0x7fb7bc103760 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc111ed0 con 0x7fb7bc102db0 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.601+0000 7fb7c0ba8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc1121d0 con 0x7fb7bc1082e0 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58262/0 (socket says 172.21.15.28:58262) 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 -- 172.21.15.28:0/4013678821 learned_addr learned my addr 172.21.15.28:0/4013678821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7ba59c640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7bc102db0 0x7fb7bc1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7bad9d640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc1082e0 0x7fb7bc19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 -- 172.21.15.28:0/4013678821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc1082e0 msgr2=0x7fb7bc19cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc1082e0 0x7fb7bc19cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 -- 172.21.15.28:0/4013678821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7bc102db0 msgr2=0x7fb7bc1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7bc102db0 0x7fb7bc1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 -- 172.21.15.28:0/4013678821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc19d580 con 0x7fb7bc103760 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7bad9d640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc1082e0 0x7fb7bc19cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b9d9b640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc198b10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009230 tx=0x7fb7a4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b37fe640 1 -- 172.21.15.28:0/4013678821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a403f070 con 0x7fb7bc103760 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b37fe640 1 -- 172.21.15.28:0/4013678821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a4031e30 con 0x7fb7bc103760 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc19d810 con 0x7fb7bc103760 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.602+0000 7fb7b37fe640 1 -- 172.21.15.28:0/4013678821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4035030 con 0x7fb7bc103760 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.603+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc1a5150 con 0x7fb7bc103760 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.604+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7bc06a7d0 con 0x7fb7bc103760 2024-09-15T21:58:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.605+0000 7fb7b37fe640 1 -- 172.21.15.28:0/4013678821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7fb7a4042070 con 0x7fb7bc103760 2024-09-15T21:58:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.606+0000 7fb7b37fe640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb7a80a3560 0x7fb7a80a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.606+0000 7fb7ba59c640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb7a80a3560 0x7fb7a80a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.606+0000 7fb7b37fe640 1 -- 172.21.15.28:0/4013678821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb7a40f0ca0 con 0x7fb7bc103760 2024-09-15T21:58:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.606+0000 7fb7ba59c640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb7a80a3560 0x7fb7a80a5a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac0058c0 tx=0x7fb7ac00a2d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.611+0000 7fb7b37fe640 1 -- 172.21.15.28:0/4013678821 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb7a40bd0e0 con 0x7fb7bc103760 2024-09-15T21:58:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.831+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7bc199930 con 0x7fb7bc103760 2024-09-15T21:58:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.832+0000 7fb7b37fe640 1 -- 172.21.15.28:0/4013678821 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fb7a40bc9e0 con 0x7fb7bc103760 2024-09-15T21:58:04.835 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:58:04.835 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:58:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:58:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb7a80a3560 msgr2=0x7fb7a80a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb7a80a3560 0x7fb7a80a5a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac0058c0 tx=0x7fb7ac00a2d0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 msgr2=0x7fb7bc198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc198b10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009230 tx=0x7fb7a4003c10 comp rx=0 tx=0).stop 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 shutdown_connections 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb7a80a3560 0x7fb7a80a5a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7bc1082e0 0x7fb7bc19cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7bc103760 0x7fb7bc198b10 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 --2- 172.21.15.28:0/4013678821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7bc102db0 0x7fb7bc1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.834+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 >> 172.21.15.28:0/4013678821 conn(0x7fb7bc0fc7d0 msgr2=0x7fb7bc106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.835+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 shutdown_connections 2024-09-15T21:58:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:04.835+0000 7fb7c0ba8640 1 -- 172.21.15.28:0/4013678821 wait complete. 2024-09-15T21:58:05.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T21:58:05.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:05 smithi203 ceph-mon[35546]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-09-15T21:58:05.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:05 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4013678821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:05 smithi174 ceph-mon[35244]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-09-15T21:58:05.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:05 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4013678821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:05 smithi028 ceph-mon[30969]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-09-15T21:58:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:05 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4013678821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 -- 172.21.15.28:0/805582885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102570 msgr2=0x7ff2d01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/805582885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102570 0x7ff2d01029b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0009900 tx=0x7ff2c002a480 comp rx=0 tx=0).stop 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 -- 172.21.15.28:0/805582885 shutdown_connections 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/805582885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2d0102ef0 0x7ff2d010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/805582885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102570 0x7ff2d01029b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/805582885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2d0100e20 0x7ff2d0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 -- 172.21.15.28:0/805582885 >> 172.21.15.28:0/805582885 conn(0x7ff2d00fc7d0 msgr2=0x7ff2d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.410+0000 7ff2d76a5640 1 -- 172.21.15.28:0/805582885 shutdown_connections 2024-09-15T21:58:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.411+0000 7ff2d76a5640 1 -- 172.21.15.28:0/805582885 wait complete. 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.411+0000 7ff2d76a5640 1 Processor -- start 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.411+0000 7ff2d76a5640 1 -- start start 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d76a5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2d0100e20 0x7ff2d019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d76a5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2d0102570 0x7ff2d019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d76a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102ef0 0x7ff2d019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d76a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0110f50 con 0x7ff2d0102ef0 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d76a5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d0110dd0 con 0x7ff2d0102570 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d76a5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01110d0 con 0x7ff2d0100e20 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102ef0 0x7ff2d019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102ef0 0x7ff2d019efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58292/0 (socket says 172.21.15.28:58292) 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 -- 172.21.15.28:0/3858501750 learned_addr learned my addr 172.21.15.28:0/3858501750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2d0102570 0x7ff2d019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2d0100e20 0x7ff2d019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 -- 172.21.15.28:0/3858501750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2d0100e20 msgr2=0x7ff2d019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2d0100e20 0x7ff2d019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 -- 172.21.15.28:0/3858501750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2d0102570 msgr2=0x7ff2d019ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2d0102570 0x7ff2d019ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 -- 172.21.15.28:0/3858501750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d019f6b0 con 0x7ff2d0102ef0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.412+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102ef0 0x7ff2d019efd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800cad0 tx=0x7ff2c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.413+0000 7ff2c67fc640 1 -- 172.21.15.28:0/3858501750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800ae20 con 0x7ff2d0102ef0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.413+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d019f9a0 con 0x7ff2d0102ef0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.413+0000 7ff2c67fc640 1 -- 172.21.15.28:0/3858501750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2c8005020 con 0x7ff2d0102ef0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.413+0000 7ff2c67fc640 1 -- 172.21.15.28:0/3858501750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800f6f0 con 0x7ff2d0102ef0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.413+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d01a7280 con 0x7ff2d0102ef0 2024-09-15T21:58:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.415+0000 7ff2c67fc640 1 -- 172.21.15.28:0/3858501750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7ff2c80051d0 con 0x7ff2d0102ef0 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.416+0000 7ff2c67fc640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff2bc0a35e0 0x7ff2bc0a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.416+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff2bc0a35e0 0x7ff2bc0a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.416+0000 7ff2c67fc640 1 -- 172.21.15.28:0/3858501750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7ff2c80d0300 con 0x7ff2d0102ef0 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.416+0000 7ff2d9a7e740 1 Processor -- start 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.416+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff2bc0a35e0 0x7ff2bc0a5aa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00bbe0 tx=0x7ff2cc00b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.416+0000 7ff2d9a7e740 1 -- start start 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e38090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e385d0 0x557219e38a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e45760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557219d9e2f0 con 0x557219e39b40 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557219d45cf0 con 0x557219e385d0 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557219d9a2b0 con 0x557219e43370 2024-09-15T21:58:06.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e38090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e38090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58300/0 (socket says 172.21.15.28:58300) 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 -- 172.21.15.28:0/1265994526 learned_addr learned my addr 172.21.15.28:0/1265994526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e385d0 0x557219e38a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e45760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 -- 172.21.15.28:0/1265994526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 msgr2=0x557219e45760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e45760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 -- 172.21.15.28:0/1265994526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e385d0 msgr2=0x557219e38a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e385d0 0x557219e38a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 -- 172.21.15.28:0/1265994526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557219d2a3a0 con 0x557219e39b40 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.417+0000 7ff2d5441640 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e38090 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc005d00 tx=0x7ff2cc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5050a6fcfcc5151d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.418+0000 7ff2bb7fe640 1 -- 172.21.15.28:0/1265994526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc005220 con 0x557219e39b40 2024-09-15T21:58:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.418+0000 7ff2bb7fe640 1 -- 172.21.15.28:0/1265994526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2cc004c80 con 0x557219e39b40 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.418+0000 7ff2bb7fe640 1 -- 172.21.15.28:0/1265994526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc01ea60 con 0x557219e39b40 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.418+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/1265994526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 msgr2=0x557219e38090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.418+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e38090 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc005d00 tx=0x7ff2cc013000 comp rx=0 tx=0).stop 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/1265994526 shutdown_connections 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e45760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e385d0 0x557219e38a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/1265994526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e38090 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/1265994526 >> 172.21.15.28:0/1265994526 conn(0x557219e01310 msgr2=0x557219e34c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/1265994526 shutdown_connections 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/1265994526 wait complete. 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 -- start start 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.419+0000 7ff2d9a7e740 1 Processor -- start 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e385d0 0x557219e42d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e40000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e405d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557219e46070 con 0x557219e39b40 2024-09-15T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557219d59060 con 0x557219e43370 2024-09-15T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557219e461f0 con 0x557219e385d0 2024-09-15T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d9a7e740 10 client.0 _ll_register_callbacks cb 0x557219aac8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e40000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e40000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58316/0 (socket says 172.21.15.28:58316) 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 -- 172.21.15.28:0/915131088 learned_addr learned my addr 172.21.15.28:0/915131088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d5441640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e385d0 0x557219e42d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 -- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e385d0 msgr2=0x557219e42d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e385d0 0x557219e42d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 -- 172.21.15.28:0/915131088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 msgr2=0x557219e405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e405d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 -- 172.21.15.28:0/915131088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557219b0f3c0 con 0x557219e39b40 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e40000 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff2c002a9a0 tx=0x7ff2c00044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfce4e79a3ce4144 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.420+0000 7ff2b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.423+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/915131088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0035040 con 0x557219e39b40 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.423+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/915131088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7ff2c0003b40 con 0x557219e39b40 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.423+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2cc004c80 con 0x557219e39b40 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.423+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/915131088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2c004e050 con 0x557219e39b40 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.423+0000 7ff2d9a7e740 10 client.15750 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.423+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557219e51af0 con 0x557219e39b40 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/915131088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2cc004c80 con 0x557219e39b40 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/915131088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff2c003c560 con 0x557219e39b40 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2d9a7e740 10 client.15750 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2d9a7e740 10 client.15750 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2d9a7e740 20 client.15750 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219e53000 0x557219e55440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2d9a7e740 4 client.15750 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:06.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557219dab860 con 0x557219e53000 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.424+0000 7ff2d5441640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219e53000 0x557219e55440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.425+0000 7ff2d5441640 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219e53000 0x557219e55440 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8858ec54eaa5e56 server_cookie=e4dd77fb7c6e1e18 in_seq=0 out_seq=0 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.425+0000 7ff2b8ff9640 10 client.15750 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.425+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/915131088 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557219dab860 con 0x557219e53000 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.425+0000 7ff2b8ff9640 10 client.15750 handle_command_reply: tid=0 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 client.15750 shutdown 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 msgr2=0x557219e40000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e40000 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff2c002a9a0 tx=0x7ff2c00044f0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 shutdown_connections 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219e53000 0x557219e55440 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e39b40 0x557219e40000 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/915131088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e385d0 0x557219e42d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 >> 172.21.15.28:0/915131088 conn(0x557219e01310 msgr2=0x557219e3a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2b8ff9640 0 client.15750 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 shutdown_connections 2024-09-15T21:58:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/915131088 wait complete. 2024-09-15T21:58:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.426+0000 7ff2d9a7e740 20 client.15750 trim_cache size 0 max 16384 2024-09-15T21:58:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 Processor -- start 2024-09-15T21:58:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 -- start start 2024-09-15T21:58:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e3ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e39b40 0x557219e4f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e5ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b1050 con 0x557219e385d0 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b2b90 con 0x557219e43370 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b1720 con 0x557219e39b40 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e3ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e3ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58326/0 (socket says 172.21.15.28:58326) 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 -- 172.21.15.28:0/3814064356 learned_addr learned my addr 172.21.15.28:0/3814064356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e5ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e39b40 0x557219e4f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 -- 172.21.15.28:0/3814064356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e39b40 msgr2=0x557219e4f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e39b40 0x557219e4f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 -- 172.21.15.28:0/3814064356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 msgr2=0x557219e5ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e5ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5441640 1 -- 172.21.15.28:0/3814064356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557219e35960 con 0x557219e385d0 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.441+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e5ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.442+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e3ffd0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc010860 tx=0x7ff2cc011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfd14f5d8e5c5b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.442+0000 7ff2baffd640 1 -- 172.21.15.28:0/3814064356 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc033430 con 0x557219e385d0 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.442+0000 7ff2baffd640 1 -- 172.21.15.28:0/3814064356 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2cc033ab0 con 0x557219e385d0 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.442+0000 7ff2baffd640 1 -- 172.21.15.28:0/3814064356 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc01e9b0 con 0x557219e385d0 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3814064356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 msgr2=0x557219e3ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e3ffd0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc010860 tx=0x7ff2cc011f10 comp rx=0 tx=0).stop 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3814064356 shutdown_connections 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e43370 0x557219e5ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e39b40 0x557219e4f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3814064356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e3ffd0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3814064356 >> 172.21.15.28:0/3814064356 conn(0x557219e01310 msgr2=0x557219e3f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3814064356 shutdown_connections 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3814064356 wait complete. 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 -- start start 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.443+0000 7ff2d9a7e740 1 Processor -- start 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e46440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e39b40 0x557219e46980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d9a7e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e46ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b24c0 con 0x557219e385d0 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e46440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01b1df0 con 0x557219e39b40 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d9a7e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d01af030 con 0x557219e43370 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e46440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58336/0 (socket says 172.21.15.28:58336) 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 -- 172.21.15.28:0/3361994369 learned_addr learned my addr 172.21.15.28:0/3361994369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d9a7e740 10 client.0 _ll_register_callbacks cb 0x7ff2c0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5c42640 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e46ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e39b40 0x557219e46980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 -- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 msgr2=0x557219e46ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e46ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 -- 172.21.15.28:0/3361994369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e39b40 msgr2=0x557219e46980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e39b40 0x557219e46980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.444+0000 7ff2d5441640 1 -- 172.21.15.28:0/3361994369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557219ebbdd0 con 0x557219e385d0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2d5441640 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e46440 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc005ba0 tx=0x7ff2cc01e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72fe392154edb02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/3361994369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc033630 con 0x557219e385d0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/3361994369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7ff2cc0337d0 con 0x557219e385d0 2024-09-15T21:58:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2cc033ab0 con 0x557219e385d0 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/3361994369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2cc04f050 con 0x557219e385d0 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2d9a7e740 10 client.15756 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557219ebd0d0 con 0x557219e385d0 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.445+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/3361994369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2cc041070 con 0x557219e385d0 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/3361994369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff2cc01ec30 con 0x557219e385d0 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d9a7e740 10 client.15756 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d9a7e740 10 client.15756 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d9a7e740 20 client.15756 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219ebd5f0 0x557219ebf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d9a7e740 4 client.15756 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff2cc033ab0 con 0x557219ebd5f0 2024-09-15T21:58:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219ebd5f0 0x557219ebf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2d4c40640 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219ebd5f0 0x557219ebf9e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=627c4dd893c5f759 server_cookie=a200e8854b41f8cf in_seq=0 out_seq=0 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.446+0000 7ff2b8ff9640 10 client.15756 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2b8ff9640 1 -- 172.21.15.28:0/3361994369 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff2cc033ab0 con 0x557219ebd5f0 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2b8ff9640 10 client.15756 handle_command_reply: tid=0 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 client.15756 shutdown 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 msgr2=0x557219e46440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e46440 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc005ba0 tx=0x7ff2cc01e420 comp rx=0 tx=0).stop 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 shutdown_connections 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x557219ebd5f0 0x557219ebf9e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557219e43370 0x557219e46ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557219e39b40 0x557219e46980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 --2- 172.21.15.28:0/3361994369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557219e385d0 0x557219e46440 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 >> 172.21.15.28:0/3361994369 conn(0x557219e01310 msgr2=0x557219e3e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2b8ff9640 0 client.15756 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 shutdown_connections 2024-09-15T21:58:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 1 -- 172.21.15.28:0/3361994369 wait complete. 2024-09-15T21:58:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.447+0000 7ff2d9a7e740 20 client.15756 trim_cache size 0 max 16384 2024-09-15T21:58:06.459 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff2bc0a35e0 msgr2=0x7ff2bc0a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff2bc0a35e0 0x7ff2bc0a5aa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00bbe0 tx=0x7ff2cc00b390 comp rx=0 tx=0).stop 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102ef0 msgr2=0x7ff2d019efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102ef0 0x7ff2d019efd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800cad0 tx=0x7ff2c800cf90 comp rx=0 tx=0).stop 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 shutdown_connections 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff2bc0a35e0 0x7ff2bc0a5aa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d0102ef0 0x7ff2d019efd0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2d0102570 0x7ff2d019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 --2- 172.21.15.28:0/3858501750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2d0100e20 0x7ff2d019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 >> 172.21.15.28:0/3858501750 conn(0x7ff2d00fc7d0 msgr2=0x7ff2d006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 shutdown_connections 2024-09-15T21:58:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:06.448+0000 7ff2d76a5640 1 -- 172.21.15.28:0/3858501750 wait complete. 2024-09-15T21:58:07.500 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.2 2024-09-15T21:58:07.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:58:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:07 smithi203 ceph-mon[35546]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-15T21:58:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:07 smithi174 ceph-mon[35244]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-15T21:58:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:07 smithi028 ceph-mon[30969]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-09-15T21:58:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:09 smithi203 ceph-mon[35546]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-09-15T21:58:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:09 smithi174 ceph-mon[35244]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-09-15T21:58:09.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:09 smithi028 ceph-mon[30969]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-09-15T21:58:10.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T21:58:10.213 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.730+0000 7f0a33994640 1 -- 172.21.15.28:0/1407292984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c100e20 msgr2=0x7f0a2c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.730+0000 7f0a33994640 1 --2- 172.21.15.28:0/1407292984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c100e20 0x7f0a2c101200 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009900 tx=0x7f0a2802a480 comp rx=0 tx=0).stop 2024-09-15T21:58:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 -- 172.21.15.28:0/1407292984 shutdown_connections 2024-09-15T21:58:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 --2- 172.21.15.28:0/1407292984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a2c102ef0 0x7f0a2c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 --2- 172.21.15.28:0/1407292984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2c102570 0x7f0a2c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 --2- 172.21.15.28:0/1407292984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c100e20 0x7f0a2c101200 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 -- 172.21.15.28:0/1407292984 >> 172.21.15.28:0/1407292984 conn(0x7f0a2c0fc7d0 msgr2=0x7f0a2c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 -- 172.21.15.28:0/1407292984 shutdown_connections 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 -- 172.21.15.28:0/1407292984 wait complete. 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 Processor -- start 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.731+0000 7f0a33994640 1 -- start start 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a33994640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2c100e20 0x7f0a2c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a33994640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102570 0x7f0a2c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a33994640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a2c102ef0 0x7f0a2c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a33994640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c110f80 con 0x7f0a2c102570 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a33994640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c110e00 con 0x7f0a2c102ef0 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a33994640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c111100 con 0x7f0a2c100e20 2024-09-15T21:58:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102570 0x7f0a2c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102570 0x7f0a2c198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55464/0 (socket says 172.21.15.28:55464) 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 -- 172.21.15.28:0/1258076243 learned_addr learned my addr 172.21.15.28:0/1258076243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a31730640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2c100e20 0x7f0a2c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a31f31640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a2c102ef0 0x7f0a2c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 -- 172.21.15.28:0/1258076243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2c100e20 msgr2=0x7f0a2c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2c100e20 0x7f0a2c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 -- 172.21.15.28:0/1258076243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a2c102ef0 msgr2=0x7f0a2c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a2c102ef0 0x7f0a2c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.732+0000 7f0a30f2f640 1 -- 172.21.15.28:0/1258076243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2c19d4e0 con 0x7f0a2c102570 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.733+0000 7f0a30f2f640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102570 0x7f0a2c198a70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000d770 tx=0x7f0a2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.733+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/1258076243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20017070 con 0x7f0a2c102570 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.733+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2c19d7d0 con 0x7f0a2c102570 2024-09-15T21:58:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.733+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/1258076243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a20005500 con 0x7f0a2c102570 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.733+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/1258076243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20004e00 con 0x7f0a2c102570 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.733+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2c1a50b0 con 0x7f0a2c102570 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.734+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2c101200 con 0x7f0a2c102570 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.736+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/1258076243 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f0a20004fa0 con 0x7f0a2c102570 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.737+0000 7f0a1e7fc640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0a180a3560 0x7f0a180a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.737+0000 7f0a31730640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0a180a3560 0x7f0a180a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.737+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/1258076243 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f0a200cf990 con 0x7f0a2c102570 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.737+0000 7f0a31730640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0a180a3560 0x7f0a180a5a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009230 tx=0x7f0a280058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.741+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/1258076243 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0a2009b780 con 0x7f0a2c102570 2024-09-15T21:58:10.841 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.982+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a2c10a0a0 con 0x7f0a2c102570 2024-09-15T21:58:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.983+0000 7f0a1e7fc640 1 -- 172.21.15.28:0/1258076243 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f0a2009fa80 con 0x7f0a2c102570 2024-09-15T21:58:10.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:58:10.984 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:58:10.985 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0a180a3560 msgr2=0x7f0a180a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0a180a3560 0x7f0a180a5a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009230 tx=0x7f0a280058d0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102570 msgr2=0x7f0a2c198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102570 0x7f0a2c198a70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000d770 tx=0x7f0a2000dc30 comp rx=0 tx=0).stop 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 shutdown_connections 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0a180a3560 0x7f0a180a5a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a2c102ef0 0x7f0a2c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a2c102570 0x7f0a2c198a70 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 --2- 172.21.15.28:0/1258076243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2c100e20 0x7f0a2c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:10.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.985+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 >> 172.21.15.28:0/1258076243 conn(0x7f0a2c0fc7d0 msgr2=0x7f0a2c06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:10.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.986+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 shutdown_connections 2024-09-15T21:58:10.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:10.986+0000 7f0a33994640 1 -- 172.21.15.28:0/1258076243 wait complete. 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.080+0000 7feb16282640 1 -- 172.21.15.28:0/635625924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10101ad0 msgr2=0x7feb10101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.080+0000 7feb16282640 1 --2- 172.21.15.28:0/635625924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10101ad0 0x7feb10101f30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009930 tx=0x7feaf802a420 comp rx=0 tx=0).stop 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.080+0000 7feb16282640 1 -- 172.21.15.28:0/635625924 shutdown_connections 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.080+0000 7feb16282640 1 --2- 172.21.15.28:0/635625924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb10102470 0x7feb1010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.080+0000 7feb16282640 1 --2- 172.21.15.28:0/635625924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10101ad0 0x7feb10101f30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.080+0000 7feb16282640 1 --2- 172.21.15.28:0/635625924 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10107b70 0x7feb10107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.080+0000 7feb16282640 1 -- 172.21.15.28:0/635625924 >> 172.21.15.28:0/635625924 conn(0x7feb100fd7d0 msgr2=0x7feb100ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.081+0000 7feb16282640 1 -- 172.21.15.28:0/635625924 shutdown_connections 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.081+0000 7feb16282640 1 -- 172.21.15.28:0/635625924 wait complete. 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.081+0000 7feb16282640 1 Processor -- start 2024-09-15T21:58:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.081+0000 7feb16282640 1 -- start start 2024-09-15T21:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb16282640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb10101ad0 0x7feb100744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb16282640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102470 0x7feb10074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb16282640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10107b70 0x7feb10074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb16282640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb10111ee0 con 0x7feb10107b70 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb16282640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb10111d60 con 0x7feb10102470 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb16282640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb10112060 con 0x7feb10101ad0 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10107b70 0x7feb10074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10107b70 0x7feb10074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55484/0 (socket says 172.21.15.28:55484) 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 -- 172.21.15.28:0/3362752742 learned_addr learned my addr 172.21.15.28:0/3362752742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb07fff640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102470 0x7feb10074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb0ffff640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb10101ad0 0x7feb100744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 -- 172.21.15.28:0/3362752742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb10101ad0 msgr2=0x7feb100744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb10101ad0 0x7feb100744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 -- 172.21.15.28:0/3362752742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102470 msgr2=0x7feb10074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102470 0x7feb10074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 -- 172.21.15.28:0/3362752742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb101a68b0 con 0x7feb10107b70 2024-09-15T21:58:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb1481f640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10107b70 0x7feb10074f70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7feb08009fd0 tx=0x7feb0800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.082+0000 7feb0dffb640 1 -- 172.21.15.28:0/3362752742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08017070 con 0x7feb10107b70 2024-09-15T21:58:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.083+0000 7feb0dffb640 1 -- 172.21.15.28:0/3362752742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb08003e50 con 0x7feb10107b70 2024-09-15T21:58:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.083+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb101a6ba0 con 0x7feb10107b70 2024-09-15T21:58:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.083+0000 7feb0dffb640 1 -- 172.21.15.28:0/3362752742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080106c0 con 0x7feb10107b70 2024-09-15T21:58:11.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.083+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb101a7070 con 0x7feb10107b70 2024-09-15T21:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.085+0000 7feb19022740 1 Processor -- start 2024-09-15T21:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.085+0000 7feb0dffb640 1 -- 172.21.15.28:0/3362752742 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7feb08003970 con 0x7feb10107b70 2024-09-15T21:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb19022740 1 -- start start 2024-09-15T21:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb19022740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb19022740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceb9f9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb19022740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9f9cb0 0x563ceb9fc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb19022740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ceb9532f0 con 0x563ceb9f9cb0 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb19022740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563ceb8fb270 con 0x563ceb9ef410 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb19022740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ceb94f2b0 con 0x563ceb9eeaf0 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb07fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9f9cb0 0x563ceb9fc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb07fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9f9cb0 0x563ceb9fc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55488/0 (socket says 172.21.15.28:55488) 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb07fff640 1 -- 172.21.15.28:0/276754912 learned_addr learned my addr 172.21.15.28:0/276754912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb0ffff640 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceb9f9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb07fff640 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb0ffff640 1 -- 172.21.15.28:0/276754912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 msgr2=0x563ceb9eeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb0dffb640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feafc0a3610 0x7feafc0a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.086+0000 7feb0ffff640 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9eeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb0ffff640 1 -- 172.21.15.28:0/276754912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9f9cb0 msgr2=0x563ceb9fc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb0ffff640 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9f9cb0 0x563ceb9fc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb0ffff640 1 -- 172.21.15.28:0/276754912 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ceb8fa340 con 0x563ceb9ef410 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb1481f640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feafc0a3610 0x7feafc0a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb07fff640 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9eeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb0dffb640 1 -- 172.21.15.28:0/3362752742 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7feb080d00f0 con 0x7feb10107b70 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb0ffff640 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceb9f9770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feb0000bb60 tx=0x7feb0000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623e55becbe6f195 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb067fc640 1 -- 172.21.15.28:0/276754912 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00015070 con 0x563ceb9ef410 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb067fc640 1 -- 172.21.15.28:0/276754912 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0000be50 con 0x563ceb9ef410 2024-09-15T21:58:11.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.087+0000 7feb067fc640 1 -- 172.21.15.28:0/276754912 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00011440 con 0x563ceb9ef410 2024-09-15T21:58:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 -- 172.21.15.28:0/276754912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 msgr2=0x563ceb9f9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceb9f9770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feb0000bb60 tx=0x7feb0000f730 comp rx=0 tx=0).stop 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb1481f640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feafc0a3610 0x7feafc0a5ad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb080cd300 tx=0x7feb080152b0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 -- 172.21.15.28:0/276754912 shutdown_connections 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9f9cb0 0x563ceb9fc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceb9f9770 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 --2- 172.21.15.28:0/276754912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9eeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 -- 172.21.15.28:0/276754912 >> 172.21.15.28:0/276754912 conn(0x563ceb9eb060 msgr2=0x563ceb9eb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 -- 172.21.15.28:0/276754912 shutdown_connections 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.088+0000 7feb19022740 1 -- 172.21.15.28:0/276754912 wait complete. 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 -- start start 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 Processor -- start 2024-09-15T21:58:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ceb9fc880 con 0x563ceb9ef410 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55490/0 (socket says 172.21.15.28:55490) 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb0ffff640 1 -- 172.21.15.28:0/298842803 learned_addr learned my addr 172.21.15.28:0/298842803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563ceb8dc290 con 0x563ceb9f9cb0 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ceb9fca00 con 0x563ceb9eeaf0 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.089+0000 7feb19022740 10 client.0 _ll_register_callbacks cb 0x563ceb90d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb07fff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb07fff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb0ffff640 1 -- 172.21.15.28:0/298842803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 msgr2=0x563ceb9f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb0ffff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb0ffff640 1 -- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 msgr2=0x563ceb9f75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb0ffff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb0ffff640 1 -- 172.21.15.28:0/298842803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ceb666210 con 0x563ceb9ef410 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb07fff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb0ffff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f7070 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feb00005780 tx=0x7feb0000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c23e5fb812385c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feaef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feaef7fe640 1 -- 172.21.15.28:0/298842803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00028060 con 0x563ceb9ef410 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feaef7fe640 1 -- 172.21.15.28:0/298842803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7feb0000bc50 con 0x563ceb9ef410 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0000be50 con 0x563ceb9ef410 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feaef7fe640 1 -- 172.21.15.28:0/298842803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb0002f050 con 0x563ceb9ef410 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb19022740 10 client.15774 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:11.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.090+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563ceb732620 con 0x563ceb9ef410 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feaef7fe640 1 -- 172.21.15.28:0/298842803 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0000be50 con 0x563ceb9ef410 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feaef7fe640 1 -- 172.21.15.28:0/298842803 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feb00011af0 con 0x563ceb9ef410 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feb19022740 10 client.15774 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feb19022740 10 client.15774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feb19022740 20 client.15774 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feb19022740 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba09cb0 0x563ceba0c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feb19022740 4 client.15774 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ceb960a20 con 0x563ceba09cb0 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.091+0000 7feb07fff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba09cb0 0x563ceba0c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.092+0000 7feb07fff640 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba09cb0 0x563ceba0c0a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9f1f0c48e27f8d1 server_cookie=e623744caac681d0 in_seq=0 out_seq=0 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.092+0000 7feaef7fe640 10 client.15774 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:11.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feaef7fe640 1 -- 172.21.15.28:0/298842803 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563ceb960a20 con 0x563ceba09cb0 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feaef7fe640 10 client.15774 handle_command_reply: tid=0 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 client.15774 shutdown 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 msgr2=0x563ceb9f7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f7070 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feb00005780 tx=0x7feb0000b8a0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 shutdown_connections 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba09cb0 0x563ceba0c0a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f7070 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 --2- 172.21.15.28:0/298842803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 >> 172.21.15.28:0/298842803 conn(0x563ceb9eb060 msgr2=0x563ceb9f0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feaef7fe640 0 client.15774 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:11.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 shutdown_connections 2024-09-15T21:58:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 1 -- 172.21.15.28:0/298842803 wait complete. 2024-09-15T21:58:11.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.093+0000 7feb19022740 20 client.15774 trim_cache size 0 max 16384 2024-09-15T21:58:11.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 Processor -- start 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 -- start start 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb101b1000 con 0x563ceb9ef410 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb101b2b40 con 0x563ceb9f9cb0 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.125+0000 7feb19022740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb101b16d0 con 0x563ceb9eeaf0 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:37346/0 (socket says 172.21.15.28:37346) 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 -- 172.21.15.28:0/2008479618 learned_addr learned my addr 172.21.15.28:0/2008479618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb0ffff640 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 -- 172.21.15.28:0/2008479618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 msgr2=0x563ceb9f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 -- 172.21.15.28:0/2008479618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 msgr2=0x563ceb9f8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 -- 172.21.15.28:0/2008479618 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ceb9e62d0 con 0x563ceb9eeaf0 2024-09-15T21:58:11.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb07fff640 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f73e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feaf8005940 tx=0x7feaf802a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf867104a8b50c18 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb05ffb640 1 -- 172.21.15.28:0/2008479618 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8052040 con 0x563ceb9eeaf0 2024-09-15T21:58:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb05ffb640 1 -- 172.21.15.28:0/2008479618 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaf8005500 con 0x563ceb9eeaf0 2024-09-15T21:58:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.126+0000 7feb05ffb640 1 -- 172.21.15.28:0/2008479618 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8035040 con 0x563ceb9eeaf0 2024-09-15T21:58:11.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.127+0000 7feb19022740 1 -- 172.21.15.28:0/2008479618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 msgr2=0x563ceb9f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.127+0000 7feb19022740 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f73e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feaf8005940 tx=0x7feaf802a420 comp rx=0 tx=0).stop 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 -- 172.21.15.28:0/2008479618 shutdown_connections 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9f9cb0 0x563ceb9f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9ef410 0x563ceb9f8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 --2- 172.21.15.28:0/2008479618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9eeaf0 0x563ceb9f73e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 -- 172.21.15.28:0/2008479618 >> 172.21.15.28:0/2008479618 conn(0x563ceb9eb060 msgr2=0x563ceb9f8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 -- 172.21.15.28:0/2008479618 shutdown_connections 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 -- 172.21.15.28:0/2008479618 wait complete. 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.128+0000 7feb19022740 1 -- start start 2024-09-15T21:58:11.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb19022740 1 Processor -- start 2024-09-15T21:58:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb19022740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9eeaf0 0x563ceba0ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb19022740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceba0f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb07fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9eeaf0 0x563ceba0ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb19022740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9f9cb0 0x563ceba0a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb07fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9eeaf0 0x563ceba0ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55500/0 (socket says 172.21.15.28:55500) 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb07fff640 1 -- 172.21.15.28:0/1174603610 learned_addr learned my addr 172.21.15.28:0/1174603610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb101b2470 con 0x563ceb9eeaf0 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.129+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb101b1da0 con 0x563ceb9ef410 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb101aefe0 con 0x563ceb9f9cb0 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb19022740 10 client.0 _ll_register_callbacks cb 0x7feaf80095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb0ffff640 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceba0f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb07fff640 1 -- 172.21.15.28:0/1174603610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9f9cb0 msgr2=0x563ceba0a300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb07fff640 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9f9cb0 0x563ceba0a300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb07fff640 1 -- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 msgr2=0x563ceba0f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:11.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb07fff640 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceba0f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb07fff640 1 -- 172.21.15.28:0/1174603610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ceba0ab30 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feb07fff640 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9eeaf0 0x563ceba0ef80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feaf802abb0 tx=0x7feaf8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296fa36a9ae56a16 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.130+0000 7feaef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feaef7fe640 1 -- 172.21.15.28:0/1174603610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8035030 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feaef7fe640 1 -- 172.21.15.28:0/1174603610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7feaf8004030 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaf8005500 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feaef7fe640 1 -- 172.21.15.28:0/1174603610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feaf8004230 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feb19022740 10 client.15780 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563ceba74150 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feaef7fe640 1 -- 172.21.15.28:0/1174603610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaf8005500 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.131+0000 7feaef7fe640 1 -- 172.21.15.28:0/1174603610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feaf80316d0 con 0x563ceb9eeaf0 2024-09-15T21:58:11.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb19022740 10 client.15780 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb19022740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb19022740 20 client.15780 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb19022740 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba747f0 0x563ceba76c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb19022740 4 client.15780 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ceb95ad70 con 0x563ceba747f0 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb0ffff640 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba747f0 0x563ceba76c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feb0ffff640 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba747f0 0x563ceba76c10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=864d62a86bdd1383 server_cookie=fab7862f62baeb8a in_seq=0 out_seq=0 2024-09-15T21:58:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.132+0000 7feaef7fe640 10 client.15780 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feaef7fe640 1 -- 172.21.15.28:0/1174603610 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563ceb95ad70 con 0x563ceba747f0 2024-09-15T21:58:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feaef7fe640 10 client.15780 handle_command_reply: tid=0 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 client.15780 shutdown 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9eeaf0 msgr2=0x563ceba0ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9eeaf0 0x563ceba0ef80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feaf802abb0 tx=0x7feaf8005d50 comp rx=0 tx=0).stop 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 shutdown_connections 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563ceba747f0 0x563ceba76c10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ceb9f9cb0 0x563ceba0a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ceb9ef410 0x563ceba0f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feaef7fe640 0 client.15780 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 --2- 172.21.15.28:0/1174603610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ceb9eeaf0 0x563ceba0ef80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 >> 172.21.15.28:0/1174603610 conn(0x563ceb9eb060 msgr2=0x563ceb9f0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:11.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 shutdown_connections 2024-09-15T21:58:11.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 1 -- 172.21.15.28:0/1174603610 wait complete. 2024-09-15T21:58:11.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.142+0000 7feb19022740 20 client.15780 trim_cache size 0 max 16384 2024-09-15T21:58:11.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T21:58:11.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T21:58:11.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T21:58:11.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T21:58:11.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feafc0a3610 msgr2=0x7feafc0a5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feafc0a3610 0x7feafc0a5ad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb080cd300 tx=0x7feb080152b0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10107b70 msgr2=0x7feb10074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10107b70 0x7feb10074f70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7feb08009fd0 tx=0x7feb0800b260 comp rx=0 tx=0).stop 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 shutdown_connections 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feafc0a3610 0x7feafc0a5ad0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb10107b70 0x7feb10074f70 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb10102470 0x7feb10074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 --2- 172.21.15.28:0/3362752742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb10101ad0 0x7feb100744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 >> 172.21.15.28:0/3362752742 conn(0x7feb100fd7d0 msgr2=0x7feb100fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 shutdown_connections 2024-09-15T21:58:11.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:11.143+0000 7feb16282640 1 -- 172.21.15.28:0/3362752742 wait complete. 2024-09-15T21:58:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:11 smithi028 ceph-mon[30969]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-09-15T21:58:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:11 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1258076243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:11.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:11 smithi203 ceph-mon[35546]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-09-15T21:58:11.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:11 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1258076243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:11 smithi174 ceph-mon[35244]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-09-15T21:58:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:11 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1258076243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:11.847 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0' 2024-09-15T21:58:11.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0 2024-09-15T21:58:12.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 -- 172.21.15.28:0/1658820588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 msgr2=0x7ff1e41067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/1658820588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e41067f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009900 tx=0x7ff1d402a410 comp rx=0 tx=0).stop 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 -- 172.21.15.28:0/1658820588 shutdown_connections 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/1658820588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e4106f20 0x7ff1e410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/1658820588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e41067f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/1658820588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1e4101df0 0x7ff1e41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 -- 172.21.15.28:0/1658820588 >> 172.21.15.28:0/1658820588 conn(0x7ff1e40fd8f0 msgr2=0x7ff1e40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.103+0000 7ff1eb46d640 1 -- 172.21.15.28:0/1658820588 shutdown_connections 2024-09-15T21:58:12.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.104+0000 7ff1eb46d640 1 -- 172.21.15.28:0/1658820588 wait complete. 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 Processor -- start 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 -- start start 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1e4101df0 0x7ff1e419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e4106f20 0x7ff1e419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e4111fd0 con 0x7ff1e41027a0 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e4111e50 con 0x7ff1e4106f20 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1eb46d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e4112150 con 0x7ff1e4101df0 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1e8a08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1e8a08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e419a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55524/0 (socket says 172.21.15.28:55524) 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1e8a08640 1 -- 172.21.15.28:0/85215554 learned_addr learned my addr 172.21.15.28:0/85215554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:12.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1e9209640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1e4101df0 0x7ff1e419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.105+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e4106f20 0x7ff1e419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1e8a08640 1 -- 172.21.15.28:0/85215554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1e4101df0 msgr2=0x7ff1e419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1e4101df0 0x7ff1e419a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1e8a08640 1 -- 172.21.15.28:0/85215554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e4106f20 msgr2=0x7ff1e419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e4106f20 0x7ff1e419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1e8a08640 1 -- 172.21.15.28:0/85215554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e419f430 con 0x7ff1e41027a0 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e419a9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009850 tx=0x7ff1d4030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1da7fc640 1 -- 172.21.15.28:0/85215554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d403f070 con 0x7ff1e41027a0 2024-09-15T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e419f6c0 con 0x7ff1e41027a0 2024-09-15T21:58:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1da7fc640 1 -- 172.21.15.28:0/85215554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d4004460 con 0x7ff1e41027a0 2024-09-15T21:58:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1da7fc640 1 -- 172.21.15.28:0/85215554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4035030 con 0x7ff1e41027a0 2024-09-15T21:58:12.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.106+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e41a7000 con 0x7ff1e41027a0 2024-09-15T21:58:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.110+0000 7ff1da7fc640 1 -- 172.21.15.28:0/85215554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7ff1d4042070 con 0x7ff1e41027a0 2024-09-15T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.111+0000 7ff1ed846740 1 Processor -- start 2024-09-15T21:58:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.111+0000 7ff1ed846740 1 -- start start 2024-09-15T21:58:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.111+0000 7ff1da7fc640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1d00a3560 0x7ff1d00a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.111+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd09090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.111+0000 7ff1da7fc640 1 -- 172.21.15.28:0/85215554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7ff1d40f0920 con 0x7ff1e41027a0 2024-09-15T21:58:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.111+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1d00a3560 0x7ff1d00a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd09a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1ed846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558eccc6f2f0 con 0x558eccd095d0 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1ed846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558eccc16cf0 con 0x558eccd14370 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1ed846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558eccc6b2b0 con 0x558eccd0ab40 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1d00a3560 0x7ff1d00a5a20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc005fd0 tx=0x7ff1dc009500 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1e8a08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd09a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1e8a08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd09a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55534/0 (socket says 172.21.15.28:55534) 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1e8a08640 1 -- 172.21.15.28:0/849662409 learned_addr learned my addr 172.21.15.28:0/849662409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1e9209640 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd09090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.112+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.113+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/849662409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 msgr2=0x558eccd09090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.113+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd09090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.113+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/849662409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 msgr2=0x558eccd09a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.113+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd09a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.113+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/849662409 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eccbfb3a0 con 0x558eccd14370 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.113+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd16760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc003b00 tx=0x7ff1dc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbeb60d4c499ca85 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.114+0000 7ff1cf7fe640 1 -- 172.21.15.28:0/849662409 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc004b20 con 0x558eccd14370 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.114+0000 7ff1cf7fe640 1 -- 172.21.15.28:0/849662409 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1dc004cc0 con 0x558eccd14370 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.114+0000 7ff1cf7fe640 1 -- 172.21.15.28:0/849662409 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc01e630 con 0x558eccd14370 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 -- 172.21.15.28:0/849662409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 msgr2=0x558eccd16760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd16760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc003b00 tx=0x7ff1dc014000 comp rx=0 tx=0).stop 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 -- 172.21.15.28:0/849662409 shutdown_connections 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd16760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd09a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 --2- 172.21.15.28:0/849662409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd09090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 -- 172.21.15.28:0/849662409 >> 172.21.15.28:0/849662409 conn(0x558ecccd2310 msgr2=0x558eccd05c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 -- 172.21.15.28:0/849662409 shutdown_connections 2024-09-15T21:58:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.115+0000 7ff1ed846740 1 -- 172.21.15.28:0/849662409 wait complete. 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.116+0000 7ff1ed846740 1 -- start start 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.116+0000 7ff1ed846740 1 Processor -- start 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.116+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd13c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd10f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd11480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1ed846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558eccd17070 con 0x558eccd095d0 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9209640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd13c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1ed846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558eccc2a060 con 0x558eccd14370 2024-09-15T21:58:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1ed846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558eccd171f0 con 0x558eccd0ab40 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9209640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd13c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55550/0 (socket says 172.21.15.28:55550) 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9209640 1 -- 172.21.15.28:0/3192414654 learned_addr learned my addr 172.21.15.28:0/3192414654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1ed846740 10 client.0 _ll_register_callbacks cb 0x558ecc97d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd11480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd10f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/3192414654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 msgr2=0x558eccd10f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd10f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/3192414654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 msgr2=0x558eccd13c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.117+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd13c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.118+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/3192414654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ecc9e03c0 con 0x558eccd14370 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.118+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd11480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc004020 tx=0x7ff1dc004c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a9800f1986c894c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.118+0000 7ff1ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.118+0000 7ff1ccff9640 1 -- 172.21.15.28:0/3192414654 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc005180 con 0x558eccd14370 2024-09-15T21:58:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.118+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1dc004cc0 con 0x558eccd14370 2024-09-15T21:58:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.118+0000 7ff1ccff9640 1 -- 172.21.15.28:0/3192414654 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7ff1dc005320 con 0x558eccd14370 2024-09-15T21:58:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.118+0000 7ff1ccff9640 1 -- 172.21.15.28:0/3192414654 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1dc005520 con 0x558eccd14370 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.119+0000 7ff1ed846740 10 client.25403 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.119+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558eccd22a10 con 0x558eccd14370 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.119+0000 7ff1ccff9640 1 -- 172.21.15.28:0/3192414654 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1dc004cc0 con 0x558eccd14370 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.119+0000 7ff1ccff9640 1 -- 172.21.15.28:0/3192414654 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1dc0266d0 con 0x558eccd14370 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.120+0000 7ff1ed846740 10 client.25403 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.120+0000 7ff1ed846740 10 client.25403 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.120+0000 7ff1ed846740 20 client.25403 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.120+0000 7ff1ed846740 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd261b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.120+0000 7ff1ed846740 4 client.25403 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.120+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558eccc7c860 con 0x558eccd23dc0 2024-09-15T21:58:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.121+0000 7ff1e9209640 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd261b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.121+0000 7ff1ccff9640 10 client.25403 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.121+0000 7ff1e9209640 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd261b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e7e8d0e8ac371cb9 server_cookie=84ca5328eb0c16f5 in_seq=0 out_seq=0 2024-09-15T21:58:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.122+0000 7ff1ccff9640 1 -- 172.21.15.28:0/3192414654 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558eccc7c860 con 0x558eccd23dc0 2024-09-15T21:58:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.122+0000 7ff1ccff9640 10 client.25403 handle_command_reply: tid=0 2024-09-15T21:58:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.122+0000 7ff1ed846740 1 client.25403 shutdown 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 msgr2=0x558eccd11480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd11480 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc004020 tx=0x7ff1dc004c50 comp rx=0 tx=0).stop 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 shutdown_connections 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd11480 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd10f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd261b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 --2- 172.21.15.28:0/3192414654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd095d0 0x558eccd13c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ccff9640 0 client.25403 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.123+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 >> 172.21.15.28:0/3192414654 conn(0x558ecccd2310 msgr2=0x558eccd0b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.124+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 shutdown_connections 2024-09-15T21:58:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.124+0000 7ff1ed846740 1 -- 172.21.15.28:0/3192414654 wait complete. 2024-09-15T21:58:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.124+0000 7ff1ed846740 20 client.25403 trim_cache size 0 max 16384 2024-09-15T21:58:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 Processor -- start 2024-09-15T21:58:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 -- start start 2024-09-15T21:58:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd095d0 0x558eccd00c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd20310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd14370 0x558eccd12eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e41b11e0 con 0x558eccd14370 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e41b2d20 con 0x558eccd095d0 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1ed846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e41b18b0 con 0x558eccd0ab40 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1e9a0a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd14370 0x558eccd12eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1e9a0a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd14370 0x558eccd12eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55564/0 (socket says 172.21.15.28:55564) 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/4067758164 learned_addr learned my addr 172.21.15.28:0/4067758164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1e9209640 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd095d0 0x558eccd00c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.139+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd20310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/4067758164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 msgr2=0x558eccd20310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd20310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/4067758164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd095d0 msgr2=0x558eccd00c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd095d0 0x558eccd00c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1e9a0a640 1 -- 172.21.15.28:0/4067758164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eccd06960 con 0x558eccd14370 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd14370 0x558eccd12eb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc011ca0 tx=0x7ff1dc005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c65f8008362a27c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1ceffd640 1 -- 172.21.15.28:0/4067758164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc004cc0 con 0x558eccd14370 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1ceffd640 1 -- 172.21.15.28:0/4067758164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1dc02b070 con 0x558eccd14370 2024-09-15T21:58:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.140+0000 7ff1ceffd640 1 -- 172.21.15.28:0/4067758164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc01e980 con 0x558eccd14370 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 -- 172.21.15.28:0/4067758164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd14370 msgr2=0x558eccd12eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd14370 0x558eccd12eb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc011ca0 tx=0x7ff1dc005910 comp rx=0 tx=0).stop 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 -- 172.21.15.28:0/4067758164 shutdown_connections 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd14370 0x558eccd12eb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd0ab40 0x558eccd20310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 --2- 172.21.15.28:0/4067758164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd095d0 0x558eccd00c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 -- 172.21.15.28:0/4067758164 >> 172.21.15.28:0/4067758164 conn(0x558ecccd2310 msgr2=0x558eccd127a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.141+0000 7ff1ed846740 1 -- 172.21.15.28:0/4067758164 shutdown_connections 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.142+0000 7ff1ed846740 1 -- 172.21.15.28:0/4067758164 wait complete. 2024-09-15T21:58:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.142+0000 7ff1ed846740 1 -- start start 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.142+0000 7ff1ed846740 1 Processor -- start 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd095d0 0x558eccd1c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd0ab40 0x558eccd1cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1ed846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd17d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1ed846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e41b2650 con 0x558eccd0ab40 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1ed846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e41b1f80 con 0x558eccd14370 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1ed846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e41af1c0 con 0x558eccd095d0 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1ed846740 10 client.0 _ll_register_callbacks cb 0x7ff1dc003a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd0ab40 0x558eccd1cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd0ab40 0x558eccd1cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55570/0 (socket says 172.21.15.28:55570) 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 -- 172.21.15.28:0/2214080672 learned_addr learned my addr 172.21.15.28:0/2214080672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd17d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e9209640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd095d0 0x558eccd1c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 -- 172.21.15.28:0/2214080672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd095d0 msgr2=0x558eccd1c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd095d0 0x558eccd1c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 -- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 msgr2=0x558eccd17d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd17d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.143+0000 7ff1e8a08640 1 -- 172.21.15.28:0/2214080672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eccd18580 con 0x558eccd0ab40 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1e9a0a640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd17d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1e8a08640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd0ab40 0x558eccd1cd60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40edc10 tx=0x7ff1d4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb55ff79a89a149e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ccff9640 1 -- 172.21.15.28:0/2214080672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4049040 con 0x558eccd0ab40 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ccff9640 1 -- 172.21.15.28:0/2214080672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7ff1d40bc970 con 0x558eccd0ab40 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1dc02b070 con 0x558eccd0ab40 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ccff9640 1 -- 172.21.15.28:0/2214080672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1d400b0b0 con 0x558eccd0ab40 2024-09-15T21:58:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ed846740 10 client.15798 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558eccd96c50 con 0x558eccd0ab40 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.144+0000 7ff1ccff9640 1 -- 172.21.15.28:0/2214080672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d40bc710 con 0x558eccd0ab40 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1ccff9640 1 -- 172.21.15.28:0/2214080672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1d409c030 con 0x558eccd0ab40 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1ed846740 10 client.15798 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1ed846740 10 client.15798 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1ed846740 20 client.15798 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1ed846740 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1ed846740 4 client.15798 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff1dc02b070 con 0x558eccd23dc0 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.145+0000 7ff1e9209640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.146+0000 7ff1e9209640 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd994e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3ead0f23acc68fff server_cookie=d82c5b3aaf84fe49 in_seq=0 out_seq=0 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.146+0000 7ff1ccff9640 10 client.15798 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.146+0000 7ff1ccff9640 1 -- 172.21.15.28:0/2214080672 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff1dc02b070 con 0x558eccd23dc0 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.146+0000 7ff1ccff9640 10 client.15798 handle_command_reply: tid=0 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.146+0000 7ff1ed846740 1 client.15798 shutdown 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.146+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd0ab40 msgr2=0x558eccd1cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.146+0000 7ff1ed846740 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd0ab40 0x558eccd1cd60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40edc10 tx=0x7ff1d4005fd0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 shutdown_connections 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558eccd14370 0x558eccd17d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558eccd23dc0 0x558eccd994e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558eccd0ab40 0x558eccd1cd60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 --2- 172.21.15.28:0/2214080672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558eccd095d0 0x558eccd1c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ccff9640 0 client.15798 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 >> 172.21.15.28:0/2214080672 conn(0x558ecccd2310 msgr2=0x558eccd0ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:12.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 shutdown_connections 2024-09-15T21:58:12.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 1 -- 172.21.15.28:0/2214080672 wait complete. 2024-09-15T21:58:12.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.147+0000 7ff1ed846740 20 client.15798 trim_cache size 0 max 16384 2024-09-15T21:58:12.152 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1d00a3560 msgr2=0x7ff1d00a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1d00a3560 0x7ff1d00a5a20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc005fd0 tx=0x7ff1dc009500 comp rx=0 tx=0).stop 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 msgr2=0x7ff1e419a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e419a9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009850 tx=0x7ff1d4030650 comp rx=0 tx=0).stop 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 shutdown_connections 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1d00a3560 0x7ff1d00a5a20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e4106f20 0x7ff1e419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e41027a0 0x7ff1e419a9c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 --2- 172.21.15.28:0/85215554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1e4101df0 0x7ff1e419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 >> 172.21.15.28:0/85215554 conn(0x7ff1e40fd8f0 msgr2=0x7ff1e410cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 shutdown_connections 2024-09-15T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:12.148+0000 7ff1eb46d640 1 -- 172.21.15.28:0/85215554 wait complete. 2024-09-15T21:58:12.410 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:13.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T21:58:13.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:13 smithi028 ceph-mon[30969]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 30 op/s 2024-09-15T21:58:13.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 -- 172.21.15.28:0/1604761827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102ef0 msgr2=0x7f9d1c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 --2- 172.21.15.28:0/1604761827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102ef0 0x7f9d1c10da40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009ca0 tx=0x7f9d1402a950 comp rx=0 tx=0).stop 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 -- 172.21.15.28:0/1604761827 shutdown_connections 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 --2- 172.21.15.28:0/1604761827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102ef0 0x7f9d1c10da40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 --2- 172.21.15.28:0/1604761827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c102570 0x7f9d1c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 --2- 172.21.15.28:0/1604761827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d1c100e20 0x7f9d1c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 -- 172.21.15.28:0/1604761827 >> 172.21.15.28:0/1604761827 conn(0x7f9d1c0fc7d0 msgr2=0x7f9d1c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 -- 172.21.15.28:0/1604761827 shutdown_connections 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.567+0000 7f9d24416640 1 -- 172.21.15.28:0/1604761827 wait complete. 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.568+0000 7f9d24416640 1 Processor -- start 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.568+0000 7f9d24416640 1 -- start start 2024-09-15T21:58:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d24416640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c100e20 0x7f9d1c1a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d24416640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102570 0x7f9d1c1a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d24416640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d1c102ef0 0x7f9d1c10cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d24416640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1120a0 con 0x7f9d1c102570 2024-09-15T21:58:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d24416640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c111f20 con 0x7f9d1c100e20 2024-09-15T21:58:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d24416640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c112220 con 0x7f9d1c102ef0 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d221b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c100e20 0x7f9d1c1a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d229b3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d1c102ef0 0x7f9d1c10cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d221b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c100e20 0x7f9d1c1a47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:46754/0 (socket says 172.21.15.28:46754) 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d221b2640 1 -- 172.21.15.28:0/2316473454 learned_addr learned my addr 172.21.15.28:0/2316473454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d229b3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d1c102ef0 0x7f9d1c10cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:37408/0 (socket says 172.21.15.28:37408) 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.569+0000 7f9d219b1640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102570 0x7f9d1c1a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.570+0000 7f9d219b1640 1 -- 172.21.15.28:0/2316473454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d1c102ef0 msgr2=0x7f9d1c10cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.570+0000 7f9d219b1640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d1c102ef0 0x7f9d1c10cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.570+0000 7f9d219b1640 1 -- 172.21.15.28:0/2316473454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c100e20 msgr2=0x7f9d1c1a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.570+0000 7f9d219b1640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c100e20 0x7f9d1c1a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.570+0000 7f9d219b1640 1 -- 172.21.15.28:0/2316473454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c10d3a0 con 0x7f9d1c102570 2024-09-15T21:58:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.570+0000 7f9d219b1640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102570 0x7f9d1c1a4cf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00e9f0 tx=0x7f9d0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.571+0000 7f9d137fe640 1 -- 172.21.15.28:0/2316473454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00ccc0 con 0x7f9d1c102570 2024-09-15T21:58:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.571+0000 7f9d137fe640 1 -- 172.21.15.28:0/2316473454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c010910 con 0x7f9d1c102570 2024-09-15T21:58:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.571+0000 7f9d137fe640 1 -- 172.21.15.28:0/2316473454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0188e0 con 0x7f9d1c102570 2024-09-15T21:58:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.571+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1c10d690 con 0x7f9d1c102570 2024-09-15T21:58:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.571+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1c1ab5a0 con 0x7f9d1c102570 2024-09-15T21:58:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.574+0000 7f9d267ef740 1 Processor -- start 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.574+0000 7f9d267ef740 1 -- start start 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9293ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92945400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd92945940 0x55bd92947d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d267ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bd928a02a0 con 0x55bd92945940 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d267ef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bd92847b10 con 0x55bd9293a760 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d267ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bd9289c260 con 0x55bd9293b080 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d137fe640 1 -- 172.21.15.28:0/2316473454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f9d0c010ac0 con 0x7f9d1c102570 2024-09-15T21:58:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d221b2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92945400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d221b2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92945400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:37424/0 (socket says 172.21.15.28:37424) 2024-09-15T21:58:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d221b2640 1 -- 172.21.15.28:0/182789591 learned_addr learned my addr 172.21.15.28:0/182789591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.575+0000 7f9d219b1640 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd92945940 0x55bd92947d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d229b3640 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9293ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d221b2640 1 -- 172.21.15.28:0/182789591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 msgr2=0x55bd9293ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d221b2640 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9293ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d221b2640 1 -- 172.21.15.28:0/182789591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd92945940 msgr2=0x55bd92947d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d221b2640 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd92945940 0x55bd92947d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d221b2640 1 -- 172.21.15.28:0/182789591 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd92611b90 con 0x55bd9293b080 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d219b1640 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd92945940 0x55bd92947d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d221b2640 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92945400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800bda0 tx=0x7f9d1800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=720e4c944953ddaa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d137fe640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d080a35e0 0x7f9d080a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d10dea640 1 -- 172.21.15.28:0/182789591 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18015070 con 0x55bd9293b080 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d229b3640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d080a35e0 0x7f9d080a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.576+0000 7f9d137fe640 1 -- 172.21.15.28:0/2316473454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9d0c0d0800 con 0x7f9d1c102570 2024-09-15T21:58:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.577+0000 7f9d10dea640 1 -- 172.21.15.28:0/182789591 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d1801a940 con 0x55bd9293b080 2024-09-15T21:58:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.577+0000 7f9d10dea640 1 -- 172.21.15.28:0/182789591 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18011a80 con 0x55bd9293b080 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 -- 172.21.15.28:0/182789591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 msgr2=0x55bd92945400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92945400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800bda0 tx=0x7f9d1800f730 comp rx=0 tx=0).stop 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d229b3640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d080a35e0 0x7f9d080a5aa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9d14005e30 tx=0x7f9d14005c50 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 -- 172.21.15.28:0/182789591 shutdown_connections 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd92945940 0x55bd92947d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92945400 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 --2- 172.21.15.28:0/182789591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9293ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 -- 172.21.15.28:0/182789591 >> 172.21.15.28:0/182789591 conn(0x55bd92936cf0 msgr2=0x55bd929370d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 -- 172.21.15.28:0/182789591 shutdown_connections 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 -- 172.21.15.28:0/182789591 wait complete. 2024-09-15T21:58:13.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.578+0000 7f9d267ef740 1 -- start start 2024-09-15T21:58:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 1 Processor -- start 2024-09-15T21:58:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293a760 0x55bd929427b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92942cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd92943230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bd92948560 con 0x55bd9293a760 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bd928286e0 con 0x55bd92945940 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bd929486e0 con 0x55bd9293b080 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d267ef740 10 client.0 _ll_register_callbacks cb 0x55bd9284ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d221b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293a760 0x55bd929427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d221b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293a760 0x55bd929427b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55602/0 (socket says 172.21.15.28:55602) 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d221b2640 1 -- 172.21.15.28:0/98457883 learned_addr learned my addr 172.21.15.28:0/98457883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.579+0000 7f9d219b1640 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92942cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d229b3640 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd92943230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d221b2640 1 -- 172.21.15.28:0/98457883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 msgr2=0x55bd92942cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d221b2640 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92942cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d221b2640 1 -- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 msgr2=0x55bd92943230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d221b2640 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd92943230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d221b2640 1 -- 172.21.15.28:0/98457883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd929519d0 con 0x55bd9293a760 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d221b2640 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293a760 0x55bd929427b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006010 tx=0x7f9d1801a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e7a8522e1b605a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d05ffb640 1 -- 172.21.15.28:0/98457883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18015070 con 0x55bd9293a760 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d05ffb640 1 -- 172.21.15.28:0/98457883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9d1800fde0 con 0x55bd9293a760 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.580+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d1801a940 con 0x55bd9293a760 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.581+0000 7f9d05ffb640 1 -- 172.21.15.28:0/98457883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d1802c050 con 0x55bd9293a760 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.581+0000 7f9d267ef740 10 client.15810 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.581+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bd92954340 con 0x55bd9293a760 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.581+0000 7f9d05ffb640 1 -- 172.21.15.28:0/98457883 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d1801a940 con 0x55bd9293a760 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.581+0000 7f9d05ffb640 1 -- 172.21.15.28:0/98457883 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9d18011b30 con 0x55bd9293a760 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.581+0000 7f9d267ef740 10 client.15810 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.581+0000 7f9d267ef740 10 client.15810 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.582+0000 7f9d267ef740 20 client.15810 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.582+0000 7f9d267ef740 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd92955750 0x55bd92957b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.582+0000 7f9d267ef740 4 client.15810 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.582+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd928ad9d0 con 0x55bd92955750 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.582+0000 7f9d219b1640 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd92955750 0x55bd92957b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.582+0000 7f9d05ffb640 10 client.15810 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.582+0000 7f9d219b1640 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd92955750 0x55bd92957b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8296ade45a8c4166 server_cookie=4bd9eae6f033702d in_seq=0 out_seq=0 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.583+0000 7f9d05ffb640 1 -- 172.21.15.28:0/98457883 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bd928ad9d0 con 0x55bd92955750 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.583+0000 7f9d05ffb640 10 client.15810 handle_command_reply: tid=0 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.583+0000 7f9d267ef740 1 client.15810 shutdown 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293a760 msgr2=0x55bd929427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293a760 0x55bd929427b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006010 tx=0x7f9d1801a690 comp rx=0 tx=0).stop 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 shutdown_connections 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd92955750 0x55bd92957b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd92943230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293b080 0x55bd92942cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 --2- 172.21.15.28:0/98457883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293a760 0x55bd929427b0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 >> 172.21.15.28:0/98457883 conn(0x55bd92936cf0 msgr2=0x55bd92935e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d05ffb640 0 client.15810 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 shutdown_connections 2024-09-15T21:58:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.584+0000 7f9d267ef740 1 -- 172.21.15.28:0/98457883 wait complete. 2024-09-15T21:58:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.585+0000 7f9d267ef740 20 client.15810 trim_cache size 0 max 16384 2024-09-15T21:58:13.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.604+0000 7f9d267ef740 1 Processor -- start 2024-09-15T21:58:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.604+0000 7f9d267ef740 1 -- start start 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.604+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293a760 0x55bd92944680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.604+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd9293d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd9295d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d267ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b5520 con 0x55bd9293b080 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d267ef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b7060 con 0x55bd92945940 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d267ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b5bf0 con 0x55bd9293a760 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd9293d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd9293d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55608/0 (socket says 172.21.15.28:55608) 2024-09-15T21:58:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 -- 172.21.15.28:0/4070867161 learned_addr learned my addr 172.21.15.28:0/4070867161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d229b3640 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd9295d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 -- 172.21.15.28:0/4070867161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293a760 msgr2=0x55bd92944680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d221b2640 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293a760 0x55bd92944680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293a760 0x55bd92944680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 -- 172.21.15.28:0/4070867161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 msgr2=0x55bd9295d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd9295d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 -- 172.21.15.28:0/4070867161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd92932260 con 0x55bd9293b080 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d221b2640 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293a760 0x55bd92944680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d219b1640 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd9293d1c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00c6a0 tx=0x7f9d0c0d6530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5a17c96239573b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.605+0000 7f9d229b3640 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd9295d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.606+0000 7f9d07fff640 1 -- 172.21.15.28:0/4070867161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c09bea0 con 0x55bd9293b080 2024-09-15T21:58:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.607+0000 7f9d07fff640 1 -- 172.21.15.28:0/4070867161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c0bc040 con 0x55bd9293b080 2024-09-15T21:58:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.607+0000 7f9d07fff640 1 -- 172.21.15.28:0/4070867161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c098bb0 con 0x55bd9293b080 2024-09-15T21:58:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.607+0000 7f9d267ef740 1 -- 172.21.15.28:0/4070867161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 msgr2=0x55bd9293d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.607+0000 7f9d267ef740 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd9293d1c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00c6a0 tx=0x7f9d0c0d6530 comp rx=0 tx=0).stop 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 -- 172.21.15.28:0/4070867161 shutdown_connections 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd92945940 0x55bd9295d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd9293d1c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 --2- 172.21.15.28:0/4070867161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd9293a760 0x55bd92944680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 -- 172.21.15.28:0/4070867161 >> 172.21.15.28:0/4070867161 conn(0x55bd92936cf0 msgr2=0x55bd92941770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 -- 172.21.15.28:0/4070867161 shutdown_connections 2024-09-15T21:58:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 -- 172.21.15.28:0/4070867161 wait complete. 2024-09-15T21:58:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.608+0000 7f9d267ef740 1 -- start start 2024-09-15T21:58:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d267ef740 1 Processor -- start 2024-09-15T21:58:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9294e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd92949450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d267ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd92945940 0x55bd929499d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d219b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd92949450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d267ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b6990 con 0x55bd9293b080 2024-09-15T21:58:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d219b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd92949450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55612/0 (socket says 172.21.15.28:55612) 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d219b1640 1 -- 172.21.15.28:0/3245595007 learned_addr learned my addr 172.21.15.28:0/3245595007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.609+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b62c0 con 0x55bd9293a760 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b3500 con 0x55bd92945940 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d221b2640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9294e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d229b3640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd92945940 0x55bd929499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d267ef740 10 client.0 _ll_register_callbacks cb 0x7f9d1800bc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d221b2640 1 -- 172.21.15.28:0/3245595007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd92945940 msgr2=0x55bd929499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d221b2640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd92945940 0x55bd929499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d221b2640 1 -- 172.21.15.28:0/3245595007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 msgr2=0x55bd92949450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d221b2640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd92949450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d221b2640 1 -- 172.21.15.28:0/3245595007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9294a200 con 0x55bd9293a760 2024-09-15T21:58:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d0c0bc040 con 0x55bd9293a760 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d219b1640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd92949450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d229b3640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd92945940 0x55bd929499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d221b2640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9294e120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55bd9293e280 tx=0x7f9d1800ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19ea332755e05c27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d05ffb640 10 client.25427 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.610+0000 7f9d05ffb640 1 -- 172.21.15.28:0/3245595007 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18015070 con 0x55bd9293a760 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d05ffb640 1 -- 172.21.15.28:0/3245595007 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9d18020040 con 0x55bd9293a760 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d05ffb640 1 -- 172.21.15.28:0/3245595007 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d18004790 con 0x55bd9293a760 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 10 client.25427 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55bd929c0060 con 0x55bd9293a760 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d05ffb640 1 -- 172.21.15.28:0/3245595007 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9d180115c0 con 0x55bd9293a760 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 10 client.25427 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 10 client.25427 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 20 client.25427 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd929c0630 0x55bd929c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 4 client.25427 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}] 2024-09-15T21:58:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.611+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}) -- 0x55bd928a7d20 con 0x55bd929c0630 2024-09-15T21:58:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.612+0000 7f9d219b1640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd929c0630 0x55bd929c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.612+0000 7f9d05ffb640 10 client.25427 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.612+0000 7f9d219b1640 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd929c0630 0x55bd929c2a20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7534b47059937a7d server_cookie=3b6bf3a9eae52d64 in_seq=0 out_seq=0 2024-09-15T21:58:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.629+0000 7f9d05ffb640 1 -- 172.21.15.28:0/3245595007 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55bd928a7d20 con 0x55bd929c0630 2024-09-15T21:58:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.629+0000 7f9d05ffb640 10 client.25427 handle_command_reply: tid=0 2024-09-15T21:58:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 client.25427 shutdown 2024-09-15T21:58:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 msgr2=0x55bd9294e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9294e120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55bd9293e280 tx=0x7f9d1800ba00 comp rx=0 tx=0).stop 2024-09-15T21:58:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 shutdown_connections 2024-09-15T21:58:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bd929c0630 0x55bd929c2a20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd92945940 0x55bd929499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d05ffb640 0 client.25427 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd9293b080 0x55bd92949450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 --2- 172.21.15.28:0/3245595007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9293a760 0x55bd9294e120 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 >> 172.21.15.28:0/3245595007 conn(0x55bd92936cf0 msgr2=0x55bd92940f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 shutdown_connections 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.630+0000 7f9d267ef740 1 -- 172.21.15.28:0/3245595007 wait complete. 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.631+0000 7f9d267ef740 20 client.25427 trim_cache size 0 max 16384 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 29, 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T21:58:13.633 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 2, 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "b0814eb3": { 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 2, 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 90.0, 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 150.2, 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-09-15T21:58:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:13.635 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.631+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d080a35e0 msgr2=0x7f9d080a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.631+0000 7f9d24416640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d080a35e0 0x7f9d080a5aa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9d14005e30 tx=0x7f9d14005c50 comp rx=0 tx=0).stop 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.631+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102570 msgr2=0x7f9d1c1a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.631+0000 7f9d24416640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102570 0x7f9d1c1a4cf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00e9f0 tx=0x7f9d0c00eeb0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 shutdown_connections 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d080a35e0 0x7f9d080a5aa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d1c102ef0 0x7f9d1c10cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c102570 0x7f9d1c1a4cf0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 --2- 172.21.15.28:0/2316473454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c100e20 0x7f9d1c1a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 >> 172.21.15.28:0/2316473454 conn(0x7f9d1c0fc7d0 msgr2=0x7f9d1c106d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 shutdown_connections 2024-09-15T21:58:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:13.632+0000 7f9d24416640 1 -- 172.21.15.28:0/2316473454 wait complete. 2024-09-15T21:58:13.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:13 smithi203 ceph-mon[35546]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 30 op/s 2024-09-15T21:58:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:13 smithi174 ceph-mon[35244]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 30 op/s 2024-09-15T21:58:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.359+0000 7f2c864d9640 1 -- 172.21.15.28:0/3497767818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80101de0 msgr2=0x7f2c801021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.359+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3497767818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80101de0 0x7f2c801021c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2c70009900 tx=0x7f2c7002a410 comp rx=0 tx=0).stop 2024-09-15T21:58:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.359+0000 7f2c864d9640 1 -- 172.21.15.28:0/3497767818 shutdown_connections 2024-09-15T21:58:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.359+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3497767818 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c80106f10 0x7f2c8010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.359+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3497767818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c80102790 0x7f2c801067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.359+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3497767818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80101de0 0x7f2c801021c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.360+0000 7f2c864d9640 1 -- 172.21.15.28:0/3497767818 >> 172.21.15.28:0/3497767818 conn(0x7f2c800fd8c0 msgr2=0x7f2c800ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.360+0000 7f2c864d9640 1 -- 172.21.15.28:0/3497767818 shutdown_connections 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.360+0000 7f2c864d9640 1 -- 172.21.15.28:0/3497767818 wait complete. 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.360+0000 7f2c864d9640 1 Processor -- start 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.360+0000 7f2c864d9640 1 -- start start 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c864d9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c80101de0 0x7f2c80198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c864d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c80102790 0x7f2c80198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c864d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80106f10 0x7f2c8019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c864d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80111f70 con 0x7f2c80106f10 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c864d9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80111df0 con 0x7f2c80101de0 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c864d9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801120f0 con 0x7f2c80102790 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80106f10 0x7f2c8019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80106f10 0x7f2c8019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55630/0 (socket says 172.21.15.28:55630) 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 -- 172.21.15.28:0/3756439633 learned_addr learned my addr 172.21.15.28:0/3756439633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c80102790 0x7f2c80198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c80101de0 0x7f2c80198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 -- 172.21.15.28:0/3756439633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c80102790 msgr2=0x7f2c80198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c80102790 0x7f2c80198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 -- 172.21.15.28:0/3756439633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c80101de0 msgr2=0x7f2c80198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c80101de0 0x7f2c80198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.361+0000 7f2c84a76640 1 -- 172.21.15.28:0/3756439633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8019d3e0 con 0x7f2c80106f10 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.362+0000 7f2c84a76640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80106f10 0x7f2c8019cd00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800eaa0 tx=0x7f2c7800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.362+0000 7f2c7d7fa640 1 -- 172.21.15.28:0/3756439633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7800cdb0 con 0x7f2c80106f10 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.362+0000 7f2c7d7fa640 1 -- 172.21.15.28:0/3756439633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c78005020 con 0x7f2c80106f10 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.362+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8019d6d0 con 0x7f2c80106f10 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.362+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c801a4fb0 con 0x7f2c80106f10 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.363+0000 7f2c7d7fa640 1 -- 172.21.15.28:0/3756439633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c780116b0 con 0x7f2c80106f10 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.365+0000 7f2c888b2740 1 Processor -- start 2024-09-15T21:58:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.365+0000 7f2c7d7fa640 1 -- 172.21.15.28:0/3756439633 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f2c78005560 con 0x7f2c80106f10 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c888b2740 1 -- start start 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592b6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b65d0 0x558a592b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592c3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c888b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a5921c2f0 con 0x558a592b7b40 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c888b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558a591c3cf0 con 0x558a592c1370 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c888b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558a592182b0 con 0x558a592b65d0 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c7ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b65d0 0x558a592b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c7ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b65d0 0x558a592b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:37486/0 (socket says 172.21.15.28:37486) 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c7ffff640 1 -- 172.21.15.28:0/2481787816 learned_addr learned my addr 172.21.15.28:0/2481787816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.366+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7d7fa640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2c680a35e0 0x7f2c680a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7ffff640 1 -- 172.21.15.28:0/2481787816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 msgr2=0x558a592c3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592c3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7ffff640 1 -- 172.21.15.28:0/2481787816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 msgr2=0x558a592b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c84a76640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2c680a35e0 0x7f2c680a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7d7fa640 1 -- 172.21.15.28:0/3756439633 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2c780d04b0 con 0x7f2c80106f10 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592b6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7ffff640 1 -- 172.21.15.28:0/2481787816 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a591a83a0 con 0x558a592b65d0 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b65d0 0x558a592b6a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c70009900 tx=0x7f2c70005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd649fd3dce31274 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c84a76640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2c680a35e0 0x7f2c680a5aa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2c780cd6c0 tx=0x7f2c7806aee0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.367+0000 7f2c667fc640 1 -- 172.21.15.28:0/2481787816 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7003f070 con 0x558a592b65d0 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c667fc640 1 -- 172.21.15.28:0/2481787816 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c70004030 con 0x558a592b65d0 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c667fc640 1 -- 172.21.15.28:0/2481787816 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c70035070 con 0x558a592b65d0 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c888b2740 1 -- 172.21.15.28:0/2481787816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b65d0 msgr2=0x558a592b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b65d0 0x558a592b6a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c70009900 tx=0x7f2c70005dd0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c888b2740 1 -- 172.21.15.28:0/2481787816 shutdown_connections 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592c3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b65d0 0x558a592b6a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2481787816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.368+0000 7f2c888b2740 1 -- 172.21.15.28:0/2481787816 >> 172.21.15.28:0/2481787816 conn(0x558a5927f310 msgr2=0x558a592b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.369+0000 7f2c888b2740 1 -- 172.21.15.28:0/2481787816 shutdown_connections 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.369+0000 7f2c888b2740 1 -- 172.21.15.28:0/2481787816 wait complete. 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.369+0000 7f2c888b2740 1 -- start start 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.369+0000 7f2c888b2740 1 Processor -- start 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592c0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592c0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592c0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:46826/0 (socket says 172.21.15.28:46826) 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592bdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55642/0 (socket says 172.21.15.28:55642) 2024-09-15T21:58:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/1754874665 learned_addr learned my addr 172.21.15.28:0/1754874665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c888b2740 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592c1370 0x558a592be570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a592c4010 con 0x558a592b7b40 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558a591d7060 con 0x558a592b65d0 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558a592c4190 con 0x558a592c1370 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592c1370 msgr2=0x558a592be570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592c1370 0x558a592be570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/1754874665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 msgr2=0x558a592bdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592bdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/1754874665 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a58f8d3c0 con 0x558a592b65d0 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.370+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592bdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592c0d20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009bc0 tx=0x7f2c74009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f208b17869799a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c5f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c888b2740 10 client.0 _ll_register_callbacks cb 0x558a58f2a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c70004030 con 0x558a592b65d0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/1754874665 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400fde0 con 0x558a592b65d0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/1754874665 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2c7401f440 con 0x558a592b65d0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/1754874665 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2c7401f680 con 0x558a592b65d0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c888b2740 10 client.25439 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.371+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x558a592cfa30 con 0x558a592b65d0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/1754874665 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2c7401d070 con 0x558a592b65d0 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c888b2740 10 client.25439 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c888b2740 10 client.25439 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c888b2740 20 client.25439 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c888b2740 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a592d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c888b2740 4 client.25439 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a59229860 con 0x558a592d0f40 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.372+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a592d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.373+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a592d33d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f5027e0599390b8b server_cookie=8a68be621bb587d8 in_seq=0 out_seq=0 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.373+0000 7f2c5f7fe640 10 client.25439 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.373+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/1754874665 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a59229860 con 0x558a592d0f40 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.373+0000 7f2c5f7fe640 10 client.25439 handle_command_reply: tid=0 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.373+0000 7f2c888b2740 1 client.25439 shutdown 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 msgr2=0x558a592c0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592c0d20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009bc0 tx=0x7f2c74009f10 comp rx=0 tx=0).stop 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 shutdown_connections 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a592d33d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c5f7fe640 0 client.25439 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592c1370 0x558a592be570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b7b40 0x558a592bdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 --2- 172.21.15.28:0/1754874665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592c0d20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 >> 172.21.15.28:0/1754874665 conn(0x558a5927f310 msgr2=0x558a592b8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 shutdown_connections 2024-09-15T21:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 1 -- 172.21.15.28:0/1754874665 wait complete. 2024-09-15T21:58:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.374+0000 7f2c888b2740 20 client.25439 trim_cache size 0 max 16384 2024-09-15T21:58:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.399+0000 7f2c888b2740 1 Processor -- start 2024-09-15T21:58:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.399+0000 7f2c888b2740 1 -- start start 2024-09-15T21:58:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592c1370 0x558a592c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c7f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:46828/0 (socket says 172.21.15.28:46828) 2024-09-15T21:58:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/842694131 learned_addr learned my addr 172.21.15.28:0/842694131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801aee70 con 0x558a592c1370 2024-09-15T21:58:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b09b0 con 0x558a592b65d0 2024-09-15T21:58:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801af540 con 0x558a592b7b40 2024-09-15T21:58:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.400+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/842694131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 msgr2=0x558a592c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/842694131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592c1370 msgr2=0x558a592c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592c1370 0x558a592c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/842694131 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a592c3ba0 con 0x558a592b65d0 2024-09-15T21:58:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592cd2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009690 tx=0x7f2c74012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dda73aeea585ddc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c65ffb640 1 -- 172.21.15.28:0/842694131 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c74034070 con 0x558a592b65d0 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c65ffb640 1 -- 172.21.15.28:0/842694131 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c70004030 con 0x558a592b65d0 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.401+0000 7f2c65ffb640 1 -- 172.21.15.28:0/842694131 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c74029700 con 0x558a592b65d0 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 msgr2=0x558a592cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592cd2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009690 tx=0x7f2c74012040 comp rx=0 tx=0).stop 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 shutdown_connections 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592c1370 0x558a592c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 --2- 172.21.15.28:0/842694131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592b65d0 0x558a592cd2e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 >> 172.21.15.28:0/842694131 conn(0x558a5927f310 msgr2=0x558a592bd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 shutdown_connections 2024-09-15T21:58:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.402+0000 7f2c888b2740 1 -- 172.21.15.28:0/842694131 wait complete. 2024-09-15T21:58:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.403+0000 7f2c888b2740 1 -- start start 2024-09-15T21:58:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.403+0000 7f2c888b2740 1 Processor -- start 2024-09-15T21:58:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.403+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b65d0 0x558a592c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.403+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.403+0000 7f2c888b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592bac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b65d0 0x558a592c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b65d0 0x558a592c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55670/0 (socket says 172.21.15.28:55670) 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/2360734301 learned_addr learned my addr 172.21.15.28:0/2360734301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b02e0 con 0x558a592b65d0 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801afc10 con 0x558a592c1370 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801ace50 con 0x558a592b7b40 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c888b2740 10 client.0 _ll_register_callbacks cb 0x7f2c74009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592bac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 msgr2=0x558a592c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/2360734301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 msgr2=0x558a592bac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592bac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 -- 172.21.15.28:0/2360734301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a592c1090 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c7f7fe640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b65d0 0x558a592c97f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c74010bd0 tx=0x7f2c740059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44477ce3d193d0e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.404+0000 7f2c5f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.405+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/2360734301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c74034040 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.405+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/2360734301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2c74005310 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.405+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c70004030 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.405+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/2360734301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2c7403d050 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.405+0000 7f2c888b2740 10 client.15831 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.405+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a592c7f40 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.405+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/2360734301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c70004030 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/2360734301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2c7401d070 con 0x558a592b65d0 2024-09-15T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c888b2740 10 client.15831 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c888b2740 10 client.15831 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c888b2740 20 client.15831 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a59337dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c888b2740 4 client.15831 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a59223d70 con 0x558a592d0f40 2024-09-15T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.406+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a59337dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c5f7fe640 10 client.15831 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c7ffff640 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a59337dd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=272f810ea2c6652f server_cookie=85d94179dd506e75 in_seq=0 out_seq=0 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c5f7fe640 1 -- 172.21.15.28:0/2360734301 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a59223d70 con 0x558a592d0f40 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c5f7fe640 10 client.15831 handle_command_reply: tid=0 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 client.15831 shutdown 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b65d0 msgr2=0x558a592c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b65d0 0x558a592c97f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c74010bd0 tx=0x7f2c740059b0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 shutdown_connections 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x558a592d0f40 0x558a59337dd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c5f7fe640 0 client.15831 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a592c1370 0x558a592bac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558a592b7b40 0x558a592c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 --2- 172.21.15.28:0/2360734301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a592b65d0 0x558a592c97f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 >> 172.21.15.28:0/2360734301 conn(0x558a5927f310 msgr2=0x558a592b8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 shutdown_connections 2024-09-15T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.407+0000 7f2c888b2740 1 -- 172.21.15.28:0/2360734301 wait complete. 2024-09-15T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c888b2740 20 client.15831 trim_cache size 0 max 16384 2024-09-15T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2c680a35e0 msgr2=0x7f2c680a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2c680a35e0 0x7f2c680a5aa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2c780cd6c0 tx=0x7f2c7806aee0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80106f10 msgr2=0x7f2c8019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80106f10 0x7f2c8019cd00 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800eaa0 tx=0x7f2c7800ef60 comp rx=0 tx=0).stop 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 shutdown_connections 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2c680a35e0 0x7f2c680a5aa0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80106f10 0x7f2c8019cd00 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c80102790 0x7f2c80198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 --2- 172.21.15.28:0/3756439633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c80101de0 0x7f2c80198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.408+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 >> 172.21.15.28:0/3756439633 conn(0x7f2c800fd8c0 msgr2=0x7f2c800fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.409+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 shutdown_connections 2024-09-15T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:14.409+0000 7f2c864d9640 1 -- 172.21.15.28:0/3756439633 wait complete. 2024-09-15T21:58:14.870 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b0814eb3', quiesce took 7.4 seconds. Will release after: 22.799999999999997 2024-09-15T21:58:15.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:15 smithi028 ceph-mon[30969]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-15T21:58:15.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T21:58:15.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:15 smithi203 ceph-mon[35546]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-15T21:58:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:15 smithi174 ceph-mon[35244]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-15T21:58:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.397+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/2110554682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54102dd0 msgr2=0x7fcf541031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.397+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/2110554682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54102dd0 0x7fcf541031b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009930 tx=0x7fcf5002a420 comp rx=0 tx=0).stop 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/2110554682 shutdown_connections 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/2110554682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf54108300 0x7fcf5410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/2110554682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf54103780 0x7fcf54107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/2110554682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54102dd0 0x7fcf541031b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/2110554682 >> 172.21.15.28:0/2110554682 conn(0x7fcf540fc7d0 msgr2=0x7fcf540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/2110554682 shutdown_connections 2024-09-15T21:58:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/2110554682 wait complete. 2024-09-15T21:58:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.398+0000 7fcf5b8b8640 1 Processor -- start 2024-09-15T21:58:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf5b8b8640 1 -- start start 2024-09-15T21:58:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf5b8b8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf54102dd0 0x7fcf54100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf5b8b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54103780 0x7fcf54100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf5b8b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf54108300 0x7fcf541014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf58e53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54103780 0x7fcf54100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf58e53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54103780 0x7fcf54100f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55698/0 (socket says 172.21.15.28:55698) 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf58e53640 1 -- 172.21.15.28:0/4132556082 learned_addr learned my addr 172.21.15.28:0/4132556082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf54112090 con 0x7fcf54103780 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf54111f10 con 0x7fcf54102dd0 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf54112210 con 0x7fcf54108300 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf59654640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf54102dd0 0x7fcf54100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf59e55640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf54108300 0x7fcf541014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf58e53640 1 -- 172.21.15.28:0/4132556082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf54108300 msgr2=0x7fcf541014b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.399+0000 7fcf58e53640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf54108300 0x7fcf541014b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf58e53640 1 -- 172.21.15.28:0/4132556082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf54102dd0 msgr2=0x7fcf54100a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf58e53640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf54102dd0 0x7fcf54100a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf58e53640 1 -- 172.21.15.28:0/4132556082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf541a6b10 con 0x7fcf54103780 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf59654640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf54102dd0 0x7fcf54100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf58e53640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54103780 0x7fcf54100f70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400d740 tx=0x7fcf4400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf4a7fc640 1 -- 172.21.15.28:0/4132556082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44017070 con 0x7fcf54103780 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf541a6e00 con 0x7fcf54103780 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf541a7300 con 0x7fcf54103780 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf4a7fc640 1 -- 172.21.15.28:0/4132556082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4400de50 con 0x7fcf54103780 2024-09-15T21:58:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.400+0000 7fcf4a7fc640 1 -- 172.21.15.28:0/4132556082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400f6b0 con 0x7fcf54103780 2024-09-15T21:58:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 Processor -- start 2024-09-15T21:58:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 -- start start 2024-09-15T21:58:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de81b40 0x559e8de80090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de80a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de8d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e8dde62f0 con 0x559e8de805d0 2024-09-15T21:58:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559e8dd8dcf0 con 0x559e8de8b370 2024-09-15T21:58:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559e8dde22b0 con 0x559e8de81b40 2024-09-15T21:58:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf4a7fc640 1 -- 172.21.15.28:0/4132556082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7fcf4401a070 con 0x7fcf54103780 2024-09-15T21:58:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.404+0000 7fcf59e55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de81b40 0x559e8de80090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf58e53640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de8d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59e55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de81b40 0x559e8de80090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:37554/0 (socket says 172.21.15.28:37554) 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59e55640 1 -- 172.21.15.28:0/3187797887 learned_addr learned my addr 172.21.15.28:0/3187797887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59654640 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de80a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59e55640 1 -- 172.21.15.28:0/3187797887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 msgr2=0x559e8de8d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59e55640 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de8d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59e55640 1 -- 172.21.15.28:0/3187797887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 msgr2=0x559e8de80a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59e55640 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de80a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf59e55640 1 -- 172.21.15.28:0/3187797887 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8dd723a0 con 0x559e8de81b40 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.405+0000 7fcf4a7fc640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcf400a3560 0x7fcf400a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.406+0000 7fcf59e55640 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de81b40 0x559e8de80090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009f20 tx=0x7fcf4c00f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca94ade5a3fa1520 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.406+0000 7fcf4a7fc640 1 -- 172.21.15.28:0/4132556082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fcf440cfde0 con 0x7fcf54103780 2024-09-15T21:58:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.406+0000 7fcf3f7fe640 1 -- 172.21.15.28:0/3187797887 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c015070 con 0x559e8de81b40 2024-09-15T21:58:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.406+0000 7fcf3f7fe640 1 -- 172.21.15.28:0/3187797887 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4c01ab40 con 0x559e8de81b40 2024-09-15T21:58:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.406+0000 7fcf59654640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcf400a3560 0x7fcf400a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.406+0000 7fcf3f7fe640 1 -- 172.21.15.28:0/3187797887 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c011ca0 con 0x559e8de81b40 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 -- 172.21.15.28:0/3187797887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de81b40 msgr2=0x559e8de80090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de81b40 0x559e8de80090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009f20 tx=0x7fcf4c00f7f0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf59654640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcf400a3560 0x7fcf400a5a20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf50000c00 tx=0x7fcf50005b50 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 -- 172.21.15.28:0/3187797887 shutdown_connections 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de8d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de80a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/3187797887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de81b40 0x559e8de80090 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 -- 172.21.15.28:0/3187797887 >> 172.21.15.28:0/3187797887 conn(0x559e8de49310 msgr2=0x559e8de7cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 -- 172.21.15.28:0/3187797887 shutdown_connections 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.407+0000 7fcf5dc91740 1 -- 172.21.15.28:0/3187797887 wait complete. 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 -- start start 2024-09-15T21:58:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 Processor -- start 2024-09-15T21:58:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de805d0 0x559e8de8ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de88090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8de887d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e8de8e070 con 0x559e8de81b40 2024-09-15T21:58:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559e8dda1060 con 0x559e8de805d0 2024-09-15T21:58:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559e8de8e1f0 con 0x559e8de8b370 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.408+0000 7fcf5dc91740 10 client.0 _ll_register_callbacks cb 0x559e8daf48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59654640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de805d0 0x559e8de8ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59654640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de805d0 0x559e8de8ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:46868/0 (socket says 172.21.15.28:46868) 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59654640 1 -- 172.21.15.28:0/2502250727 learned_addr learned my addr 172.21.15.28:0/2502250727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf58e53640 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de88090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59e55640 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8de887d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59e55640 1 -- 172.21.15.28:0/2502250727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de805d0 msgr2=0x559e8de8ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59e55640 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de805d0 0x559e8de8ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59e55640 1 -- 172.21.15.28:0/2502250727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 msgr2=0x559e8de88090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59e55640 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de88090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59e55640 1 -- 172.21.15.28:0/2502250727 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e8db573c0 con 0x559e8de8b370 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf59e55640 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8de887d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0092e0 tx=0x7fcf4c009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c61bef730f9e6bf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.409+0000 7fcf3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf3cff9640 1 -- 172.21.15.28:0/2502250727 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c015070 con 0x559e8de8b370 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf4c01ab40 con 0x559e8de8b370 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf3cff9640 1 -- 172.21.15.28:0/2502250727 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fcf4c004c50 con 0x559e8de8b370 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf3cff9640 1 -- 172.21.15.28:0/2502250727 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf4c004e50 con 0x559e8de8b370 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf5dc91740 10 client.25414 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e8de99ba0 con 0x559e8de8b370 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf3cff9640 1 -- 172.21.15.28:0/2502250727 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4c01ab40 con 0x559e8de8b370 2024-09-15T21:58:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.410+0000 7fcf3cff9640 1 -- 172.21.15.28:0/2502250727 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf4c005180 con 0x559e8de8b370 2024-09-15T21:58:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf5dc91740 10 client.25414 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf5dc91740 10 client.25414 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T21:58:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf5dc91740 20 client.25414 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8de9d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf5dc91740 4 client.25414 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e8ddf3860 con 0x559e8de9aef0 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf59654640 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8de9d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf59654640 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8de9d2e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89ae00f056c96e24 server_cookie=2658108925eb8d3b in_seq=0 out_seq=0 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.411+0000 7fcf3cff9640 10 client.25414 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf3cff9640 1 -- 172.21.15.28:0/2502250727 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e8ddf3860 con 0x559e8de9aef0 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf3cff9640 10 client.25414 handle_command_reply: tid=0 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 client.25414 shutdown 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 msgr2=0x559e8de887d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8de887d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0092e0 tx=0x7fcf4c009c90 comp rx=0 tx=0).stop 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 shutdown_connections 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8de9d2e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf3cff9640 0 client.25414 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:58:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8de887d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de88090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/2502250727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de805d0 0x559e8de8ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.412+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 >> 172.21.15.28:0/2502250727 conn(0x559e8de49310 msgr2=0x559e8de82960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.413+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 shutdown_connections 2024-09-15T21:58:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.413+0000 7fcf5dc91740 1 -- 172.21.15.28:0/2502250727 wait complete. 2024-09-15T21:58:16.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.413+0000 7fcf5dc91740 20 client.25414 trim_cache size 0 max 16384 2024-09-15T21:58:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 Processor -- start 2024-09-15T21:58:16.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 -- start start 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de97490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de81b40 0x559e8dea0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8dea0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf541b12d0 con 0x559e8de805d0 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf541b2e10 con 0x559e8de81b40 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf541b19a0 con 0x559e8de8b370 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf59654640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de97490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf59654640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de97490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55720/0 (socket says 172.21.15.28:55720) 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.426+0000 7fcf59654640 1 -- 172.21.15.28:0/1403783721 learned_addr learned my addr 172.21.15.28:0/1403783721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf58e53640 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de81b40 0x559e8dea0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59e55640 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8dea0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59654640 1 -- 172.21.15.28:0/1403783721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 msgr2=0x559e8dea0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59654640 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8dea0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59654640 1 -- 172.21.15.28:0/1403783721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de81b40 msgr2=0x559e8dea0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59654640 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de81b40 0x559e8dea0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59654640 1 -- 172.21.15.28:0/1403783721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8de8e110 con 0x559e8de805d0 2024-09-15T21:58:16.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59e55640 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8dea0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:16.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf59654640 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de97490 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009230 tx=0x7fcf5003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29a88e6d4bcde3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:16.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf3effd640 1 -- 172.21.15.28:0/1403783721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50056070 con 0x559e8de805d0 2024-09-15T21:58:16.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf3effd640 1 -- 172.21.15.28:0/1403783721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf50003d90 con 0x559e8de805d0 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.427+0000 7fcf3effd640 1 -- 172.21.15.28:0/1403783721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50035040 con 0x559e8de805d0 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 -- 172.21.15.28:0/1403783721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 msgr2=0x559e8de97490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de97490 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009230 tx=0x7fcf5003e000 comp rx=0 tx=0).stop 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 -- 172.21.15.28:0/1403783721 shutdown_connections 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de8b370 0x559e8dea0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de81b40 0x559e8dea0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/1403783721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de805d0 0x559e8de97490 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 -- 172.21.15.28:0/1403783721 >> 172.21.15.28:0/1403783721 conn(0x559e8de49310 msgr2=0x559e8de8a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.428+0000 7fcf5dc91740 1 -- 172.21.15.28:0/1403783721 shutdown_connections 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.429+0000 7fcf5dc91740 1 -- 172.21.15.28:0/1403783721 wait complete. 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.429+0000 7fcf5dc91740 1 -- start start 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.429+0000 7fcf5dc91740 1 Processor -- start 2024-09-15T21:58:16.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de805d0 0x559e8dea0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de9df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf5dc91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de9e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf541b2740 con 0x559e8de81b40 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf541b2070 con 0x559e8de8b370 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf5dc91740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf541af2b0 con 0x559e8de805d0 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf5dc91740 10 client.0 _ll_register_callbacks cb 0x7fcf50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de9df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de9df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55732/0 (socket says 172.21.15.28:55732) 2024-09-15T21:58:16.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 -- 172.21.15.28:0/920364424 learned_addr learned my addr 172.21.15.28:0/920364424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf59654640 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de805d0 0x559e8dea0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf59e55640 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de9e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 -- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de805d0 msgr2=0x559e8dea0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de805d0 0x559e8dea0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 -- 172.21.15.28:0/920364424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 msgr2=0x559e8de9e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de9e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 -- 172.21.15.28:0/920364424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e8de7c510 con 0x559e8de81b40 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf58e53640 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de9df90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fcf440cd080 tx=0x7fcf4409b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5843cd52b5ac3421 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.430+0000 7fcf3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:16.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf3cff9640 1 -- 172.21.15.28:0/920364424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44021040 con 0x559e8de81b40 2024-09-15T21:58:16.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf3cff9640 1 -- 172.21.15.28:0/920364424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fcf440a4300 con 0x559e8de81b40 2024-09-15T21:58:16.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf50003d90 con 0x559e8de81b40 2024-09-15T21:58:16.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf3cff9640 1 -- 172.21.15.28:0/920364424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf4409bb20 con 0x559e8de81b40 2024-09-15T21:58:16.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 10 client.15849 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:16.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e8df057b0 con 0x559e8de81b40 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf3cff9640 1 -- 172.21.15.28:0/920364424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf50003d90 con 0x559e8de81b40 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf3cff9640 1 -- 172.21.15.28:0/920364424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf4409a9e0 con 0x559e8de81b40 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 10 client.15849 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 10 client.15849 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 20 client.15849 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8df07fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 4 client.15849 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.431+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e8ddedd70 con 0x559e8de9aef0 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.432+0000 7fcf59654640 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8df07fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.432+0000 7fcf3cff9640 10 client.15849 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.432+0000 7fcf59654640 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8df07fc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9fe9accd4cfb8b7a server_cookie=1b61cf0ff73b685d in_seq=0 out_seq=0 2024-09-15T21:58:16.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf3cff9640 1 -- 172.21.15.28:0/920364424 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e8ddedd70 con 0x559e8de9aef0 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf3cff9640 10 client.15849 handle_command_reply: tid=0 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 client.15849 shutdown 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 msgr2=0x559e8de9df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de9df90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fcf440cd080 tx=0x7fcf4409b480 comp rx=0 tx=0).stop 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 shutdown_connections 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x559e8de9aef0 0x559e8df07fc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e8de8b370 0x559e8de9e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e8de81b40 0x559e8de9df90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 --2- 172.21.15.28:0/920364424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559e8de805d0 0x559e8dea0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 >> 172.21.15.28:0/920364424 conn(0x559e8de49310 msgr2=0x559e8de898f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.433+0000 7fcf3cff9640 0 client.15849 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:58:16.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.434+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 shutdown_connections 2024-09-15T21:58:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.434+0000 7fcf5dc91740 1 -- 172.21.15.28:0/920364424 wait complete. 2024-09-15T21:58:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.434+0000 7fcf5dc91740 20 client.15849 trim_cache size 0 max 16384 2024-09-15T21:58:16.440 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcf400a3560 msgr2=0x7fcf400a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcf400a3560 0x7fcf400a5a20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf50000c00 tx=0x7fcf50005b50 comp rx=0 tx=0).stop 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54103780 msgr2=0x7fcf54100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54103780 0x7fcf54100f70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400d740 tx=0x7fcf4400dc00 comp rx=0 tx=0).stop 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 shutdown_connections 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcf400a3560 0x7fcf400a5a20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf54108300 0x7fcf541014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf54103780 0x7fcf54100f70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 --2- 172.21.15.28:0/4132556082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf54102dd0 0x7fcf54100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 >> 172.21.15.28:0/4132556082 conn(0x7fcf540fc7d0 msgr2=0x7fcf54105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 shutdown_connections 2024-09-15T21:58:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:16.436+0000 7fcf5b8b8640 1 -- 172.21.15.28:0/4132556082 wait complete. 2024-09-15T21:58:17.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T21:58:17.454 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:17.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:17 smithi028 ceph-mon[30969]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-09-15T21:58:17.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:17 smithi203 ceph-mon[35546]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-09-15T21:58:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:17 smithi174 ceph-mon[35244]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-09-15T21:58:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 -- 172.21.15.28:0/1207658081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49041036e0 msgr2=0x7f4904107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 --2- 172.21.15.28:0/1207658081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49041036e0 0x7f4904107b90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009930 tx=0x7f48f402a420 comp rx=0 tx=0).stop 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 -- 172.21.15.28:0/1207658081 shutdown_connections 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 --2- 172.21.15.28:0/1207658081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49041082a0 0x7f490410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 --2- 172.21.15.28:0/1207658081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49041036e0 0x7f4904107b90 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 --2- 172.21.15.28:0/1207658081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4904102d30 0x7f4904103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 -- 172.21.15.28:0/1207658081 >> 172.21.15.28:0/1207658081 conn(0x7f49040fc7d0 msgr2=0x7f49040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 -- 172.21.15.28:0/1207658081 shutdown_connections 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.960+0000 7f490bc09640 1 -- 172.21.15.28:0/1207658081 wait complete. 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.961+0000 7f490bc09640 1 Processor -- start 2024-09-15T21:58:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.961+0000 7f490bc09640 1 -- start start 2024-09-15T21:58:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.961+0000 7f490bc09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904102d30 0x7f4904100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.961+0000 7f490bc09640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49041036e0 0x7f4904100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f490bc09640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49041082a0 0x7f4904101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904102d30 0x7f4904100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f490bc09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49041120a0 con 0x7f4904102d30 2024-09-15T21:58:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f490bc09640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4904111f20 con 0x7f49041082a0 2024-09-15T21:58:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f490bc09640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4904112220 con 0x7f49041036e0 2024-09-15T21:58:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904102d30 0x7f4904100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55748/0 (socket says 172.21.15.28:55748) 2024-09-15T21:58:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 -- 172.21.15.28:0/3950054186 learned_addr learned my addr 172.21.15.28:0/3950054186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49091a4640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49041036e0 0x7f4904100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f490a1a6640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49041082a0 0x7f4904101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 -- 172.21.15.28:0/3950054186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49041036e0 msgr2=0x7f4904100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49041036e0 0x7f4904100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 -- 172.21.15.28:0/3950054186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49041082a0 msgr2=0x7f4904101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49041082a0 0x7f4904101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 -- 172.21.15.28:0/3950054186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49041a6a60 con 0x7f4904102d30 2024-09-15T21:58:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.962+0000 7f49099a5640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904102d30 0x7f4904100990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f490000e9f0 tx=0x7f490000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.963+0000 7f48faffd640 1 -- 172.21.15.28:0/3950054186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490000cd80 con 0x7f4904102d30 2024-09-15T21:58:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.963+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49041a6d50 con 0x7f4904102d30 2024-09-15T21:58:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.963+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49041a7250 con 0x7f4904102d30 2024-09-15T21:58:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.963+0000 7f48faffd640 1 -- 172.21.15.28:0/3950054186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4900010910 con 0x7f4904102d30 2024-09-15T21:58:17.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.963+0000 7f48faffd640 1 -- 172.21.15.28:0/3950054186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4900018990 con 0x7f4904102d30 2024-09-15T21:58:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.966+0000 7f490dfe2740 1 Processor -- start 2024-09-15T21:58:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.966+0000 7f490dfe2740 1 -- start start 2024-09-15T21:58:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f48faffd640 1 -- 172.21.15.28:0/3950054186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f4900010ac0 con 0x7f4904102d30 2024-09-15T21:58:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413beda090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bedaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bee5370 0x56413bee7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490dfe2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56413be402f0 con 0x56413bedbb40 2024-09-15T21:58:17.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490dfe2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56413bde7cf0 con 0x56413bee5370 2024-09-15T21:58:17.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490dfe2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56413be3c2b0 con 0x56413beda5d0 2024-09-15T21:58:17.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413beda090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f49091a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bee5370 0x56413bee7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413beda090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55750/0 (socket says 172.21.15.28:55750) 2024-09-15T21:58:17.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 -- 172.21.15.28:0/2626489163 learned_addr learned my addr 172.21.15.28:0/2626489163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:17.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f49099a5640 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bedaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 -- 172.21.15.28:0/2626489163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 msgr2=0x56413bedaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:17.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bedaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 -- 172.21.15.28:0/2626489163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bee5370 msgr2=0x56413bee7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:17.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bee5370 0x56413bee7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f490a1a6640 1 -- 172.21.15.28:0/2626489163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413bdcc3a0 con 0x56413bedbb40 2024-09-15T21:58:17.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.967+0000 7f49091a4640 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bee5370 0x56413bee7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:17.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.968+0000 7f490a1a6640 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413beda090 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009f20 tx=0x7f48fc00f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e7fbc09ee4ae81 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.968+0000 7f48effff640 1 -- 172.21.15.28:0/2626489163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc015070 con 0x56413bedbb40 2024-09-15T21:58:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.968+0000 7f48faffd640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48f00a35e0 0x7f48f00a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.968+0000 7f48effff640 1 -- 172.21.15.28:0/2626489163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48fc01ab40 con 0x56413bedbb40 2024-09-15T21:58:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.968+0000 7f48effff640 1 -- 172.21.15.28:0/2626489163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc011c50 con 0x56413bedbb40 2024-09-15T21:58:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.968+0000 7f49099a5640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48f00a35e0 0x7f48f00a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.968+0000 7f48faffd640 1 -- 172.21.15.28:0/3950054186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f49000d0670 con 0x7f4904102d30 2024-09-15T21:58:17.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 -- 172.21.15.28:0/2626489163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 msgr2=0x56413beda090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:17.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f49099a5640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48f00a35e0 0x7f48f00a5aa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f49000cd880 tx=0x7f49000d6530 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:17.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413beda090 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f48fc009f20 tx=0x7f48fc00f7f0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 -- 172.21.15.28:0/2626489163 shutdown_connections 2024-09-15T21:58:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bee5370 0x56413bee7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bedaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2626489163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413beda090 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 -- 172.21.15.28:0/2626489163 >> 172.21.15.28:0/2626489163 conn(0x56413bea3310 msgr2=0x56413bed6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.969+0000 7f490dfe2740 1 -- 172.21.15.28:0/2626489163 shutdown_connections 2024-09-15T21:58:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.970+0000 7f490dfe2740 1 -- 172.21.15.28:0/2626489163 wait complete. 2024-09-15T21:58:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.970+0000 7f490dfe2740 1 -- start start 2024-09-15T21:58:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.970+0000 7f490dfe2740 1 Processor -- start 2024-09-15T21:58:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bee4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bee5370 0x56413bee25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490dfe2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56413bee8070 con 0x56413bee5370 2024-09-15T21:58:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490dfe2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56413bdfb060 con 0x56413bedbb40 2024-09-15T21:58:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490dfe2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56413bee81f0 con 0x56413beda5d0 2024-09-15T21:58:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490dfe2740 10 client.0 _ll_register_callbacks cb 0x56413bb4e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bee4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bee4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:37618/0 (socket says 172.21.15.28:37618) 2024-09-15T21:58:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 -- 172.21.15.28:0/3686684429 learned_addr learned my addr 172.21.15.28:0/3686684429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bee5370 0x56413bee25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f49091a4640 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 -- 172.21.15.28:0/3686684429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 msgr2=0x56413bee4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bee4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 -- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 msgr2=0x56413bee1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.971+0000 7f490a1a6640 1 -- 172.21.15.28:0/3686684429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413bbb13c0 con 0x56413bee5370 2024-09-15T21:58:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f490a1a6640 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bee5370 0x56413bee25d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f48fc011930 tx=0x7f48fc009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532c57f03bd2d8ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f48ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:17.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f48ed7fa640 1 -- 172.21.15.28:0/3686684429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc015040 con 0x56413bee5370 2024-09-15T21:58:17.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f48ed7fa640 1 -- 172.21.15.28:0/3686684429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f48fc012da0 con 0x56413bee5370 2024-09-15T21:58:17.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48fc01ab40 con 0x56413bee5370 2024-09-15T21:58:17.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f48ed7fa640 1 -- 172.21.15.28:0/3686684429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f48fc004a40 con 0x56413bee5370 2024-09-15T21:58:17.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f490dfe2740 10 client.15861 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.972+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56413bef3ad0 con 0x56413bee5370 2024-09-15T21:58:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f48ed7fa640 1 -- 172.21.15.28:0/3686684429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48fc01a940 con 0x56413bee5370 2024-09-15T21:58:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f48ed7fa640 1 -- 172.21.15.28:0/3686684429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f48fc005370 con 0x56413bee5370 2024-09-15T21:58:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f490dfe2740 10 client.15861 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f490dfe2740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f490dfe2740 20 client.15861 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bef4e20 0x56413bef7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f490dfe2740 4 client.15861 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.973+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f48fc01ab40 con 0x56413bef4e20 2024-09-15T21:58:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.974+0000 7f49091a4640 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bef4e20 0x56413bef7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.974+0000 7f48ed7fa640 10 client.15861 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.974+0000 7f49091a4640 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bef4e20 0x56413bef7210 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=298315e09416e246 server_cookie=625bab44d7253c6f in_seq=0 out_seq=0 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.975+0000 7f48ed7fa640 1 -- 172.21.15.28:0/3686684429 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f48fc01ab40 con 0x56413bef4e20 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.975+0000 7f48ed7fa640 10 client.15861 handle_command_reply: tid=0 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.975+0000 7f490dfe2740 1 client.15861 shutdown 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.975+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bee5370 msgr2=0x56413bee25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.975+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bee5370 0x56413bee25d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f48fc011930 tx=0x7f48fc009330 comp rx=0 tx=0).stop 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 shutdown_connections 2024-09-15T21:58:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bef4e20 0x56413bef7210 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bee5370 0x56413bee25d0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3686684429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413beda5d0 0x56413bee4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 >> 172.21.15.28:0/3686684429 conn(0x56413bea3310 msgr2=0x56413bedc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f48ed7fa640 0 client.15861 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 shutdown_connections 2024-09-15T21:58:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 1 -- 172.21.15.28:0/3686684429 wait complete. 2024-09-15T21:58:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.976+0000 7f490dfe2740 20 client.15861 trim_cache size 0 max 16384 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.999+0000 7f490dfe2740 1 Processor -- start 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:17.999+0000 7f490dfe2740 1 -- start start 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413beda5d0 0x56413bef1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413bedd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 0x56413bedd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490a1a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413beda5d0 0x56413bef1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490a1a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413beda5d0 0x56413bef1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:46928/0 (socket says 172.21.15.28:46928) 2024-09-15T21:58:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490a1a6640 1 -- 172.21.15.28:0/3685665946 learned_addr learned my addr 172.21.15.28:0/3685665946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49041b5610 con 0x56413bedbb40 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49041b7150 con 0x56413beda5d0 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f49091a4640 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413bedd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49041b5ce0 con 0x56413bee5370 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.000+0000 7f490a1a6640 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 0x56413bedd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f49091a4640 1 -- 172.21.15.28:0/3685665946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 msgr2=0x56413bedd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f49091a4640 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 0x56413bedd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f49091a4640 1 -- 172.21.15.28:0/3685665946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413beda5d0 msgr2=0x56413bef1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f49091a4640 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413beda5d0 0x56413bef1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f49091a4640 1 -- 172.21.15.28:0/3685665946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413bee7f50 con 0x56413bedbb40 2024-09-15T21:58:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f49091a4640 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413bedd120 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f48f402a940 tx=0x7f48f40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46882a82111e1e86 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f48ef7fe640 1 -- 172.21.15.28:0/3685665946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f404a070 con 0x56413bedbb40 2024-09-15T21:58:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f48ef7fe640 1 -- 172.21.15.28:0/3685665946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48f4005520 con 0x56413bedbb40 2024-09-15T21:58:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.001+0000 7f48ef7fe640 1 -- 172.21.15.28:0/3685665946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4035040 con 0x56413bedbb40 2024-09-15T21:58:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 msgr2=0x56413bedd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413bedd120 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f48f402a940 tx=0x7f48f40058d0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 shutdown_connections 2024-09-15T21:58:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 0x56413bedd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413bedbb40 0x56413bedd120 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 --2- 172.21.15.28:0/3685665946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413beda5d0 0x56413bef1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 >> 172.21.15.28:0/3685665946 conn(0x56413bea3310 msgr2=0x56413bed5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 shutdown_connections 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.002+0000 7f490dfe2740 1 -- 172.21.15.28:0/3685665946 wait complete. 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.003+0000 7f490dfe2740 1 -- start start 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.003+0000 7f490dfe2740 1 Processor -- start 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.003+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413beda5d0 0x56413bee9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:18.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490dfe2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 0x56413bee9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490dfe2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49041b6a80 con 0x56413beda5d0 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490dfe2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49041b63b0 con 0x56413bedbb40 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490dfe2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49041b35f0 con 0x56413bee5370 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490dfe2740 10 client.0 _ll_register_callbacks cb 0x7f49000d6400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490a1a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413beda5d0 0x56413bee9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:46944/0 (socket says 172.21.15.28:46944) 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 -- 172.21.15.28:0/2832387902 learned_addr learned my addr 172.21.15.28:0/2832387902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 -- 172.21.15.28:0/2832387902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 msgr2=0x56413bee9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 0x56413bee9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 -- 172.21.15.28:0/2832387902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413beda5d0 msgr2=0x56413bee9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413beda5d0 0x56413bee9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 -- 172.21.15.28:0/2832387902 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413bf5e8a0 con 0x56413bedbb40 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f490a1a6640 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413beda5d0 0x56413bee9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f49091a4640 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee9670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009850 tx=0x7f48f4032dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1f2f5cc0405559 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.004+0000 7f48ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.005+0000 7f48ed7fa640 1 -- 172.21.15.28:0/2832387902 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48f4053060 con 0x56413bedbb40 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.005+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48f4005520 con 0x56413bedbb40 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.005+0000 7f48ed7fa640 1 -- 172.21.15.28:0/2832387902 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f48f40052c0 con 0x56413bedbb40 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.005+0000 7f48ed7fa640 1 -- 172.21.15.28:0/2832387902 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f48f404a040 con 0x56413bedbb40 2024-09-15T21:58:18.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.005+0000 7f490dfe2740 10 client.25478 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.005+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56413bf5fb30 con 0x56413bedbb40 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.005+0000 7f48ed7fa640 1 -- 172.21.15.28:0/2832387902 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48f4004770 con 0x56413bedbb40 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f48ed7fa640 1 -- 172.21.15.28:0/2832387902 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f48f4003890 con 0x56413bedbb40 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f490dfe2740 10 client.25478 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f490dfe2740 10 client.25478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f490dfe2740 20 client.25478 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bf60290 0x56413bf626b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f490dfe2740 4 client.25478 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:58:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f48f4005520 con 0x56413bf60290 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.006+0000 7f490a1a6640 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bf60290 0x56413bf626b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.007+0000 7f48ed7fa640 10 client.25478 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.007+0000 7f490a1a6640 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bf60290 0x56413bf626b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d36fd9ecdadcaee1 server_cookie=10dac784d3570e63 in_seq=0 out_seq=0 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.007+0000 7f48ed7fa640 1 -- 172.21.15.28:0/2832387902 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f48f4005520 con 0x56413bf60290 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.007+0000 7f48ed7fa640 10 client.25478 handle_command_reply: tid=0 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.007+0000 7f490dfe2740 1 client.25478 shutdown 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.007+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 msgr2=0x56413bee9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.007+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee9670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009850 tx=0x7f48f4032dd0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 shutdown_connections 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56413bf60290 0x56413bf626b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56413bee5370 0x56413bee9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f48ed7fa640 0 client.25478 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56413bedbb40 0x56413bee9670 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 --2- 172.21.15.28:0/2832387902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56413beda5d0 0x56413bee9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 >> 172.21.15.28:0/2832387902 conn(0x56413bea3310 msgr2=0x56413bee3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 shutdown_connections 2024-09-15T21:58:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 1 -- 172.21.15.28:0/2832387902 wait complete. 2024-09-15T21:58:18.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.008+0000 7f490dfe2740 20 client.25478 trim_cache size 0 max 16384 2024-09-15T21:58:18.016 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:58:18.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48f00a35e0 msgr2=0x7f48f00a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:18.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48f00a35e0 0x7f48f00a5aa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f49000cd880 tx=0x7f49000d6530 comp rx=0 tx=0).stop 2024-09-15T21:58:18.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904102d30 msgr2=0x7f4904100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:18.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904102d30 0x7f4904100990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f490000e9f0 tx=0x7f490000eeb0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 shutdown_connections 2024-09-15T21:58:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48f00a35e0 0x7f48f00a5aa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49041082a0 0x7f4904101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49041036e0 0x7f4904100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 --2- 172.21.15.28:0/3950054186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904102d30 0x7f4904100990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.009+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 >> 172.21.15.28:0/3950054186 conn(0x7f49040fc7d0 msgr2=0x7f4904105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.010+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 shutdown_connections 2024-09-15T21:58:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:18.010+0000 7f490bc09640 1 -- 172.21.15.28:0/3950054186 wait complete. 2024-09-15T21:58:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:19 smithi203 ceph-mon[35546]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T21:58:19.725 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T21:58:19.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T21:58:19.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:19 smithi028 ceph-mon[30969]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T21:58:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:19 smithi174 ceph-mon[35244]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T21:58:20.097 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:20.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 -- 172.21.15.28:0/3335640280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac103660 msgr2=0x7f74ac10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/3335640280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac103660 0x7f74ac10bb50 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009900 tx=0x7f74a002a480 comp rx=0 tx=0).stop 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 -- 172.21.15.28:0/3335640280 shutdown_connections 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/3335640280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac10c170 0x7f74ac10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/3335640280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac103660 0x7f74ac10bb50 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/3335640280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74ac102d40 0x7f74ac103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 -- 172.21.15.28:0/3335640280 >> 172.21.15.28:0/3335640280 conn(0x7f74ac0fc7a0 msgr2=0x7f74ac0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 -- 172.21.15.28:0/3335640280 shutdown_connections 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.600+0000 7f74b0f8a640 1 -- 172.21.15.28:0/3335640280 wait complete. 2024-09-15T21:58:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.601+0000 7f74b0f8a640 1 Processor -- start 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74b0f8a640 1 -- start start 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74b0f8a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac102d40 0x7f74ac1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74b0f8a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74ac103660 0x7f74ac100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74b0f8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac10c170 0x7f74ac101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74b0f8a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac111f70 con 0x7f74ac10c170 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74b0f8a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac111df0 con 0x7f74ac102d40 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74b0f8a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1120f0 con 0x7f74ac103660 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74aa59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac102d40 0x7f74ac1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac10c170 0x7f74ac101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74aa59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac102d40 0x7f74ac1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:34822/0 (socket says 172.21.15.28:34822) 2024-09-15T21:58:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74aa59c640 1 -- 172.21.15.28:0/348001920 learned_addr learned my addr 172.21.15.28:0/348001920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.602+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74ac103660 0x7f74ac100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f74aa59c640 1 -- 172.21.15.28:0/348001920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74ac103660 msgr2=0x7f74ac100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f74aa59c640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74ac103660 0x7f74ac100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f74aa59c640 1 -- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac10c170 msgr2=0x7f74ac101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f74aa59c640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac10c170 0x7f74ac101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f74aa59c640 1 -- 172.21.15.28:0/348001920 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ac1a6930 con 0x7f74ac102d40 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f74aad9d640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac10c170 0x7f74ac101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f74aa59c640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac102d40 0x7f74ac1009a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f749c00d770 tx=0x7f749c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f749b7fe640 1 -- 172.21.15.28:0/348001920 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c017070 con 0x7f74ac102d40 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.603+0000 7f749b7fe640 1 -- 172.21.15.28:0/348001920 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749c005500 con 0x7f74ac102d40 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.604+0000 7f749b7fe640 1 -- 172.21.15.28:0/348001920 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c004e00 con 0x7f74ac102d40 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.604+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ac1a6c20 con 0x7f74ac102d40 2024-09-15T21:58:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.604+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ac1a70e0 con 0x7f74ac102d40 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 Processor -- start 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 -- start start 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d275f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d280800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d283130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56097d1db280 con 0x56097d280d40 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56097d182b10 con 0x56097d275b60 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.607+0000 7f74b3363740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56097d1d7240 con 0x56097d276480 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d283130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d283130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35330/0 (socket says 172.21.15.28:35330) 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74a9d9b640 1 -- 172.21.15.28:0/620953332 learned_addr learned my addr 172.21.15.28:0/620953332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d275f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74aad9d640 1 -- 172.21.15.28:0/620953332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 msgr2=0x56097d280800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74aad9d640 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d280800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74aad9d640 1 -- 172.21.15.28:0/620953332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 msgr2=0x56097d283130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74aad9d640 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d283130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74aad9d640 1 -- 172.21.15.28:0/620953332 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56097d18f1f0 con 0x56097d275b60 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.608+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d283130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.609+0000 7f749b7fe640 1 -- 172.21.15.28:0/348001920 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f749c009c30 con 0x7f74ac102d40 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.609+0000 7f74aad9d640 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d275f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f74a4010710 tx=0x7f74a4010bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b08d5b71627f27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.609+0000 7f7498ff9640 1 -- 172.21.15.28:0/620953332 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a4004b20 con 0x56097d275b60 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.609+0000 7f7498ff9640 1 -- 172.21.15.28:0/620953332 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a400edb0 con 0x56097d275b60 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.609+0000 7f7498ff9640 1 -- 172.21.15.28:0/620953332 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a40056d0 con 0x56097d275b60 2024-09-15T21:58:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f749b7fe640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f74940a34c0 0x7f74940a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f74b3363740 1 -- 172.21.15.28:0/620953332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 msgr2=0x56097d275f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f74b3363740 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d275f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f74a4010710 tx=0x7f74a4010bd0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f74b3363740 1 -- 172.21.15.28:0/620953332 shutdown_connections 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f74b3363740 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d283130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f74b3363740 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d280800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f74b3363740 1 --2- 172.21.15.28:0/620953332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d275f40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f74b3363740 1 -- 172.21.15.28:0/620953332 >> 172.21.15.28:0/620953332 conn(0x56097d2720f0 msgr2=0x56097d2724d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.610+0000 7f749b7fe640 1 -- 172.21.15.28:0/348001920 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f749c0cff40 con 0x7f74ac102d40 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.611+0000 7f74aa59c640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f74940a34c0 0x7f74940a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.611+0000 7f74aa59c640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f74940a34c0 0x7f74940a5980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f749c0cd150 tx=0x7f749c0bd040 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 -- 172.21.15.28:0/620953332 shutdown_connections 2024-09-15T21:58:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 -- 172.21.15.28:0/620953332 wait complete. 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 -- start start 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 Processor -- start 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d276480 0x56097d27e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d280d40 0x56097d27e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56097d283960 con 0x56097d275b60 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56097d195f10 con 0x56097d276480 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.612+0000 7f74b3363740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56097d283ae0 con 0x56097d280d40 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74b3363740 10 client.0 _ll_register_callbacks cb 0x56097d15ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d276480 0x56097d27e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d276480 0x56097d27e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:34850/0 (socket says 172.21.15.28:34850) 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 -- 172.21.15.28:0/1087058212 learned_addr learned my addr 172.21.15.28:0/1087058212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 -- 172.21.15.28:0/1087058212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d280d40 msgr2=0x56097d27e600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d280d40 0x56097d27e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 -- 172.21.15.28:0/1087058212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 msgr2=0x56097d27db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 -- 172.21.15.28:0/1087058212 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56097cf4c6a0 con 0x56097d276480 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74a400edb0 con 0x56097d276480 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74aad9d640 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d276480 0x56097d27e0c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74a0005940 tx=0x7f74a0031920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47f12e4cb669d801 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.613+0000 7f7491ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f7491ffb640 1 -- 172.21.15.28:0/1087058212 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0035040 con 0x56097d276480 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f7491ffb640 1 -- 172.21.15.28:0/1087058212 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f74a0004460 con 0x56097d276480 2024-09-15T21:58:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f7491ffb640 1 -- 172.21.15.28:0/1087058212 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f74a0005040 con 0x56097d276480 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 10 client.25490 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56097d28f490 con 0x56097d276480 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f7491ffb640 1 -- 172.21.15.28:0/1087058212 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f74a00051e0 con 0x56097d276480 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 10 client.25490 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 10 client.25490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 20 client.25490 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d2908b0 0x56097d292ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 4 client.25490 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.614+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56097d1e87f0 con 0x56097d2908b0 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.615+0000 7f74aad9d640 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d2908b0 0x56097d292ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.615+0000 7f7491ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.615+0000 7f74aad9d640 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d2908b0 0x56097d292ce0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd4f0548c8a7e37 server_cookie=44ab9757280c912f in_seq=0 out_seq=0 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f7491ffb640 1 -- 172.21.15.28:0/1087058212 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56097d1e87f0 con 0x56097d2908b0 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f7491ffb640 10 client.25490 handle_command_reply: tid=0 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 client.25490 shutdown 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d276480 msgr2=0x56097d27e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d276480 0x56097d27e0c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74a0005940 tx=0x7f74a0031920 comp rx=0 tx=0).stop 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 shutdown_connections 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d2908b0 0x56097d292ce0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d280d40 0x56097d27e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d276480 0x56097d27e0c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 --2- 172.21.15.28:0/1087058212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f7491ffb640 0 client.25490 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 >> 172.21.15.28:0/1087058212 conn(0x56097d2720f0 msgr2=0x56097d277dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 shutdown_connections 2024-09-15T21:58:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 1 -- 172.21.15.28:0/1087058212 wait complete. 2024-09-15T21:58:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.616+0000 7f74b3363740 20 client.25490 trim_cache size 0 max 16384 2024-09-15T21:58:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.634+0000 7f74b3363740 1 Processor -- start 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.634+0000 7f74b3363740 1 -- start start 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74b3363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74b3363740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2985b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74b3363740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d280d40 0x56097d298af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74b3363740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1b0d00 con 0x56097d275b60 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74b3363740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1b2840 con 0x56097d280d40 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74b3363740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1b13d0 con 0x56097d276480 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35348/0 (socket says 172.21.15.28:35348) 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 -- 172.21.15.28:0/2697670611 learned_addr learned my addr 172.21.15.28:0/2697670611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d280d40 0x56097d298af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 -- 172.21.15.28:0/2697670611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 msgr2=0x56097d2985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 -- 172.21.15.28:0/2697670611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d280d40 msgr2=0x56097d298af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d280d40 0x56097d298af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74aad9d640 1 -- 172.21.15.28:0/2697670611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56097d271f20 con 0x56097d275b60 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.635+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.636+0000 7f74aad9d640 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27e450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f74a40098b0 tx=0x7f74a4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44af57ad94c6137 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.636+0000 7f7493fff640 1 -- 172.21.15.28:0/2697670611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a4004e30 con 0x56097d275b60 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.636+0000 7f7493fff640 1 -- 172.21.15.28:0/2697670611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a4004fd0 con 0x56097d275b60 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.636+0000 7f7493fff640 1 -- 172.21.15.28:0/2697670611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a402a9b0 con 0x56097d275b60 2024-09-15T21:58:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 -- 172.21.15.28:0/2697670611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 msgr2=0x56097d27e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27e450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f74a40098b0 tx=0x7f74a4005bd0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 -- 172.21.15.28:0/2697670611 shutdown_connections 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d280d40 0x56097d298af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 --2- 172.21.15.28:0/2697670611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d275b60 0x56097d27e450 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 -- 172.21.15.28:0/2697670611 >> 172.21.15.28:0/2697670611 conn(0x56097d2720f0 msgr2=0x56097d27faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 -- 172.21.15.28:0/2697670611 shutdown_connections 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.637+0000 7f74b3363740 1 -- 172.21.15.28:0/2697670611 wait complete. 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.638+0000 7f74b3363740 1 -- start start 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.638+0000 7f74b3363740 1 Processor -- start 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.638+0000 7f74b3363740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d288d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.638+0000 7f74b3363740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2892a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.638+0000 7f74b3363740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d285d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.638+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d285d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2892a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2892a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:39408/0 (socket says 172.21.15.28:39408) 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d285d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35360/0 (socket says 172.21.15.28:35360) 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 -- 172.21.15.28:0/2005558999 learned_addr learned my addr 172.21.15.28:0/2005558999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1b2170 con 0x56097d280d40 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1b1aa0 con 0x56097d275b60 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac1aece0 con 0x56097d276480 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74b3363740 10 client.0 _ll_register_callbacks cb 0x7f74a0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 -- 172.21.15.28:0/2005558999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 msgr2=0x56097d288d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d288d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 -- 172.21.15.28:0/2005558999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 msgr2=0x56097d285d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d285d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74a9d9b640 1 -- 172.21.15.28:0/2005558999 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56097d27bb60 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.639+0000 7f74aad9d640 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d285d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f74a9d9b640 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2892a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x56097d27f1c0 tx=0x7f74a0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5172a2b834688b54 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f7491ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f7491ffb640 1 -- 172.21.15.28:0/2005558999 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0035070 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74a4004fd0 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f7491ffb640 1 -- 172.21.15.28:0/2005558999 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f74a0004820 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f7491ffb640 1 -- 172.21.15.28:0/2005558999 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f74a0065050 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f74b3363740 10 client.25444 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.640+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56097d3042f0 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f7491ffb640 1 -- 172.21.15.28:0/2005558999 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a4004fd0 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f7491ffb640 1 -- 172.21.15.28:0/2005558999 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f74a0003b80 con 0x56097d276480 2024-09-15T21:58:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f74b3363740 10 client.25444 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f74b3363740 10 client.25444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f74b3363740 20 client.25444 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f74b3363740 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d304a60 0x56097d306ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f74b3363740 4 client.25444 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56097d1e2d00 con 0x56097d304a60 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.641+0000 7f74aad9d640 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d304a60 0x56097d306ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.642+0000 7f74aad9d640 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d304a60 0x56097d306ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1c40f6fb6954843 server_cookie=83a551fd26c07cb6 in_seq=0 out_seq=0 2024-09-15T21:58:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.642+0000 7f7491ffb640 10 client.25444 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f7491ffb640 1 -- 172.21.15.28:0/2005558999 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56097d1e2d00 con 0x56097d304a60 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f7491ffb640 10 client.25444 handle_command_reply: tid=0 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 client.25444 shutdown 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 msgr2=0x56097d2892a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2892a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x56097d27f1c0 tx=0x7f74a0009260 comp rx=0 tx=0).stop 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 shutdown_connections 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56097d304a60 0x56097d306ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56097d280d40 0x56097d285d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56097d276480 0x56097d2892a0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x56097d27f1c0 tx=0x7f74a0009260 comp rx=0 tx=0).stop 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 --2- 172.21.15.28:0/2005558999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56097d275b60 0x56097d288d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 >> 172.21.15.28:0/2005558999 conn(0x56097d2720f0 msgr2=0x56097d2775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.643+0000 7f7491ffb640 0 client.25444 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.644+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 shutdown_connections 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.644+0000 7f74b3363740 1 -- 172.21.15.28:0/2005558999 wait complete. 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.644+0000 7f74b3363740 20 client.25444 trim_cache size 0 max 16384 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T21:58:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ed00990a-2e25-4387-942d-0540118cc95d", 2024-09-15T21:58:20.652 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T21:58:20.652 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:58:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f74940a34c0 msgr2=0x7f74940a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f74940a34c0 0x7f74940a5980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f749c0cd150 tx=0x7f749c0bd040 comp rx=0 tx=0).stop 2024-09-15T21:58:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac102d40 msgr2=0x7f74ac1009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac102d40 0x7f74ac1009a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f749c00d770 tx=0x7f749c00dc30 comp rx=0 tx=0).stop 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 shutdown_connections 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f74940a34c0 0x7f74940a5980 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74ac10c170 0x7f74ac101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74ac103660 0x7f74ac100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 --2- 172.21.15.28:0/348001920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74ac102d40 0x7f74ac1009a0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f749c00d770 tx=0x7f749c00dc30 comp rx=0 tx=0).stop 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.645+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 >> 172.21.15.28:0/348001920 conn(0x7f74ac0fc7a0 msgr2=0x7f74ac0fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.647+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 shutdown_connections 2024-09-15T21:58:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:20.647+0000 7f74b0f8a640 1 -- 172.21.15.28:0/348001920 wait complete. 2024-09-15T21:58:21.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T21:58:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:21 smithi028 ceph-mon[30969]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T21:58:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:21 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T21:58:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:21 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:58:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:21 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T21:58:21.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:21 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:58:21.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:21 smithi203 ceph-mon[35546]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T21:58:21.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:21 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T21:58:21.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:21 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:58:21.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:21 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T21:58:21.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:21 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:58:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:21.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:21 smithi174 ceph-mon[35244]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T21:58:21.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:21 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T21:58:21.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:21 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:58:21.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:21 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T21:58:21.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:21 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:58:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/1002079284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58101e10 msgr2=0x7f7a581021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/1002079284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58101e10 0x7f7a581021f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009900 tx=0x7f7a4802a410 comp rx=0 tx=0).stop 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/1002079284 shutdown_connections 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/1002079284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a58106f40 0x7f7a5810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/1002079284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a581027c0 0x7f7a58106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/1002079284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58101e10 0x7f7a581021f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/1002079284 >> 172.21.15.28:0/1002079284 conn(0x7f7a580fd8d0 msgr2=0x7f7a580ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/1002079284 shutdown_connections 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.258+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/1002079284 wait complete. 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.259+0000 7f7a5d3ad640 1 Processor -- start 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.259+0000 7f7a5d3ad640 1 -- start start 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a5d3ad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a58101e10 0x7f7a58198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a5d3ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a581027c0 0x7f7a581989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a5d3ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58106f40 0x7f7a5819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a5d3ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58111ff0 con 0x7f7a58106f40 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a5d3ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58111e70 con 0x7f7a581027c0 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a5d3ad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58112170 con 0x7f7a58101e10 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58106f40 0x7f7a5819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58106f40 0x7f7a5819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35378/0 (socket says 172.21.15.28:35378) 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 -- 172.21.15.28:0/3593413801 learned_addr learned my addr 172.21.15.28:0/3593413801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a58101e10 0x7f7a58198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a567fc640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a581027c0 0x7f7a581989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 -- 172.21.15.28:0/3593413801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a58101e10 msgr2=0x7f7a58198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a58101e10 0x7f7a58198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 -- 172.21.15.28:0/3593413801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a581027c0 msgr2=0x7f7a581989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a581027c0 0x7f7a581989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a577fe640 1 -- 172.21.15.28:0/3593413801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5819d420 con 0x7f7a58106f40 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a58101e10 0x7f7a58198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.260+0000 7f7a567fc640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a581027c0 0x7f7a581989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.261+0000 7f7a577fe640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58106f40 0x7f7a5819cd40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000eb10 tx=0x7f7a5000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.261+0000 7f7a3ffff640 1 -- 172.21.15.28:0/3593413801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50004b20 con 0x7f7a58106f40 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.261+0000 7f7a3ffff640 1 -- 172.21.15.28:0/3593413801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a50004cc0 con 0x7f7a58106f40 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.261+0000 7f7a3ffff640 1 -- 172.21.15.28:0/3593413801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50010760 con 0x7f7a58106f40 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.261+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5819d710 con 0x7f7a58106f40 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.261+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a58074420 con 0x7f7a58106f40 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.264+0000 7f7a3ffff640 1 -- 172.21.15.28:0/3593413801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f7a50005020 con 0x7f7a58106f40 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.264+0000 7f7a3ffff640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7a400a35e0 0x7f7a400a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.264+0000 7f7a5f786740 1 Processor -- start 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.264+0000 7f7a5f786740 1 -- start start 2024-09-15T21:58:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a3ffff640 1 -- 172.21.15.28:0/3593413801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f7a500d0930 con 0x7f7a58106f40 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7a400a35e0 0x7f7a400a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c10e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c119130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c119130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c119130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35392/0 (socket says 172.21.15.28:35392) 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c11ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a5f786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55656c0742e0 con 0x55656c10edb0 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a5f786740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55656c01bce0 con 0x55656c119670 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a5f786740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55656c0702a0 con 0x55656c10e490 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a56ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c10e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.265+0000 7f7a567fc640 1 -- 172.21.15.28:0/3595732200 learned_addr learned my addr 172.21.15.28:0/3595732200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a577fe640 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c11ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a567fc640 1 -- 172.21.15.28:0/3595732200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 msgr2=0x55656c10e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a567fc640 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c10e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a567fc640 1 -- 172.21.15.28:0/3595732200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 msgr2=0x55656c11ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a567fc640 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c11ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a567fc640 1 -- 172.21.15.28:0/3595732200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55656c000440 con 0x55656c10edb0 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a567fc640 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c119130 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00bfd0 tx=0x7f7a4c010820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e2e45596702a9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7a400a35e0 0x7f7a400a5aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a48000c00 tx=0x7f7a48005120 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.266+0000 7f7a3d7fa640 1 -- 172.21.15.28:0/3595732200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c005220 con 0x55656c10edb0 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a3d7fa640 1 -- 172.21.15.28:0/3595732200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c004c80 con 0x55656c10edb0 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a3d7fa640 1 -- 172.21.15.28:0/3595732200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c00e980 con 0x55656c10edb0 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 -- 172.21.15.28:0/3595732200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 msgr2=0x55656c119130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c119130 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00bfd0 tx=0x7f7a4c010820 comp rx=0 tx=0).stop 2024-09-15T21:58:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 -- 172.21.15.28:0/3595732200 shutdown_connections 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c11ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c119130 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 --2- 172.21.15.28:0/3595732200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c10e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 -- 172.21.15.28:0/3595732200 >> 172.21.15.28:0/3595732200 conn(0x55656c10aa20 msgr2=0x55656c10ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 -- 172.21.15.28:0/3595732200 shutdown_connections 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.267+0000 7f7a5f786740 1 -- 172.21.15.28:0/3595732200 wait complete. 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 -- start start 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 Processor -- start 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c118f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c116240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c116780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55656c11c240 con 0x55656c10edb0 2024-09-15T21:58:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55656c02f050 con 0x55656c10e490 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55656c11c3c0 con 0x55656c119670 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a5f786740 10 client.0 _ll_register_callbacks cb 0x55656c06f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35400/0 (socket says 172.21.15.28:35400) 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a567fc640 1 -- 172.21.15.28:0/469908944 learned_addr learned my addr 172.21.15.28:0/469908944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.268+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c118f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a577fe640 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a567fc640 1 -- 172.21.15.28:0/469908944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 msgr2=0x55656c116780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a567fc640 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c116780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a567fc640 1 -- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 msgr2=0x55656c118f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a567fc640 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c118f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a567fc640 1 -- 172.21.15.28:0/469908944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55656bde53c0 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a567fc640 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c116240 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00e480 tx=0x7f7a4c005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277e0f23ec307837 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a3aff5640 1 -- 172.21.15.28:0/469908944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c00bc70 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4c004c80 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.269+0000 7f7a3aff5640 1 -- 172.21.15.28:0/469908944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f7a4c0046c0 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a3aff5640 1 -- 172.21.15.28:0/469908944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a4c0048c0 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 10 client.15894 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55656c127bc0 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a3aff5640 1 -- 172.21.15.28:0/469908944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c004c80 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a3aff5640 1 -- 172.21.15.28:0/469908944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7a4c00ecb0 con 0x55656c10edb0 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 10 client.15894 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 10 client.15894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 20 client.15894 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c128fd0 0x55656c12b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 4 client.15894 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55656c081850 con 0x55656c128fd0 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.270+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c128fd0 0x55656c12b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.271+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c128fd0 0x55656c12b3c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6551afc016c2d2f server_cookie=1c0f3e0d692936d6 in_seq=0 out_seq=0 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.271+0000 7f7a3aff5640 10 client.15894 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a3aff5640 1 -- 172.21.15.28:0/469908944 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55656c081850 con 0x55656c128fd0 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a3aff5640 10 client.15894 handle_command_reply: tid=0 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 client.15894 shutdown 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 msgr2=0x55656c116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c116240 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00e480 tx=0x7f7a4c005230 comp rx=0 tx=0).stop 2024-09-15T21:58:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 shutdown_connections 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c128fd0 0x55656c12b3c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c116780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c10edb0 0x55656c116240 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 --2- 172.21.15.28:0/469908944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c118f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 >> 172.21.15.28:0/469908944 conn(0x55656c10aa20 msgr2=0x55656c110700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a3aff5640 0 client.15894 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 shutdown_connections 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 1 -- 172.21.15.28:0/469908944 wait complete. 2024-09-15T21:58:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.272+0000 7f7a5f786740 20 client.15894 trim_cache size 0 max 16384 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 Processor -- start 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 -- start start 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c1254a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c130990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c130ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581ace60 con 0x55656c116870 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581ae9a0 con 0x55656c119670 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a5f786740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581ad530 con 0x55656c10e490 2024-09-15T21:58:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c130ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c130ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35402/0 (socket says 172.21.15.28:35402) 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a577fe640 1 -- 172.21.15.28:0/500363532 learned_addr learned my addr 172.21.15.28:0/500363532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c1254a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.290+0000 7f7a567fc640 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c130990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a567fc640 1 -- 172.21.15.28:0/500363532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 msgr2=0x55656c1254a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a567fc640 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c1254a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a567fc640 1 -- 172.21.15.28:0/500363532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 msgr2=0x55656c130ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a567fc640 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c130ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a567fc640 1 -- 172.21.15.28:0/500363532 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55656c106000 con 0x55656c119670 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a577fe640 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c130ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a567fc640 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c130990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55656c1177f0 tx=0x7f7a4c010f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f283b077fd623595 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a3cff9640 1 -- 172.21.15.28:0/500363532 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c041070 con 0x55656c119670 2024-09-15T21:58:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a3cff9640 1 -- 172.21.15.28:0/500363532 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c010d20 con 0x55656c119670 2024-09-15T21:58:22.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.291+0000 7f7a3cff9640 1 -- 172.21.15.28:0/500363532 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c004ce0 con 0x55656c119670 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 -- 172.21.15.28:0/500363532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 msgr2=0x55656c130990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c130990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55656c1177f0 tx=0x7f7a4c010f90 comp rx=0 tx=0).stop 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 -- 172.21.15.28:0/500363532 shutdown_connections 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c130ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c119670 0x55656c130990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 --2- 172.21.15.28:0/500363532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c10e490 0x55656c1254a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 -- 172.21.15.28:0/500363532 >> 172.21.15.28:0/500363532 conn(0x55656c10aa20 msgr2=0x55656c1118f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 -- 172.21.15.28:0/500363532 shutdown_connections 2024-09-15T21:58:22.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 -- 172.21.15.28:0/500363532 wait complete. 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 -- start start 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.292+0000 7f7a5f786740 1 Processor -- start 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c121dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c11d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a5f786740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c11d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a5f786740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581ae2d0 con 0x55656c116870 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c11d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c11d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35404/0 (socket says 172.21.15.28:35404) 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 -- 172.21.15.28:0/2267526908 learned_addr learned my addr 172.21.15.28:0/2267526908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581adc00 con 0x55656c10e490 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a581aae40 con 0x55656c119670 2024-09-15T21:58:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c121dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a5f786740 10 client.0 _ll_register_callbacks cb 0x7f7a4c00bed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a577fe640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c11d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 -- 172.21.15.28:0/2267526908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 msgr2=0x55656c11d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c11d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 -- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 msgr2=0x55656c121dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c121dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a567fc640 1 -- 172.21.15.28:0/2267526908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55656c11de90 con 0x55656c116870 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.293+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c121dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a567fc640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c11d100 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c000c00 tx=0x7f7a4c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed378c4ffedf4496 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a3aff5640 1 -- 172.21.15.28:0/2267526908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c041020 con 0x55656c116870 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a3aff5640 1 -- 172.21.15.28:0/2267526908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f7a4c004a60 con 0x55656c116870 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4c010d20 con 0x55656c116870 2024-09-15T21:58:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a3aff5640 1 -- 172.21.15.28:0/2267526908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a4c0036e0 con 0x55656c116870 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a5f786740 10 client.15900 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.294+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55656c1939e0 con 0x55656c116870 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a3aff5640 1 -- 172.21.15.28:0/2267526908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c010d20 con 0x55656c116870 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a3aff5640 1 -- 172.21.15.28:0/2267526908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7a4c00e9c0 con 0x55656c116870 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a5f786740 10 client.15900 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a5f786740 10 client.15900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a5f786740 20 client.15900 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a5f786740 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c193f50 0x55656c196340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a5f786740 4 client.15900 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55656c07bd60 con 0x55656c193f50 2024-09-15T21:58:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.295+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c193f50 0x55656c196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a3aff5640 10 client.15900 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a56ffd640 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c193f50 0x55656c196340 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58700b02e607192e server_cookie=b7d2b411f5100cf3 in_seq=0 out_seq=0 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a3aff5640 1 -- 172.21.15.28:0/2267526908 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55656c07bd60 con 0x55656c193f50 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a3aff5640 10 client.15900 handle_command_reply: tid=0 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a5f786740 1 client.15900 shutdown 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 msgr2=0x55656c11d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a5f786740 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c11d100 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c000c00 tx=0x7f7a4c0042f0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.296+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 shutdown_connections 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55656c193f50 0x55656c196340 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55656c119670 0x55656c11d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656c116870 0x55656c11d100 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 1 --2- 172.21.15.28:0/2267526908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55656c10e490 0x55656c121dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 >> 172.21.15.28:0/2267526908 conn(0x55656c10aa20 msgr2=0x55656c117aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a3aff5640 0 client.15900 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 shutdown_connections 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 1 -- 172.21.15.28:0/2267526908 wait complete. 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.297+0000 7f7a5f786740 20 client.15900 trim_cache size 0 max 16384 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout: "ed00990a-2e25-4387-942d-0540118cc95d": { 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ed00990a-2e25-4387-942d-0540118cc95d", 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-15T21:58:22.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7a400a35e0 msgr2=0x7f7a400a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7a400a35e0 0x7f7a400a5aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a48000c00 tx=0x7f7a48005120 comp rx=0 tx=0).stop 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58106f40 msgr2=0x7f7a5819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58106f40 0x7f7a5819cd40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000eb10 tx=0x7f7a5000c2d0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 shutdown_connections 2024-09-15T21:58:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7a400a35e0 0x7f7a400a5aa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a58106f40 0x7f7a5819cd40 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a581027c0 0x7f7a581989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 --2- 172.21.15.28:0/3593413801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a58101e10 0x7f7a58198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 >> 172.21.15.28:0/3593413801 conn(0x7f7a580fd8d0 msgr2=0x7f7a580fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 shutdown_connections 2024-09-15T21:58:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:22.298+0000 7f7a5d3ad640 1 -- 172.21.15.28:0/3593413801 wait complete. 2024-09-15T21:58:23.039 INFO:tasks.cephfs.filesystem:scrub status for tag:ed00990a-2e25-4387-942d-0540118cc95d - {'path': '/', 'tag': 'ed00990a-2e25-4387-942d-0540118cc95d', 'options': 'recursive,force'} 2024-09-15T21:58:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:23 smithi203 ceph-mon[35546]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.1 KiB/s wr, 16 op/s 2024-09-15T21:58:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:23 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T21:58:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:23 smithi174 ceph-mon[35244]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.1 KiB/s wr, 16 op/s 2024-09-15T21:58:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:23 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T21:58:23.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:23 smithi028 ceph-mon[30969]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.1 KiB/s wr, 16 op/s 2024-09-15T21:58:23.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:23 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T21:58:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 -- 172.21.15.28:0/2868414296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a81035a0 msgr2=0x7f27a810ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 --2- 172.21.15.28:0/2868414296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a81035a0 0x7f27a810ba80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f279c009900 tx=0x7f279c02a480 comp rx=0 tx=0).stop 2024-09-15T21:58:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 -- 172.21.15.28:0/2868414296 shutdown_connections 2024-09-15T21:58:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 --2- 172.21.15.28:0/2868414296 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a810c0d0 0x7f27a810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 --2- 172.21.15.28:0/2868414296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a81035a0 0x7f27a810ba80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 --2- 172.21.15.28:0/2868414296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27a8102c80 0x7f27a8103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 -- 172.21.15.28:0/2868414296 >> 172.21.15.28:0/2868414296 conn(0x7f27a8077990 msgr2=0x7f27a80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 -- 172.21.15.28:0/2868414296 shutdown_connections 2024-09-15T21:58:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.257+0000 7f27ad01e640 1 -- 172.21.15.28:0/2868414296 wait complete. 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.258+0000 7f27ad01e640 1 Processor -- start 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.258+0000 7f27ad01e640 1 -- start start 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27ad01e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a8102c80 0x7f27a8196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27ad01e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27a81035a0 0x7f27a8196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27ad01e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a810c0d0 0x7f27a819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27ad01e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27a810fca0 con 0x7f27a810c0d0 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27ad01e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f27a810fb20 con 0x7f27a81035a0 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27ad01e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27a810fe20 con 0x7f27a8102c80 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a810c0d0 0x7f27a819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a810c0d0 0x7f27a819a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35418/0 (socket says 172.21.15.28:35418) 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 -- 172.21.15.28:0/3951982791 learned_addr learned my addr 172.21.15.28:0/3951982791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a659c640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27a81035a0 0x7f27a8196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 -- 172.21.15.28:0/3951982791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a8102c80 msgr2=0x7f27a8196130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a8102c80 0x7f27a8196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a8102c80 0x7f27a8196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 -- 172.21.15.28:0/3951982791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27a81035a0 msgr2=0x7f27a8196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27a81035a0 0x7f27a8196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a759e640 1 -- 172.21.15.28:0/3951982791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a819b120 con 0x7f27a810c0d0 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.259+0000 7f27a659c640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27a81035a0 0x7f27a8196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.260+0000 7f27a759e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a810c0d0 0x7f27a819a9f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f27a000eb10 tx=0x7f27a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.260+0000 7f278ffff640 1 -- 172.21.15.28:0/3951982791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a0004b20 con 0x7f27a810c0d0 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.260+0000 7f278ffff640 1 -- 172.21.15.28:0/3951982791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27a0004cc0 con 0x7f27a810c0d0 2024-09-15T21:58:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.260+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a819b410 con 0x7f27a810c0d0 2024-09-15T21:58:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.261+0000 7f278ffff640 1 -- 172.21.15.28:0/3951982791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a0010790 con 0x7f27a810c0d0 2024-09-15T21:58:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.261+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a81a2ca0 con 0x7f27a810c0d0 2024-09-15T21:58:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.264+0000 7f27af9fd740 1 Processor -- start 2024-09-15T21:58:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f278ffff640 1 -- 172.21.15.28:0/3951982791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f27a0010930 con 0x7f27a810c0d0 2024-09-15T21:58:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f27af9fd740 1 -- start start 2024-09-15T21:58:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d700af0 0x55f91d700ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d701410 0x55f91d70b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d70e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f27af9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f91d6652f0 con 0x55f91d701410 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f27af9fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f91d60d270 con 0x55f91d70bcb0 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.265+0000 7f27af9fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f91d6612b0 con 0x55f91d700af0 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d701410 0x55f91d70b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d701410 0x55f91d70b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35434/0 (socket says 172.21.15.28:35434) 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 -- 172.21.15.28:0/22977682 learned_addr learned my addr 172.21.15.28:0/22977682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a659c640 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d70e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 -- 172.21.15.28:0/22977682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d700af0 msgr2=0x55f91d700ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a659c640 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d700af0 0x55f91d700ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f278ffff640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27900a3760 0x7f27900a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d700af0 0x55f91d700ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 -- 172.21.15.28:0/22977682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 msgr2=0x55f91d70e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d70e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 -- 172.21.15.28:0/22977682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f91d60c340 con 0x55f91d701410 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a659c640 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d700af0 0x55f91d700ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f278ffff640 1 -- 172.21.15.28:0/3951982791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f27a00d0970 con 0x7f27a810c0d0 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a759e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27900a3760 0x7f27900a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.266+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d701410 0x55f91d70b770 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f279800da40 tx=0x7f2798005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50454a5ba915a9e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.267+0000 7f278d7fa640 1 -- 172.21.15.28:0/22977682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2798004020 con 0x55f91d701410 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.267+0000 7f278d7fa640 1 -- 172.21.15.28:0/22977682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27980041c0 con 0x55f91d701410 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.267+0000 7f278d7fa640 1 -- 172.21.15.28:0/22977682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27980196b0 con 0x55f91d701410 2024-09-15T21:58:24.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 -- 172.21.15.28:0/22977682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d701410 msgr2=0x55f91d70b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27a759e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27900a3760 0x7f27900a5c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27a00cdb80 tx=0x7f27a0013000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d701410 0x55f91d70b770 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f279800da40 tx=0x7f2798005f90 comp rx=0 tx=0).stop 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 -- 172.21.15.28:0/22977682 shutdown_connections 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d70e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d701410 0x55f91d70b770 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 --2- 172.21.15.28:0/22977682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d700af0 0x55f91d700ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 -- 172.21.15.28:0/22977682 >> 172.21.15.28:0/22977682 conn(0x55f91d6fd060 msgr2=0x55f91d6fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 -- 172.21.15.28:0/22977682 shutdown_connections 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 -- 172.21.15.28:0/22977682 wait complete. 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 -- start start 2024-09-15T21:58:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.268+0000 7f27af9fd740 1 Processor -- start 2024-09-15T21:58:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d700af0 0x55f91d708bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d7090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27af9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f91d70e880 con 0x55f91d700af0 2024-09-15T21:58:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27af9fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f91d5ee290 con 0x55f91d701410 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27af9fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f91d70ea00 con 0x55f91d70bcb0 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27af9fd740 10 client.0 _ll_register_callbacks cb 0x55f91d61f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d700af0 0x55f91d708bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d700af0 0x55f91d708bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35442/0 (socket says 172.21.15.28:35442) 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 -- 172.21.15.28:0/3937632365 learned_addr learned my addr 172.21.15.28:0/3937632365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d7090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 -- 172.21.15.28:0/3937632365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 msgr2=0x55f91d709630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 -- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 msgr2=0x55f91d7090f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d7090f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.269+0000 7f27a659c640 1 -- 172.21.15.28:0/3937632365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f91d378210 con 0x55f91d700af0 2024-09-15T21:58:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f27a659c640 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d700af0 0x55f91d708bb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f279c033000 tx=0x7f279c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7917e31eba22486b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f278aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f278aff5640 1 -- 172.21.15.28:0/3937632365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c041070 con 0x55f91d700af0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f278aff5640 1 -- 172.21.15.28:0/3937632365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f279c003a00 con 0x55f91d700af0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27980041c0 con 0x55f91d700af0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f278aff5640 1 -- 172.21.15.28:0/3937632365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f279c035040 con 0x55f91d700af0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f27af9fd740 10 client.15912 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f91d444620 con 0x55f91d700af0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.270+0000 7f278aff5640 1 -- 172.21.15.28:0/3937632365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27980041c0 con 0x55f91d700af0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f278aff5640 1 -- 172.21.15.28:0/3937632365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f279c03a050 con 0x55f91d700af0 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f27af9fd740 10 client.15912 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f27af9fd740 10 client.15912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f27af9fd740 20 client.15912 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f27af9fd740 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d71bf50 0x55f91d71e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f27af9fd740 4 client.15912 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f91d672a20 con 0x55f91d71bf50 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.271+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d71bf50 0x55f91d71e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.272+0000 7f278aff5640 10 client.15912 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.272+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d71bf50 0x55f91d71e390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13a2cc31fb8fc47 server_cookie=29393d2eb4db2886 in_seq=0 out_seq=0 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.272+0000 7f278aff5640 1 -- 172.21.15.28:0/3937632365 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f91d672a20 con 0x55f91d71bf50 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.272+0000 7f278aff5640 10 client.15912 handle_command_reply: tid=0 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.272+0000 7f27af9fd740 1 client.15912 shutdown 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d700af0 msgr2=0x55f91d708bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d700af0 0x55f91d708bb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f279c033000 tx=0x7f279c004270 comp rx=0 tx=0).stop 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 shutdown_connections 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d71bf50 0x55f91d71e390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d7090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 --2- 172.21.15.28:0/3937632365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d700af0 0x55f91d708bb0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 >> 172.21.15.28:0/3937632365 conn(0x55f91d6fd060 msgr2=0x55f91d702d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f278aff5640 0 client.15912 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 shutdown_connections 2024-09-15T21:58:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 1 -- 172.21.15.28:0/3937632365 wait complete. 2024-09-15T21:58:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.273+0000 7f27af9fd740 20 client.15912 trim_cache size 0 max 16384 2024-09-15T21:58:24.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.301+0000 7f27af9fd740 1 Processor -- start 2024-09-15T21:58:24.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.301+0000 7f27af9fd740 1 -- start start 2024-09-15T21:58:24.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d703560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d7240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27af9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81b5510 con 0x55f91d723c60 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27af9fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81b7050 con 0x55f91d701410 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27af9fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81b5be0 con 0x55f91d70bcb0 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d7240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:39508/0 (socket says 172.21.15.28:39508) 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 -- 172.21.15.28:0/1710627712 learned_addr learned my addr 172.21.15.28:0/1710627712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d7240c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35450/0 (socket says 172.21.15.28:35450) 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a659c640 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d703560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 -- 172.21.15.28:0/1710627712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 msgr2=0x55f91d703560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d703560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 -- 172.21.15.28:0/1710627712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 msgr2=0x55f91d7240c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:24.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d7240c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.302+0000 7f27a6d9d640 1 -- 172.21.15.28:0/1710627712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f91d700920 con 0x55f91d70bcb0 2024-09-15T21:58:24.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.303+0000 7f27a659c640 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d703560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:24.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.303+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f279800da10 tx=0x7f2798003e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c0211b767aac665 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:24.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.303+0000 7f278cff9640 1 -- 172.21.15.28:0/1710627712 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279802f070 con 0x55f91d70bcb0 2024-09-15T21:58:24.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.303+0000 7f278cff9640 1 -- 172.21.15.28:0/1710627712 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f279800b320 con 0x55f91d70bcb0 2024-09-15T21:58:24.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.303+0000 7f278cff9640 1 -- 172.21.15.28:0/1710627712 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27980052a0 con 0x55f91d70bcb0 2024-09-15T21:58:24.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 -- 172.21.15.28:0/1710627712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 msgr2=0x55f91d709460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f279800da10 tx=0x7f2798003e30 comp rx=0 tx=0).stop 2024-09-15T21:58:24.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 -- 172.21.15.28:0/1710627712 shutdown_connections 2024-09-15T21:58:24.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d7240c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d70bcb0 0x55f91d709460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 --2- 172.21.15.28:0/1710627712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d701410 0x55f91d703560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 -- 172.21.15.28:0/1710627712 >> 172.21.15.28:0/1710627712 conn(0x55f91d6fd060 msgr2=0x55f91d70ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 -- 172.21.15.28:0/1710627712 shutdown_connections 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 -- 172.21.15.28:0/1710627712 wait complete. 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.304+0000 7f27af9fd740 1 -- start start 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27af9fd740 1 Processor -- start 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d701410 0x55f91d705b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d7060d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27af9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d706610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27af9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81b6980 con 0x55f91d723c60 2024-09-15T21:58:24.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27af9fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81b62b0 con 0x55f91d70bcb0 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27af9fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81b34f0 con 0x55f91d701410 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d706610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d706610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35454/0 (socket says 172.21.15.28:35454) 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 -- 172.21.15.28:0/4183041686 learned_addr learned my addr 172.21.15.28:0/4183041686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d7060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d701410 0x55f91d705b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 -- 172.21.15.28:0/4183041686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d701410 msgr2=0x55f91d705b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d701410 0x55f91d705b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 -- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 msgr2=0x55f91d7060d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d7060d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.311 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.305+0000 7f27a659c640 1 -- 172.21.15.28:0/4183041686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f91d785c30 con 0x55f91d723c60 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f27a659c640 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d706610 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f279c031550 tx=0x7f279c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b5e750bdf71b83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f278aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f278aff5640 1 -- 172.21.15.28:0/4183041686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c041040 con 0x55f91d723c60 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f27af9fd740 10 client.0 _ll_register_callbacks cb 0x7f27a806a900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f278aff5640 1 -- 172.21.15.28:0/4183041686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f279c003a00 con 0x55f91d723c60 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f279800b320 con 0x55f91d723c60 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f278aff5640 1 -- 172.21.15.28:0/4183041686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f279c066050 con 0x55f91d723c60 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f27af9fd740 10 client.15918 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.306+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55f91d786d20 con 0x55f91d723c60 2024-09-15T21:58:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f278aff5640 1 -- 172.21.15.28:0/4183041686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f279c03a040 con 0x55f91d723c60 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27af9fd740 10 client.15918 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27af9fd740 10 client.15918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27af9fd740 20 client.15918 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27af9fd740 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d787340 0x55f91d7897c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27af9fd740 4 client.15918 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f91d66cd70 con 0x55f91d787340 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d787340 0x55f91d7897c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f278aff5640 10 client.15918 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.307+0000 7f27a6d9d640 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d787340 0x55f91d7897c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aaf07a1e780c9e9 server_cookie=d2bb5ca0ca929230 in_seq=0 out_seq=0 2024-09-15T21:58:24.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f278aff5640 1 -- 172.21.15.28:0/4183041686 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f91d66cd70 con 0x55f91d787340 2024-09-15T21:58:24.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f278aff5640 10 client.15918 handle_command_reply: tid=0 2024-09-15T21:58:24.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 client.15918 shutdown 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 msgr2=0x55f91d706610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d706610 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f279c031550 tx=0x7f279c000bc0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 shutdown_connections 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55f91d787340 0x55f91d7897c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f91d723c60 0x55f91d706610 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f91d70bcb0 0x55f91d7060d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 --2- 172.21.15.28:0/4183041686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f91d701410 0x55f91d705b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.315+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 >> 172.21.15.28:0/4183041686 conn(0x55f91d6fd060 msgr2=0x55f91d723470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f278aff5640 0 client.15918 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 shutdown_connections 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f27af9fd740 1 -- 172.21.15.28:0/4183041686 wait complete. 2024-09-15T21:58:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f27af9fd740 20 client.15918 trim_cache size 0 max 16384 2024-09-15T21:58:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T21:58:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T21:58:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T21:58:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27900a3760 msgr2=0x7f27900a5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f27ad01e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27900a3760 0x7f27900a5c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27a00cdb80 tx=0x7f27a0013000 comp rx=0 tx=0).stop 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a810c0d0 msgr2=0x7f27a819a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.316+0000 7f27ad01e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a810c0d0 0x7f27a819a9f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f27a000eb10 tx=0x7f27a000c2d0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 shutdown_connections 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f27900a3760 0x7f27900a5c20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27a810c0d0 0x7f27a819a9f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27a81035a0 0x7f27a8196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 --2- 172.21.15.28:0/3951982791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27a8102c80 0x7f27a8196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 >> 172.21.15.28:0/3951982791 conn(0x7f27a8077990 msgr2=0x7f27a80fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:24.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 shutdown_connections 2024-09-15T21:58:24.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:24.317+0000 7f27ad01e640 1 -- 172.21.15.28:0/3951982791 wait complete. 2024-09-15T21:58:25.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:25 smithi203 ceph-mon[35546]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-09-15T21:58:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:25 smithi174 ceph-mon[35244]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-09-15T21:58:25.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:25 smithi028 ceph-mon[30969]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 9 op/s 2024-09-15T21:58:27.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:27 smithi203 ceph-mon[35546]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-09-15T21:58:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:27 smithi174 ceph-mon[35244]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-09-15T21:58:27.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:27 smithi028 ceph-mon[30969]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 2.0 KiB/s wr, 15 op/s 2024-09-15T21:58:29.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:29 smithi203 ceph-mon[35546]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T21:58:29.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:29 smithi174 ceph-mon[35244]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T21:58:29.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:29 smithi028 ceph-mon[30969]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T21:58:30.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:30 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:58:30.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:30 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:58:30.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:30 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:58:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T21:58:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:58:31.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:58:31.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T21:58:31.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:58:31.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:58:31.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:58:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:58:31.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:58:33.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:33 smithi174 ceph-mon[35244]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-09-15T21:58:33.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:33 smithi028 ceph-mon[30969]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-09-15T21:58:33.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:33 smithi203 ceph-mon[35546]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-09-15T21:58:35.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:35 smithi174 ceph-mon[35244]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-15T21:58:35.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:35 smithi028 ceph-mon[30969]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-15T21:58:35.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:35 smithi203 ceph-mon[35546]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-15T21:58:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 -- 172.21.15.28:0/1083593481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead0106b70 msgr2=0x7fead0106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 --2- 172.21.15.28:0/1083593481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead0106b70 0x7fead0106f50 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac002a410 comp rx=0 tx=0).stop 2024-09-15T21:58:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 -- 172.21.15.28:0/1083593481 shutdown_connections 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 --2- 172.21.15.28:0/1083593481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fead01020c0 0x7fead010d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 --2- 172.21.15.28:0/1083593481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fead0101740 0x7fead0101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 --2- 172.21.15.28:0/1083593481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead0106b70 0x7fead0106f50 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 -- 172.21.15.28:0/1083593481 >> 172.21.15.28:0/1083593481 conn(0x7fead0075ef0 msgr2=0x7fead0076300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 -- 172.21.15.28:0/1083593481 shutdown_connections 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.454+0000 7fead5427640 1 -- 172.21.15.28:0/1083593481 wait complete. 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 Processor -- start 2024-09-15T21:58:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 -- start start 2024-09-15T21:58:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fead0101740 0x7fead0198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead01020c0 0x7fead01988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fead0106b70 0x7fead019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7feace7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead01020c0 0x7fead01988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7feace7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead01020c0 0x7fead01988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35312/0 (socket says 172.21.15.28:35312) 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fead0112020 con 0x7fead01020c0 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7feace7fc640 1 -- 172.21.15.28:0/11516501 learned_addr learned my addr 172.21.15.28:0/11516501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fead0111ea0 con 0x7fead0101740 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.455+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fead01121a0 con 0x7fead0106b70 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feaceffd640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fead0101740 0x7fead0198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feacf7fe640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fead0106b70 0x7fead019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feaceffd640 1 -- 172.21.15.28:0/11516501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fead0106b70 msgr2=0x7fead019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feaceffd640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fead0106b70 0x7fead019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feaceffd640 1 -- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead01020c0 msgr2=0x7fead01988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feaceffd640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead01020c0 0x7fead01988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feaceffd640 1 -- 172.21.15.28:0/11516501 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead019d360 con 0x7fead0101740 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feace7fc640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead01020c0 0x7fead01988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feaceffd640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fead0101740 0x7fead0198370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feac002a930 tx=0x7feac0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.456+0000 7feab7fff640 1 -- 172.21.15.28:0/11516501 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac003f070 con 0x7fead0101740 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.457+0000 7feab7fff640 1 -- 172.21.15.28:0/11516501 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac0031c90 con 0x7fead0101740 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.457+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead019d5f0 con 0x7fead0101740 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.457+0000 7feab7fff640 1 -- 172.21.15.28:0/11516501 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0035030 con 0x7fead0101740 2024-09-15T21:58:37.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.457+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead01a4ee0 con 0x7fead0101740 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7feab7fff640 1 -- 172.21.15.28:0/11516501 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7feac0042070 con 0x7fead0101740 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 Processor -- start 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 -- start start 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044d4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587044d70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55870442e2f0 con 0x5587044ca410 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5587043d6270 con 0x5587044c9af0 2024-09-15T21:58:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.460+0000 7fead81c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55870442a2b0 con 0x5587044d4cb0 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044d4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044d4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35318/0 (socket says 172.21.15.28:35318) 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 -- 172.21.15.28:0/2172132067 learned_addr learned my addr 172.21.15.28:0/2172132067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feacf7fe640 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feace7fc640 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587044d70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 -- 172.21.15.28:0/2172132067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 msgr2=0x5587044d70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587044d70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 -- 172.21.15.28:0/2172132067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 msgr2=0x5587044c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 -- 172.21.15.28:0/2172132067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587043d5340 con 0x5587044ca410 2024-09-15T21:58:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.461+0000 7feaceffd640 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044d4770 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7feac00edb70 tx=0x7feac00f4c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e357b9819b9c1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.462+0000 7feab57fa640 1 -- 172.21.15.28:0/2172132067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac00fb6e0 con 0x5587044ca410 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.462+0000 7feab57fa640 1 -- 172.21.15.28:0/2172132067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac00f0e50 con 0x5587044ca410 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.462+0000 7feab57fa640 1 -- 172.21.15.28:0/2172132067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0100d60 con 0x5587044ca410 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.462+0000 7feab7fff640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feab80a3610 0x7feab80a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.462+0000 7feacf7fe640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feab80a3610 0x7feab80a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.462+0000 7feab7fff640 1 -- 172.21.15.28:0/11516501 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7feac00f0a20 con 0x7fead0101740 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 -- 172.21.15.28:0/2172132067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 msgr2=0x5587044d4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7feacf7fe640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feab80a3610 0x7feab80a5ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feac8005fd0 tx=0x7feac8009600 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044d4770 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7feac00edb70 tx=0x7feac00f4c40 comp rx=0 tx=0).stop 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 -- 172.21.15.28:0/2172132067 shutdown_connections 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587044d70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044d4770 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 --2- 172.21.15.28:0/2172132067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 -- 172.21.15.28:0/2172132067 >> 172.21.15.28:0/2172132067 conn(0x5587044c6060 msgr2=0x5587044c6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 -- 172.21.15.28:0/2172132067 shutdown_connections 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 -- 172.21.15.28:0/2172132067 wait complete. 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.463+0000 7fead81c0740 1 -- start start 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 1 Processor -- start 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044d2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587044dba50 con 0x5587044d4cb0 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5587043b7290 con 0x5587044c9af0 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5587044dbbd0 con 0x5587044ca410 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7fead81c0740 10 client.0 _ll_register_callbacks cb 0x5587043e8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7feacf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7feaceffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7feacf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044d2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35328/0 (socket says 172.21.15.28:35328) 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7feacf7fe640 1 -- 172.21.15.28:0/2150959716 learned_addr learned my addr 172.21.15.28:0/2150959716 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.464+0000 7feace7fc640 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feaceffd640 1 -- 172.21.15.28:0/2150959716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 msgr2=0x5587044d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feaceffd640 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feaceffd640 1 -- 172.21.15.28:0/2150959716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 msgr2=0x5587044d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feaceffd640 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044d2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feaceffd640 1 -- 172.21.15.28:0/2150959716 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558704141210 con 0x5587044c9af0 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feacf7fe640 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feaceffd640 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d1b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac003b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c979db8198ac8e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feab2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feab2ff5640 1 -- 172.21.15.28:0/2150959716 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0049070 con 0x5587044c9af0 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feab2ff5640 1 -- 172.21.15.28:0/2150959716 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7feac003b4f0 con 0x5587044c9af0 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feac00f0e50 con 0x5587044c9af0 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feab2ff5640 1 -- 172.21.15.28:0/2150959716 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feac00b8ab0 con 0x5587044c9af0 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7fead81c0740 10 client.25538 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55870420d620 con 0x5587044c9af0 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.465+0000 7feab2ff5640 1 -- 172.21.15.28:0/2150959716 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac0100410 con 0x5587044c9af0 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7feab2ff5640 1 -- 172.21.15.28:0/2150959716 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feac00bca70 con 0x5587044c9af0 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7fead81c0740 10 client.25538 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7fead81c0740 10 client.25538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7fead81c0740 20 client.25538 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7fead81c0740 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x5587044eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7fead81c0740 4 client.25538 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feac00f0e50 con 0x5587044e8b90 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7feace7fc640 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x5587044eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7feab2ff5640 10 client.25538 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.466+0000 7feace7fc640 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x5587044eaf80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d27f94be22260b7 server_cookie=1cf5faa23e5bf9d5 in_seq=0 out_seq=0 2024-09-15T21:58:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7feab2ff5640 1 -- 172.21.15.28:0/2150959716 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7feac00f0e50 con 0x5587044e8b90 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7feab2ff5640 10 client.25538 handle_command_reply: tid=0 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 client.25538 shutdown 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 msgr2=0x5587044d1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d1b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac003b460 comp rx=0 tx=0).stop 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 shutdown_connections 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x5587044eaf80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7feab2ff5640 0 client.25538 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 --2- 172.21.15.28:0/2150959716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d1b80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.467+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 >> 172.21.15.28:0/2150959716 conn(0x5587044c6060 msgr2=0x5587044cbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.468+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 shutdown_connections 2024-09-15T21:58:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.468+0000 7fead81c0740 1 -- 172.21.15.28:0/2150959716 wait complete. 2024-09-15T21:58:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.468+0000 7fead81c0740 20 client.25538 trim_cache size 0 max 16384 2024-09-15T21:58:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 Processor -- start 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 -- start start 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fead01acbf0 con 0x5587044d4cb0 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fead01ae730 con 0x5587044c9af0 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.497+0000 7fead81c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fead01ad2c0 con 0x5587044ca410 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044f0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35334/0 (socket says 172.21.15.28:35334) 2024-09-15T21:58:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 -- 172.21.15.28:0/3657347693 learned_addr learned my addr 172.21.15.28:0/3657347693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feace7fc640 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feaceffd640 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 -- 172.21.15.28:0/3657347693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 msgr2=0x5587044f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 -- 172.21.15.28:0/3657347693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 msgr2=0x5587044d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 -- 172.21.15.28:0/3657347693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587044c9870 con 0x5587044d4cb0 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feace7fc640 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feacf7fe640 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044f0b20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7feac8005c10 tx=0x7feac8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c8f094df359852 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.498+0000 7feab4ff9640 1 -- 172.21.15.28:0/3657347693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8004b20 con 0x5587044d4cb0 2024-09-15T21:58:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.499+0000 7feab4ff9640 1 -- 172.21.15.28:0/3657347693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac8003e50 con 0x5587044d4cb0 2024-09-15T21:58:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.499+0000 7feab4ff9640 1 -- 172.21.15.28:0/3657347693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac80205e0 con 0x5587044d4cb0 2024-09-15T21:58:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.499+0000 7fead81c0740 1 -- 172.21.15.28:0/3657347693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 msgr2=0x5587044f0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044f0b20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7feac8005c10 tx=0x7feac8016000 comp rx=0 tx=0).stop 2024-09-15T21:58:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 -- 172.21.15.28:0/3657347693 shutdown_connections 2024-09-15T21:58:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044d4cb0 0x5587044f0b20 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044ca410 0x5587044f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 --2- 172.21.15.28:0/3657347693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 -- 172.21.15.28:0/3657347693 >> 172.21.15.28:0/3657347693 conn(0x5587044c6060 msgr2=0x5587044d33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 -- 172.21.15.28:0/3657347693 shutdown_connections 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 -- 172.21.15.28:0/3657347693 wait complete. 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 -- start start 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.500+0000 7fead81c0740 1 Processor -- start 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7fead81c0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044dcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7fead81c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044dd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7fead81c0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587045525b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7fead81c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fead01ae060 con 0x5587044ca410 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7fead81c0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fead01ad990 con 0x5587044c9af0 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7fead81c0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fead01aabd0 con 0x5587044d4cb0 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7fead81c0740 10 client.0 _ll_register_callbacks cb 0x7feac8009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7feace7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044dd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7feace7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044dd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35350/0 (socket says 172.21.15.28:35350) 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7feaceffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044dcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7feace7fc640 1 -- 172.21.15.28:0/1988795745 learned_addr learned my addr 172.21.15.28:0/1988795745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.501+0000 7feacf7fe640 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587045525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feaceffd640 1 -- 172.21.15.28:0/1988795745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 msgr2=0x5587045525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feaceffd640 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587045525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feaceffd640 1 -- 172.21.15.28:0/1988795745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 msgr2=0x5587044dd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feaceffd640 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044dd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feaceffd640 1 -- 172.21.15.28:0/1988795745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587044ddd50 con 0x5587044c9af0 2024-09-15T21:58:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feace7fc640 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044dd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feaceffd640 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044dcf10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feac00b91e0 tx=0x7feac0003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e788f18a013998c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feab2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feab2ff5640 1 -- 172.21.15.28:0/1988795745 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0049070 con 0x5587044c9af0 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7feab2ff5640 1 -- 172.21.15.28:0/1988795745 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7feac00bca70 con 0x5587044c9af0 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.502+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feac8003e50 con 0x5587044c9af0 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7feab2ff5640 1 -- 172.21.15.28:0/1988795745 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feac00bc170 con 0x5587044c9af0 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 10 client.25544 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5587044ddd50 con 0x5587044c9af0 2024-09-15T21:58:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7feab2ff5640 1 -- 172.21.15.28:0/1988795745 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac8003e50 con 0x5587044c9af0 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7feab2ff5640 1 -- 172.21.15.28:0/1988795745 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feac003b4f0 con 0x5587044c9af0 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 10 client.25544 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 10 client.25544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 20 client.25544 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x558704555fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 4 client.25544 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55870443ba20 con 0x5587044e8b90 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.503+0000 7feace7fc640 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x558704555fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.504+0000 7feab2ff5640 10 client.25544 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.504+0000 7feace7fc640 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x558704555fa0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c41fc3b97259d9c9 server_cookie=591dad3272364a1b in_seq=0 out_seq=0 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.507+0000 7feab2ff5640 1 -- 172.21.15.28:0/1988795745 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55870443ba20 con 0x5587044e8b90 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.507+0000 7feab2ff5640 10 client.25544 handle_command_reply: tid=0 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 client.25544 shutdown 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 msgr2=0x5587044dcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044dcf10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feac00b91e0 tx=0x7feac0003f90 comp rx=0 tx=0).stop 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 shutdown_connections 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5587044e8b90 0x558704555fa0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587044d4cb0 0x5587045525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587044ca410 0x5587044dd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 --2- 172.21.15.28:0/1988795745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5587044c9af0 0x5587044dcf10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 >> 172.21.15.28:0/1988795745 conn(0x5587044c6060 msgr2=0x5587044c49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7feab2ff5640 0 client.25544 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 shutdown_connections 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 1 -- 172.21.15.28:0/1988795745 wait complete. 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead81c0740 20 client.25544 trim_cache size 0 max 16384 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T21:58:37.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T21:58:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T21:58:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feab80a3610 msgr2=0x7feab80a5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead5427640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feab80a3610 0x7feab80a5ad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feac8005fd0 tx=0x7feac8009600 comp rx=0 tx=0).stop 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fead0101740 msgr2=0x7fead0198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.508+0000 7fead5427640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fead0101740 0x7fead0198370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feac002a930 tx=0x7feac0003b90 comp rx=0 tx=0).stop 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 shutdown_connections 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feab80a3610 0x7feab80a5ad0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fead0106b70 0x7fead019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fead01020c0 0x7fead01988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 --2- 172.21.15.28:0/11516501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fead0101740 0x7fead0198370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 >> 172.21.15.28:0/11516501 conn(0x7fead0075ef0 msgr2=0x7fead00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 shutdown_connections 2024-09-15T21:58:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:37.509+0000 7fead5427640 1 -- 172.21.15.28:0/11516501 wait complete. 2024-09-15T21:58:37.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:58:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:37 smithi174 ceph-mon[35244]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-09-15T21:58:37.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:37 smithi028 ceph-mon[30969]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-09-15T21:58:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:37 smithi203 ceph-mon[35546]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-09-15T21:58:37.994 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 -- 172.21.15.28:0/641105741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754103760 msgr2=0x7f8754107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 --2- 172.21.15.28:0/641105741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754103760 0x7f8754107ba0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8740009930 tx=0x7f874002a420 comp rx=0 tx=0).stop 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 -- 172.21.15.28:0/641105741 shutdown_connections 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 --2- 172.21.15.28:0/641105741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87541082e0 0x7f875410ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 --2- 172.21.15.28:0/641105741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754103760 0x7f8754107ba0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 --2- 172.21.15.28:0/641105741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8754102db0 0x7f8754103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 -- 172.21.15.28:0/641105741 >> 172.21.15.28:0/641105741 conn(0x7f87540fc7d0 msgr2=0x7f87540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 -- 172.21.15.28:0/641105741 shutdown_connections 2024-09-15T21:58:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.563+0000 7f875afbb640 1 -- 172.21.15.28:0/641105741 wait complete. 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 Processor -- start 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 -- start start 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754102db0 0x7f875419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8754103760 0x7f875419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87541082e0 0x7f875410dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87541120c0 con 0x7f8754102db0 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8754111f40 con 0x7f87541082e0 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f875afbb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8754112240 con 0x7f8754103760 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f8758d57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754102db0 0x7f875419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f8758d57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754102db0 0x7f875419f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35374/0 (socket says 172.21.15.28:35374) 2024-09-15T21:58:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f8758d57640 1 -- 172.21.15.28:0/930584774 learned_addr learned my addr 172.21.15.28:0/930584774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.564+0000 7f8753fff640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8754103760 0x7f875419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8758d57640 1 -- 172.21.15.28:0/930584774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8754103760 msgr2=0x7f875419f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8758d57640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8754103760 0x7f875419f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8759558640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87541082e0 0x7f875410dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8758d57640 1 -- 172.21.15.28:0/930584774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87541082e0 msgr2=0x7f875410dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8758d57640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87541082e0 0x7f875410dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8758d57640 1 -- 172.21.15.28:0/930584774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f875410e630 con 0x7f8754102db0 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8759558640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87541082e0 0x7f875410dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8758d57640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754102db0 0x7f875419f0c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f874800c920 tx=0x7f874800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f8751ffb640 1 -- 172.21.15.28:0/930584774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748007c70 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.565+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875410e8c0 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.566+0000 7f8751ffb640 1 -- 172.21.15.28:0/930584774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8748007e10 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.566+0000 7f8751ffb640 1 -- 172.21.15.28:0/930584774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8748018890 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.566+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f875419add0 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.567+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f875406a7d0 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.569+0000 7f8751ffb640 1 -- 172.21.15.28:0/930584774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f8748018a70 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.570+0000 7f8751ffb640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87440a3560 0x7f87440a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.570+0000 7f8753fff640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87440a3560 0x7f87440a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.570+0000 7f8751ffb640 1 -- 172.21.15.28:0/930584774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f87480d04a0 con 0x7f8754102db0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.571+0000 7f8753fff640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87440a3560 0x7f87440a5a20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f874002a940 tx=0x7f8740005a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.575+0000 7f8751ffb640 1 -- 172.21.15.28:0/930584774 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8748014030 con 0x7f8754102db0 2024-09-15T21:58:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.788+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87541976a0 con 0x7f8754102db0 2024-09-15T21:58:38.792 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:58:38.792 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"btime":"2024-09-15T21:57:18:724983+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:57:18.674828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T21:58:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.789+0000 7f8751ffb640 1 -- 172.21.15.28:0/930584774 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f87480bae20 con 0x7f8754102db0 2024-09-15T21:58:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87440a3560 msgr2=0x7f87440a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87440a3560 0x7f87440a5a20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f874002a940 tx=0x7f8740005a40 comp rx=0 tx=0).stop 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754102db0 msgr2=0x7f875419f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754102db0 0x7f875419f0c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f874800c920 tx=0x7f874800cde0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 shutdown_connections 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87440a3560 0x7f87440a5a20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87541082e0 0x7f875410dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8754103760 0x7f875419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 --2- 172.21.15.28:0/930584774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8754102db0 0x7f875419f0c0 secure :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f874800c920 tx=0x7f874800cde0 comp rx=0 tx=0).stop 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.792+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 >> 172.21.15.28:0/930584774 conn(0x7f87540fc7d0 msgr2=0x7f87541015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.793+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 shutdown_connections 2024-09-15T21:58:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:38.793+0000 7f875afbb640 1 -- 172.21.15.28:0/930584774 wait complete. 2024-09-15T21:58:39.456 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id b0814eb3 --release --await' 2024-09-15T21:58:39.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id b0814eb3 --release --await 2024-09-15T21:58:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:39 smithi028 ceph-mon[30969]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:39.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:39 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/930584774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:39 smithi174 ceph-mon[35244]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:39 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/930584774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:39.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:39 smithi203 ceph-mon[35546]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:39.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:39 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/930584774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:40.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 -- 172.21.15.28:0/298091899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100aa0 msgr2=0x7f4a6010ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 --2- 172.21.15.28:0/298091899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100aa0 0x7f4a6010ea20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009900 tx=0x7f4a5802a410 comp rx=0 tx=0).stop 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 -- 172.21.15.28:0/298091899 shutdown_connections 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 --2- 172.21.15.28:0/298091899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100aa0 0x7f4a6010ea20 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 --2- 172.21.15.28:0/298091899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a60100100 0x7f4a60100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 --2- 172.21.15.28:0/298091899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60106b80 0x7f4a60106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 -- 172.21.15.28:0/298091899 >> 172.21.15.28:0/298091899 conn(0x7f4a60077230 msgr2=0x7f4a600fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 -- 172.21.15.28:0/298091899 shutdown_connections 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.310+0000 7f4a66cce640 1 -- 172.21.15.28:0/298091899 wait complete. 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 Processor -- start 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 -- start start 2024-09-15T21:58:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100100 0x7f4a60198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60100aa0 0x7f4a60198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a60106b80 0x7f4a6019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a60112050 con 0x7f4a60100100 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a60111ed0 con 0x7f4a60106b80 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a66cce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601121d0 con 0x7f4a60100aa0 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.311+0000 7f4a64a6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100100 0x7f4a60198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100100 0x7f4a60198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35386/0 (socket says 172.21.15.28:35386) 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 -- 172.21.15.28:0/1357250440 learned_addr learned my addr 172.21.15.28:0/1357250440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60100aa0 0x7f4a60198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a6526b640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a60106b80 0x7f4a6019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 -- 172.21.15.28:0/1357250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60100aa0 msgr2=0x7f4a60198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60100aa0 0x7f4a60198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 -- 172.21.15.28:0/1357250440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a60106b80 msgr2=0x7f4a6019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a60106b80 0x7f4a6019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 -- 172.21.15.28:0/1357250440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6019d4e0 con 0x7f4a60100100 2024-09-15T21:58:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60100aa0 0x7f4a60198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a6526b640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a60106b80 0x7f4a6019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100100 0x7f4a60198530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800d770 tx=0x7f4a4800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a5dffb640 1 -- 172.21.15.28:0/1357250440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48017070 con 0x7f4a60100100 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.312+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6019d7d0 con 0x7f4a60100100 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.313+0000 7f4a5dffb640 1 -- 172.21.15.28:0/1357250440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a48005500 con 0x7f4a60100100 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.313+0000 7f4a5dffb640 1 -- 172.21.15.28:0/1357250440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48004e00 con 0x7f4a60100100 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.313+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a601a50b0 con 0x7f4a60100100 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 Processor -- start 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 -- start start 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6060 0x55c4738c6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6980 0x55c4738d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c47382b620 con 0x55c4738d1240 2024-09-15T21:58:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c4737d2e50 con 0x55c4738c6060 2024-09-15T21:58:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a690a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c4738275e0 con 0x55c4738c6980 2024-09-15T21:58:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35396/0 (socket says 172.21.15.28:35396) 2024-09-15T21:58:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a5ffff640 1 -- 172.21.15.28:0/3639784243 learned_addr learned my addr 172.21.15.28:0/3639784243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a6526b640 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6060 0x55c4738c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.316+0000 7f4a5dffb640 1 -- 172.21.15.28:0/1357250440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7f4a48004fa0 con 0x7f4a60100100 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a5ffff640 1 -- 172.21.15.28:0/3639784243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6980 msgr2=0x55c4738d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6980 0x55c4738d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a5ffff640 1 -- 172.21.15.28:0/3639784243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6060 msgr2=0x55c4738c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6060 0x55c4738c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a5ffff640 1 -- 172.21.15.28:0/3639784243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4737c3d40 con 0x55c4738d1240 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d3630 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000f410 tx=0x7f4a5000f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fc019363aa5d2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a56ffd640 1 -- 172.21.15.28:0/3639784243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50004d20 con 0x55c4738d1240 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a56ffd640 1 -- 172.21.15.28:0/3639784243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a500119e0 con 0x55c4738d1240 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.317+0000 7f4a56ffd640 1 -- 172.21.15.28:0/3639784243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500056f0 con 0x55c4738d1240 2024-09-15T21:58:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.318+0000 7f4a5dffb640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4a4c0a35e0 0x7f4a4c0a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.318+0000 7f4a6526b640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4a4c0a35e0 0x7f4a4c0a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.318+0000 7f4a5dffb640 1 -- 172.21.15.28:0/1357250440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f4a480cfd50 con 0x7f4a60100100 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.318+0000 7f4a690a7740 1 -- 172.21.15.28:0/3639784243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 msgr2=0x55c4738d3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.318+0000 7f4a690a7740 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d3630 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000f410 tx=0x7f4a5000f8d0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 -- 172.21.15.28:0/3639784243 shutdown_connections 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d3630 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6980 0x55c4738d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 --2- 172.21.15.28:0/3639784243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6060 0x55c4738c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 -- 172.21.15.28:0/3639784243 >> 172.21.15.28:0/3639784243 conn(0x55c4738c25f0 msgr2=0x55c4738c29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a6526b640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4a4c0a35e0 0x7f4a4c0a5aa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005940 tx=0x7f4a580058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 -- 172.21.15.28:0/3639784243 shutdown_connections 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 -- 172.21.15.28:0/3639784243 wait complete. 2024-09-15T21:58:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 -- start start 2024-09-15T21:58:40.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.319+0000 7f4a690a7740 1 Processor -- start 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738c6980 0x55c4738ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738d1240 0x55c4738cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a690a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4738d3e10 con 0x55c4738c6980 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a690a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c4737b3f10 con 0x55c4738d1240 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a690a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c4738d3f90 con 0x55c4738c6060 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a690a7740 10 client.0 _ll_register_callbacks cb 0x55c4737d7db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738c6980 0x55c4738ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738c6980 0x55c4738ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35406/0 (socket says 172.21.15.28:35406) 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 -- 172.21.15.28:0/683630548 learned_addr learned my addr 172.21.15.28:0/683630548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a6526b640 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738d1240 0x55c4738cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 -- 172.21.15.28:0/683630548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 msgr2=0x55c4738ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 -- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738d1240 msgr2=0x55c4738cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738d1240 0x55c4738cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.320+0000 7f4a5ffff640 1 -- 172.21.15.28:0/683630548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c47353a580 con 0x55c4738c6980 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738c6980 0x55c4738ce690 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000f3e0 tx=0x7f4a50012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1acd74e0e40b64 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a3ffff640 1 -- 172.21.15.28:0/683630548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50005000 con 0x55c4738c6980 2024-09-15T21:58:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a500119e0 con 0x55c4738c6980 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a3ffff640 1 -- 172.21.15.28:0/683630548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f4a50011710 con 0x55c4738c6980 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a3ffff640 1 -- 172.21.15.28:0/683630548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4a50004a40 con 0x55c4738c6980 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a690a7740 10 client.15954 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c473604960 con 0x55c4738c6980 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.321+0000 7f4a3ffff640 1 -- 172.21.15.28:0/683630548 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a500119e0 con 0x55c4738c6980 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a3ffff640 1 -- 172.21.15.28:0/683630548 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4a500056a0 con 0x55c4738c6980 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a690a7740 10 client.15954 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a690a7740 10 client.15954 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a690a7740 20 client.15954 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a690a7740 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c4738e0d90 0x55c4738e3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a690a7740 4 client.15954 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c473838b90 con 0x55c4738e0d90 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c4738e0d90 0x55c4738e3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c4738e0d90 0x55c4738e3180 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf1d5665c1b3fcb6 server_cookie=695d4c09331e0b in_seq=0 out_seq=0 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.322+0000 7f4a3ffff640 10 client.15954 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a3ffff640 1 -- 172.21.15.28:0/683630548 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c473838b90 con 0x55c4738e0d90 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a3ffff640 10 client.15954 handle_command_reply: tid=0 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 client.15954 shutdown 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738c6980 msgr2=0x55c4738ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738c6980 0x55c4738ce690 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000f3e0 tx=0x7f4a50012dd0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 shutdown_connections 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c4738e0d90 0x55c4738e3180 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738d1240 0x55c4738cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738c6980 0x55c4738ce690 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 --2- 172.21.15.28:0/683630548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 >> 172.21.15.28:0/683630548 conn(0x55c4738c25f0 msgr2=0x55c4738c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.323+0000 7f4a3ffff640 0 client.15954 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.324+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 shutdown_connections 2024-09-15T21:58:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.324+0000 7f4a690a7740 1 -- 172.21.15.28:0/683630548 wait complete. 2024-09-15T21:58:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.324+0000 7f4a690a7740 20 client.15954 trim_cache size 0 max 16384 2024-09-15T21:58:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 Processor -- start 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 -- start start 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738e88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601aef50 con 0x55c4738d1240 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601b0a90 con 0x55c4738c6980 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a690a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601af620 con 0x55c4738c6060 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a6526b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a6526b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738e8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35412/0 (socket says 172.21.15.28:35412) 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a6526b640 1 -- 172.21.15.28:0/2088849794 learned_addr learned my addr 172.21.15.28:0/2088849794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738e88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.341+0000 7f4a6526b640 1 -- 172.21.15.28:0/2088849794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 msgr2=0x55c4738cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a6526b640 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a6526b640 1 -- 172.21.15.28:0/2088849794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 msgr2=0x55c4738e88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a6526b640 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738e88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a6526b640 1 -- 172.21.15.28:0/2088849794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4738c0bb0 con 0x55c4738d1240 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a6526b640 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738e8de0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4a58033000 tx=0x7f4a5803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a96248f041d21da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a567fc640 1 -- 172.21.15.28:0/2088849794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58035040 con 0x55c4738d1240 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a567fc640 1 -- 172.21.15.28:0/2088849794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5804d070 con 0x55c4738d1240 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a567fc640 1 -- 172.21.15.28:0/2088849794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58004c20 con 0x55c4738d1240 2024-09-15T21:58:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.342+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738e88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:40.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 -- 172.21.15.28:0/2088849794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 msgr2=0x55c4738e8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738e8de0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4a58033000 tx=0x7f4a5803e000 comp rx=0 tx=0).stop 2024-09-15T21:58:40.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 -- 172.21.15.28:0/2088849794 shutdown_connections 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738e8de0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738e88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 --2- 172.21.15.28:0/2088849794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 -- 172.21.15.28:0/2088849794 >> 172.21.15.28:0/2088849794 conn(0x55c4738c25f0 msgr2=0x55c4738d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 -- 172.21.15.28:0/2088849794 shutdown_connections 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 -- 172.21.15.28:0/2088849794 wait complete. 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 -- start start 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.343+0000 7f4a690a7740 1 Processor -- start 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738d9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738d4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a690a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a690a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601b03c0 con 0x55c4738d1240 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a690a7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601afcf0 con 0x55c4738c6980 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a690a7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a601acf30 con 0x55c4738c6060 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a690a7740 10 client.0 _ll_register_callbacks cb 0x7f4a50009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35420/0 (socket says 172.21.15.28:35420) 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 -- 172.21.15.28:0/1709957096 learned_addr learned my addr 172.21.15.28:0/1709957096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738d9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738d4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 -- 172.21.15.28:0/1709957096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 msgr2=0x55c4738d9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738d9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 -- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 msgr2=0x55c4738d4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738d4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.344+0000 7f4a6526b640 1 -- 172.21.15.28:0/1709957096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4738d59c0 con 0x55c4738d1240 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a5ffff640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738d4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a6526b640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d51c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4a58031ba0 tx=0x7f4a58005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5175591d1886f389 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a3ffff640 1 -- 172.21.15.28:0/1709957096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58060060 con 0x55c4738d1240 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a3ffff640 1 -- 172.21.15.28:0/1709957096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f4a580039c0 con 0x55c4738d1240 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a5804d070 con 0x55c4738d1240 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a3ffff640 1 -- 172.21.15.28:0/1709957096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4a58067050 con 0x55c4738d1240 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a690a7740 10 client.15960 fetch_fsmap learned FSMap version 29 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c47394b880 con 0x55c4738d1240 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.345+0000 7f4a3ffff640 1 -- 172.21.15.28:0/1709957096 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5804d070 con 0x55c4738d1240 2024-09-15T21:58:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a3ffff640 1 -- 172.21.15.28:0/1709957096 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4a5803c440 con 0x55c4738d1240 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a690a7740 10 client.15960 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a690a7740 10 client.15960 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a690a7740 20 client.15960 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a690a7740 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c47394bef0 0x55c47394e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a690a7740 4 client.15960 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b0814eb3", "release": true, "await": true}] 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b0814eb3", "release": true, "await": true}) -- 0x55c4738bd950 con 0x55c47394bef0 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c47394bef0 0x55c47394e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a64a6a640 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c47394bef0 0x55c47394e310 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92d66dbbac250cdc server_cookie=5fe990303e50bab in_seq=0 out_seq=0 2024-09-15T21:58:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.346+0000 7f4a3ffff640 10 client.15960 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.348+0000 7f4a3ffff640 1 -- 172.21.15.28:0/1709957096 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55c4738bd950 con 0x55c47394bef0 2024-09-15T21:58:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.348+0000 7f4a3ffff640 10 client.15960 handle_command_reply: tid=0 2024-09-15T21:58:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.348+0000 7f4a690a7740 1 client.15960 shutdown 2024-09-15T21:58:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 msgr2=0x55c4738d51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d51c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4a58031ba0 tx=0x7f4a58005bb0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 shutdown_connections 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55c47394bef0 0x55c47394e310 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a3ffff640 0 client.15960 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4738d1240 0x55c4738d51c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c4738c6980 0x55c4738d4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 --2- 172.21.15.28:0/1709957096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c4738c6060 0x55c4738d9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 >> 172.21.15.28:0/1709957096 conn(0x55c4738c25f0 msgr2=0x55c4738cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 shutdown_connections 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 1 -- 172.21.15.28:0/1709957096 wait complete. 2024-09-15T21:58:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.349+0000 7f4a690a7740 20 client.15960 trim_cache size 0 max 16384 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 29, 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 4, 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "b0814eb3": { 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 4, 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-09-15T21:58:40.356 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 90.0, 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 150.2, 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:40.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:40.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:40.358 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:58:40.358 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:58:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4a4c0a35e0 msgr2=0x7f4a4c0a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4a4c0a35e0 0x7f4a4c0a5aa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005940 tx=0x7f4a580058d0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100100 msgr2=0x7f4a60198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100100 0x7f4a60198530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800d770 tx=0x7f4a4800dc30 comp rx=0 tx=0).stop 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 shutdown_connections 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4a4c0a35e0 0x7f4a4c0a5aa0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a60106b80 0x7f4a6019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a60100aa0 0x7f4a60198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 --2- 172.21.15.28:0/1357250440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a60100100 0x7f4a60198530 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 >> 172.21.15.28:0/1357250440 conn(0x7f4a60077230 msgr2=0x7f4a60104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 shutdown_connections 2024-09-15T21:58:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:40.350+0000 7f4a66cce640 1 -- 172.21.15.28:0/1357250440 wait complete. 2024-09-15T21:58:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:41 smithi174 ceph-mon[35244]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:41.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:41 smithi028 ceph-mon[30969]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:41.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:41 smithi203 ceph-mon[35546]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T21:58:42.779 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b0814eb3', total seconds elapsed: 35.3 2024-09-15T21:58:43.649 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 92% (11901/12889) Updating files: 93% (11987/12889) Updating files: 94% (12116/12889) Updating files: 95% (12245/12889) Updating files: 96% (12374/12889) Updating files: 97% (12503/12889) Updating files: 98% (12632/12889) Updating files: 99% (12761/12889) Updating files: 100% (12889/12889) Updating files: 100% (12889/12889), done. 2024-09-15T21:58:43.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:43 smithi174 ceph-mon[35244]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-15T21:58:43.772 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:43 smithi203 ceph-mon[35546]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-15T21:58:43.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:43 smithi028 ceph-mon[30969]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-15T21:58:44.476 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-15T21:58:44.477 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-09-15T21:58:44.478 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-15T21:58:44.478 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-09-15T21:58:44.478 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-15T21:58:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T21:58:44.478 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-15T21:58:44.478 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-15T21:58:44.482 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T21:58:44.505 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T21:58:44.507 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T21:58:44.507 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T21:58:45.191 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T21:58:45.271 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T21:58:45.314 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T21:58:45.315 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T21:58:45.315 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T21:58:45.356 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T21:58:45.360 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T21:58:45.360 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T21:58:45.417 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-09-15T21:58:45.418 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-15T21:58:45.418 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-15T21:58:45.480 INFO:tasks.workunit.client.0.smithi028.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-15T21:58:45.480 INFO:tasks.workunit.client.0.smithi028.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-09-15T21:58:45.481 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap -- retry EXIT 2024-09-15T21:58:45.481 INFO:tasks.workunit.client.0.smithi028.stderr:+ rm -rf ffsb 2024-09-15T21:58:45.481 INFO:tasks.workunit.client.0.smithi028.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-15T21:58:45.484 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into 'ffsb'... 2024-09-15T21:58:45.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:45 smithi174 ceph-mon[35244]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-09-15T21:58:45.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:45 smithi028 ceph-mon[30969]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-09-15T21:58:45.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:45 smithi203 ceph-mon[35546]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-09-15T21:58:46.346 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap - EXIT 2024-09-15T21:58:46.346 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd ffsb 2024-09-15T21:58:46.346 INFO:tasks.workunit.client.0.smithi028.stderr:+ ./configure 2024-09-15T21:58:46.466 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-15T21:58:47.473 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-09-15T21:58:47.484 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-15T21:58:47.484 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-09-15T21:58:47.501 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-09-15T21:58:47.536 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-15T21:58:47.536 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-15T21:58:47.550 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-09-15T21:58:47.554 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-09-15T21:58:47.610 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-09-15T21:58:47.611 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-09-15T21:58:47.615 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-09-15T21:58:47.654 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-09-15T21:58:47.689 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-09-15T21:58:47.724 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-09-15T21:58:47.759 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-09-15T21:58:47.759 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:47 smithi174 ceph-mon[35244]: pgmap v46: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 107 KiB/s wr, 18 op/s 2024-09-15T21:58:47.825 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-15T21:58:47.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:47 smithi028 ceph-mon[30969]: pgmap v46: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 107 KiB/s wr, 18 op/s 2024-09-15T21:58:47.924 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-09-15T21:58:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:47 smithi203 ceph-mon[35546]: pgmap v46: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 107 KiB/s wr, 18 op/s 2024-09-15T21:58:48.096 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-09-15T21:58:48.138 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-15T21:58:48.139 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-09-15T21:58:48.148 INFO:tasks.workunit.client.0.smithi028.stdout:checking for AIX... no 2024-09-15T21:58:48.165 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-09-15T21:58:48.173 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-15T21:58:48.175 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-15T21:58:48.175 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-15T21:58:48.175 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-15T21:58:48.235 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-09-15T21:58:48.285 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-09-15T21:58:48.430 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-09-15T21:58:48.493 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-15T21:58:48.544 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-09-15T21:58:48.595 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-09-15T21:58:48.644 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-09-15T21:58:48.690 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-09-15T21:58:48.745 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-09-15T21:58:48.795 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-09-15T21:58:48.842 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-09-15T21:58:48.893 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-09-15T21:58:48.944 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-09-15T21:58:49.004 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-09-15T21:58:49.026 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-09-15T21:58:49.026 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-09-15T21:58:49.077 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-09-15T21:58:49.098 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-09-15T21:58:49.098 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-09-15T21:58:49.149 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-09-15T21:58:49.170 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-09-15T21:58:49.170 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-09-15T21:58:49.183 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-09-15T21:58:49.235 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-09-15T21:58:49.255 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-09-15T21:58:49.256 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-09-15T21:58:49.269 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-09-15T21:58:49.322 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-09-15T21:58:49.343 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-09-15T21:58:49.343 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-09-15T21:58:49.388 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-09-15T21:58:49.410 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-09-15T21:58:49.410 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-09-15T21:58:49.442 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-09-15T21:58:49.476 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-15T21:58:49.477 INFO:tasks.workunit.client.0.smithi028.stdout:checking for function prototypes... yes 2024-09-15T21:58:49.505 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether setvbuf arguments are reversed... no 2024-09-15T21:58:49.577 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-09-15T21:58:49.639 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-09-15T21:58:49.700 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-09-15T21:58:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:49 smithi174 ceph-mon[35244]: pgmap v47: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 107 KiB/s wr, 14 op/s 2024-09-15T21:58:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:49 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:58:49.762 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-09-15T21:58:49.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:49 smithi028 ceph-mon[30969]: pgmap v47: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 107 KiB/s wr, 14 op/s 2024-09-15T21:58:49.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:49 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:58:49.823 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-09-15T21:58:49.886 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-09-15T21:58:49.948 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-09-15T21:58:49.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:49 smithi203 ceph-mon[35546]: pgmap v47: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 107 KiB/s wr, 14 op/s 2024-09-15T21:58:49.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:49 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T21:58:50.009 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-09-15T21:58:50.071 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-09-15T21:58:50.094 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-09-15T21:58:50.190 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-09-15T21:58:50.203 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-09-15T21:58:50.220 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-09-15T21:58:50.269 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-09-15T21:58:50.314 INFO:tasks.workunit.client.0.smithi028.stderr:+ make 2024-09-15T21:58:50.321 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-09-15T21:58:50.323 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-15T21:58:50.324 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-15T21:58:50.324 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-15T21:58:50.324 INFO:tasks.workunit.client.0.smithi028.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-15T21:58:50.324 INFO:tasks.workunit.client.0.smithi028.stderr: any GNU archive site. 2024-09-15T21:58:50.326 INFO:tasks.workunit.client.0.smithi028.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-15T21:58:50.327 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-15T21:58:50.328 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-15T21:58:50.328 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-15T21:58:50.329 INFO:tasks.workunit.client.0.smithi028.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-15T21:58:50.329 INFO:tasks.workunit.client.0.smithi028.stderr: Grab them from any GNU archive site. 2024-09-15T21:58:50.391 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-09-15T21:58:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 -- 172.21.15.28:0/3008152387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984106a00 msgr2=0x7fc984106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 --2- 172.21.15.28:0/3008152387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984106a00 0x7fc984106de0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc974009930 tx=0x7fc97402a420 comp rx=0 tx=0).stop 2024-09-15T21:58:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 -- 172.21.15.28:0/3008152387 shutdown_connections 2024-09-15T21:58:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 --2- 172.21.15.28:0/3008152387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc984101390 0x7fc98410d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 --2- 172.21.15.28:0/3008152387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9841009f0 0x7fc984100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 --2- 172.21.15.28:0/3008152387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984106a00 0x7fc984106de0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 -- 172.21.15.28:0/3008152387 >> 172.21.15.28:0/3008152387 conn(0x7fc9840fc6b0 msgr2=0x7fc9840fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 -- 172.21.15.28:0/3008152387 shutdown_connections 2024-09-15T21:58:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.599+0000 7fc98a427640 1 -- 172.21.15.28:0/3008152387 wait complete. 2024-09-15T21:58:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 Processor -- start 2024-09-15T21:58:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 -- start start 2024-09-15T21:58:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9841009f0 0x7fc984198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984101390 0x7fc9841988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc984106a00 0x7fc98419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc984110e00 con 0x7fc984101390 2024-09-15T21:58:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc984110c80 con 0x7fc984106a00 2024-09-15T21:58:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc98a427640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc984110f80 con 0x7fc9841009f0 2024-09-15T21:58:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc9837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984101390 0x7fc9841988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc9837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984101390 0x7fc9841988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54000/0 (socket says 172.21.15.28:54000) 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc9837fe640 1 -- 172.21.15.28:0/2698986336 learned_addr learned my addr 172.21.15.28:0/2698986336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc9889c4640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc984106a00 0x7fc98419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.600+0000 7fc983fff640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9841009f0 0x7fc984198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9837fe640 1 -- 172.21.15.28:0/2698986336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9841009f0 msgr2=0x7fc984198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9837fe640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9841009f0 0x7fc984198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9837fe640 1 -- 172.21.15.28:0/2698986336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc984106a00 msgr2=0x7fc98419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9837fe640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc984106a00 0x7fc98419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9837fe640 1 -- 172.21.15.28:0/2698986336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98419d370 con 0x7fc984101390 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9889c4640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc984106a00 0x7fc98419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9837fe640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984101390 0x7fc9841988c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc97800e9f0 tx=0x7fc97800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9817fa640 1 -- 172.21.15.28:0/2698986336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97800cd80 con 0x7fc984101390 2024-09-15T21:58:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9817fa640 1 -- 172.21.15.28:0/2698986336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc978010910 con 0x7fc984101390 2024-09-15T21:58:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98419d660 con 0x7fc984101390 2024-09-15T21:58:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc9817fa640 1 -- 172.21.15.28:0/2698986336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978018990 con 0x7fc984101390 2024-09-15T21:58:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.601+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9841a4ef0 con 0x7fc984101390 2024-09-15T21:58:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc9817fa640 1 -- 172.21.15.28:0/2698986336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138639+0+0 (secure 0 0 0) 0x7fc978010ac0 con 0x7fc984101390 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 Processor -- start 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 -- start start 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db5fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 0x561e1db6a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db6d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e1dac42f0 con 0x561e1db5faf0 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e1da6c270 con 0x561e1db6acb0 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561e1dac02b0 con 0x561e1db60410 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.604+0000 7fc9889c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db5fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db5fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54012/0 (socket says 172.21.15.28:54012) 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 -- 172.21.15.28:0/3539942877 learned_addr learned my addr 172.21.15.28:0/3539942877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9837fe640 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db6d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc983fff640 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 0x561e1db6a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9817fa640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc96c0a3690 0x7fc96c0a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 -- 172.21.15.28:0/3539942877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 msgr2=0x561e1db6a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 0x561e1db6a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 -- 172.21.15.28:0/3539942877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 msgr2=0x561e1db6d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db6d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 -- 172.21.15.28:0/3539942877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e1da6b340 con 0x561e1db5faf0 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9837fe640 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db6d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9817fa640 1 -- 172.21.15.28:0/2698986336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc9780d0670 con 0x7fc984101390 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc96c0a3690 0x7fc96c0a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc9889c4640 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db5fed0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc97c011790 tx=0x7fc97c011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c93a153ce3d0a6f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:50.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc96a7fc640 1 -- 172.21.15.28:0/3539942877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00f9a0 con 0x561e1db5faf0 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc96a7fc640 1 -- 172.21.15.28:0/3539942877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c00fc80 con 0x561e1db5faf0 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.605+0000 7fc96a7fc640 1 -- 172.21.15.28:0/3539942877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c01cbc0 con 0x561e1db5faf0 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 -- 172.21.15.28:0/3539942877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 msgr2=0x561e1db5fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc9889c4640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc96c0a3690 0x7fc96c0a5b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc97c020000 tx=0x7fc97c021000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db5fed0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc97c011790 tx=0x7fc97c011c50 comp rx=0 tx=0).stop 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 -- 172.21.15.28:0/3539942877 shutdown_connections 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db6d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 0x561e1db6a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/3539942877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db5fed0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 -- 172.21.15.28:0/3539942877 >> 172.21.15.28:0/3539942877 conn(0x561e1db5c060 msgr2=0x561e1db5c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 -- 172.21.15.28:0/3539942877 shutdown_connections 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.606+0000 7fc98d1c1740 1 -- 172.21.15.28:0/3539942877 wait complete. 2024-09-15T21:58:50.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 -- start start 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 Processor -- start 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db5faf0 0x561e1db67bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db60410 0x561e1db680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db6acb0 0x561e1db68630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e1db6d880 con 0x561e1db6acb0 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e1da4d290 con 0x561e1db60410 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561e1db6da00 con 0x561e1db5faf0 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc98d1c1740 10 client.0 _ll_register_callbacks cb 0x561e1da7e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc9889c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db6acb0 0x561e1db68630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc9889c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db6acb0 0x561e1db68630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54018/0 (socket says 172.21.15.28:54018) 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc9889c4640 1 -- 172.21.15.28:0/1185793423 learned_addr learned my addr 172.21.15.28:0/1185793423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:50.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc983fff640 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db5faf0 0x561e1db67bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.607+0000 7fc9837fe640 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db60410 0x561e1db680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9889c4640 1 -- 172.21.15.28:0/1185793423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db5faf0 msgr2=0x561e1db67bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9889c4640 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db5faf0 0x561e1db67bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9889c4640 1 -- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db60410 msgr2=0x561e1db680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9889c4640 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db60410 0x561e1db680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9889c4640 1 -- 172.21.15.28:0/1185793423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e1d7d7210 con 0x561e1db6acb0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9889c4640 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db6acb0 0x561e1db68630 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00a010 tx=0x7fc97c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4c7bb69b6b095 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9637fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9637fe640 1 -- 172.21.15.28:0/1185793423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00fe30 con 0x561e1db6acb0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9637fe640 1 -- 172.21.15.28:0/1185793423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc97c02e070 con 0x561e1db6acb0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc97c00fc80 con 0x561e1db6acb0 2024-09-15T21:58:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9637fe640 1 -- 172.21.15.28:0/1185793423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc97c03f050 con 0x561e1db6acb0 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc98d1c1740 10 client.15972 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561e1d8a3620 con 0x561e1db6acb0 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.608+0000 7fc9637fe640 1 -- 172.21.15.28:0/1185793423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c00fc80 con 0x561e1db6acb0 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc9637fe640 1 -- 172.21.15.28:0/1185793423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fc97c028640 con 0x561e1db6acb0 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc98d1c1740 10 client.15972 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc98d1c1740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc98d1c1740 20 client.15972 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1db7ac20 0x561e1db7d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc98d1c1740 4 client.15972 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e1dad1a20 con 0x561e1db7ac20 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc983fff640 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1db7ac20 0x561e1db7d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc9637fe640 10 client.15972 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.609+0000 7fc983fff640 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1db7ac20 0x561e1db7d010 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=336cb0ca7be37215 server_cookie=c58849995711ec55 in_seq=0 out_seq=0 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc9637fe640 1 -- 172.21.15.28:0/1185793423 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e1dad1a20 con 0x561e1db7ac20 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc9637fe640 10 client.15972 handle_command_reply: tid=0 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 client.15972 shutdown 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db6acb0 msgr2=0x561e1db68630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db6acb0 0x561e1db68630 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00a010 tx=0x7fc97c005bd0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 shutdown_connections 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1db7ac20 0x561e1db7d010 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db6acb0 0x561e1db68630 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db60410 0x561e1db680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1185793423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db5faf0 0x561e1db67bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc9637fe640 0 client.15972 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:50.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 >> 172.21.15.28:0/1185793423 conn(0x561e1db5c060 msgr2=0x561e1db61d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 shutdown_connections 2024-09-15T21:58:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1185793423 wait complete. 2024-09-15T21:58:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.610+0000 7fc98d1c1740 20 client.15972 trim_cache size 0 max 16384 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 Processor -- start 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 -- start start 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db62560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 0x561e1db68460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db827f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841aeed0 con 0x561e1db5faf0 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841b0a70 con 0x561e1db6acb0 2024-09-15T21:58:50.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841af5a0 con 0x561e1db60410 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db62560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db62560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54020/0 (socket says 172.21.15.28:54020) 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 -- 172.21.15.28:0/4051470797 learned_addr learned my addr 172.21.15.28:0/4051470797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc9889c4640 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db827f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 -- 172.21.15.28:0/4051470797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 msgr2=0x561e1db68460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 0x561e1db68460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 -- 172.21.15.28:0/4051470797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 msgr2=0x561e1db827f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db827f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 -- 172.21.15.28:0/4051470797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e1db562f0 con 0x561e1db5faf0 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc9889c4640 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db827f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc983fff640 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db62560 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc974000c00 tx=0x7fc974009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7ef1a31c59faad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.625+0000 7fc969ffb640 1 -- 172.21.15.28:0/4051470797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974049070 con 0x561e1db5faf0 2024-09-15T21:58:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc969ffb640 1 -- 172.21.15.28:0/4051470797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc974005520 con 0x561e1db5faf0 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc969ffb640 1 -- 172.21.15.28:0/4051470797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974035040 con 0x561e1db5faf0 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 -- 172.21.15.28:0/4051470797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 msgr2=0x561e1db62560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db62560 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc974000c00 tx=0x7fc974009560 comp rx=0 tx=0).stop 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 -- 172.21.15.28:0/4051470797 shutdown_connections 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db6acb0 0x561e1db827f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db60410 0x561e1db68460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/4051470797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db5faf0 0x561e1db62560 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 -- 172.21.15.28:0/4051470797 >> 172.21.15.28:0/4051470797 conn(0x561e1db5c060 msgr2=0x561e1db69c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 -- 172.21.15.28:0/4051470797 shutdown_connections 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.626+0000 7fc98d1c1740 1 -- 172.21.15.28:0/4051470797 wait complete. 2024-09-15T21:58:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 -- start start 2024-09-15T21:58:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 Processor -- start 2024-09-15T21:58:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db5faf0 0x561e1db72c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db60410 0x561e1db731d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db6acb0 0x561e1db641d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841b0340 con 0x561e1db60410 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841afc70 con 0x561e1db5faf0 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9841aceb0 con 0x561e1db6acb0 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc98d1c1740 10 client.0 _ll_register_callbacks cb 0x7fc97c01f020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db60410 0x561e1db731d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db60410 0x561e1db731d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54032/0 (socket says 172.21.15.28:54032) 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 -- 172.21.15.28:0/1854758427 learned_addr learned my addr 172.21.15.28:0/1854758427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc983fff640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db5faf0 0x561e1db72c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9889c4640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db6acb0 0x561e1db641d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 -- 172.21.15.28:0/1854758427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db6acb0 msgr2=0x561e1db641d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db6acb0 0x561e1db641d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 -- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db5faf0 msgr2=0x561e1db72c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db5faf0 0x561e1db72c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9837fe640 1 -- 172.21.15.28:0/1854758427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e1db64790 con 0x561e1db60410 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.627+0000 7fc9889c4640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db6acb0 0x561e1db641d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc9837fe640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db60410 0x561e1db731d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc9780cd910 tx=0x7fc978098710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fef3de1a7fd3bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc9637fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc9637fe640 1 -- 172.21.15.28:0/1854758427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9780a5020 con 0x561e1db60410 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc9637fe640 1 -- 172.21.15.28:0/1854758427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc9780d02b0 con 0x561e1db60410 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc974005520 con 0x561e1db60410 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc9637fe640 1 -- 172.21.15.28:0/1854758427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc9780d04b0 con 0x561e1db60410 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 10 client.15978 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561e1dbe5060 con 0x561e1db60410 2024-09-15T21:58:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc9637fe640 1 -- 172.21.15.28:0/1854758427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc974005520 con 0x561e1db60410 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc9637fe640 1 -- 172.21.15.28:0/1854758427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fc97809b990 con 0x561e1db60410 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 10 client.15978 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 10 client.15978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 20 client.15978 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1dbe57f0 0x561e1dbe7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 4 client.15978 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.628+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561e1dacbd70 con 0x561e1dbe57f0 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.629+0000 7fc983fff640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1dbe57f0 0x561e1dbe7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.629+0000 7fc983fff640 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1dbe57f0 0x561e1dbe7c10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3027127a69c0bb9a server_cookie=b73039b7a17c5ea3 in_seq=0 out_seq=0 2024-09-15T21:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.629+0000 7fc9637fe640 10 client.15978 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:50.781 INFO:tasks.workunit.client.0.smithi028.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-15T21:58:50.781 INFO:tasks.workunit.client.0.smithi028.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-15T21:58:50.781 INFO:tasks.workunit.client.0.smithi028.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-15T21:58:50.781 INFO:tasks.workunit.client.0.smithi028.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.851+0000 7fc9637fe640 1 -- 172.21.15.28:0/1854758427 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561e1dacbd70 con 0x561e1dbe57f0 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.851+0000 7fc9637fe640 10 client.15978 handle_command_reply: tid=0 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.851+0000 7fc98d1c1740 1 client.15978 shutdown 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db60410 msgr2=0x561e1db731d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db60410 0x561e1db731d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc9780cd910 tx=0x7fc978098710 comp rx=0 tx=0).stop 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 shutdown_connections 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561e1dbe57f0 0x561e1dbe7c10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e1db6acb0 0x561e1db641d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1db60410 0x561e1db731d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 --2- 172.21.15.28:0/1854758427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e1db5faf0 0x561e1db72c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 >> 172.21.15.28:0/1854758427 conn(0x561e1db5c060 msgr2=0x561e1db5a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc9637fe640 0 client.15978 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 shutdown_connections 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 1 -- 172.21.15.28:0/1854758427 wait complete. 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.852+0000 7fc98d1c1740 20 client.15978 trim_cache size 0 max 16384 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T21:58:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T21:58:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T21:58:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc96c0a3690 msgr2=0x7fc96c0a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc96c0a3690 0x7fc96c0a5b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc97c020000 tx=0x7fc97c021000 comp rx=0 tx=0).stop 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984101390 msgr2=0x7fc9841988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984101390 0x7fc9841988c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc97800e9f0 tx=0x7fc97800eeb0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 shutdown_connections 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc96c0a3690 0x7fc96c0a5b50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc984106a00 0x7fc98419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc984101390 0x7fc9841988c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 --2- 172.21.15.28:0/2698986336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9841009f0 0x7fc984198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 >> 172.21.15.28:0/2698986336 conn(0x7fc9840fc6b0 msgr2=0x7fc98406a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 shutdown_connections 2024-09-15T21:58:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:58:50.853+0000 7fc98a427640 1 -- 172.21.15.28:0/2698986336 wait complete. 2024-09-15T21:58:50.956 INFO:tasks.workunit.client.0.smithi028.stdout:/bin/sh ./config.status --recheck 2024-09-15T21:58:51.007 INFO:tasks.workunit.client.0.smithi028.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-09-15T21:58:51.105 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-15T21:58:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:51 smithi028 ceph-mon[30969]: pgmap v48: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 107 KiB/s wr, 14 op/s 2024-09-15T21:58:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:51 smithi203 ceph-mon[35546]: pgmap v48: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 107 KiB/s wr, 14 op/s 2024-09-15T21:58:52.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:51 smithi174 ceph-mon[35244]: pgmap v48: 129 pgs: 129 active+clean; 2.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 107 KiB/s wr, 14 op/s 2024-09-15T21:58:52.113 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-09-15T21:58:52.117 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-15T21:58:52.118 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-09-15T21:58:52.145 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-09-15T21:58:52.176 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-15T21:58:52.176 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-15T21:58:52.187 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-09-15T21:58:52.191 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-09-15T21:58:52.274 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-09-15T21:58:52.274 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-09-15T21:58:52.311 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-09-15T21:58:52.348 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-09-15T21:58:52.384 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-09-15T21:58:52.407 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-09-15T21:58:52.431 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-09-15T21:58:52.466 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-15T21:58:52.585 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-09-15T21:58:52.627 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-09-15T21:58:52.675 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-15T21:58:52.677 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-09-15T21:58:52.797 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-09-15T21:58:52.830 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-09-15T21:58:52.868 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-09-15T21:58:52.906 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-09-15T21:58:52.945 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-09-15T21:58:52.987 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-09-15T21:58:53.030 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-09-15T21:58:53.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T21:58:53.073 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-09-15T21:58:53.118 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-09-15T21:58:53.163 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-09-15T21:58:53.210 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h usability... no 2024-09-15T21:58:53.241 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h presence... no 2024-09-15T21:58:53.242 INFO:tasks.workunit.client.0.smithi028.stdout:checking for minix/config.h... no 2024-09-15T21:58:53.282 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-15T21:58:53.283 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-09-15T21:58:53.303 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:53.330 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-15T21:58:53.330 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-15T21:58:53.330 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-15T21:58:53.331 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-15T21:58:53.430 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-09-15T21:58:53.525 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-09-15T21:58:53.526 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... (cached) yes 2024-09-15T21:58:53.576 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-15T21:58:53.634 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-09-15T21:58:53.657 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-09-15T21:58:53.657 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-09-15T21:58:53.704 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-09-15T21:58:53.721 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-09-15T21:58:53.722 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-09-15T21:58:53.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:53 smithi028 ceph-mon[30969]: pgmap v49: 129 pgs: 129 active+clean; 5.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 460 KiB/s wr, 176 op/s 2024-09-15T21:58:53.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:53 smithi028 ceph-mon[30969]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-09-15T21:58:53.771 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-09-15T21:58:53.787 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-09-15T21:58:53.787 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-09-15T21:58:53.800 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-09-15T21:58:53.854 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-09-15T21:58:53.872 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-09-15T21:58:53.872 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-09-15T21:58:53.886 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-09-15T21:58:53.952 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-09-15T21:58:53.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:53 smithi203 ceph-mon[35546]: pgmap v49: 129 pgs: 129 active+clean; 5.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 460 KiB/s wr, 176 op/s 2024-09-15T21:58:53.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:53 smithi203 ceph-mon[35546]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-09-15T21:58:53.970 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-09-15T21:58:53.970 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-09-15T21:58:54.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:53 smithi174 ceph-mon[35244]: pgmap v49: 129 pgs: 129 active+clean; 5.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 460 KiB/s wr, 176 op/s 2024-09-15T21:58:54.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:53 smithi174 ceph-mon[35244]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-09-15T21:58:54.039 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-09-15T21:58:54.072 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-09-15T21:58:54.072 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-09-15T21:58:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 -- 172.21.15.28:0/1067841433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8108290 msgr2=0x7efed810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 --2- 172.21.15.28:0/1067841433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8108290 0x7efed810ec30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7efed0009cd0 tx=0x7efed002a8e0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 -- 172.21.15.28:0/1067841433 shutdown_connections 2024-09-15T21:58:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 --2- 172.21.15.28:0/1067841433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8108290 0x7efed810ec30 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 --2- 172.21.15.28:0/1067841433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed8103740 0x7efed8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 --2- 172.21.15.28:0/1067841433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efed8102d90 0x7efed8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 -- 172.21.15.28:0/1067841433 >> 172.21.15.28:0/1067841433 conn(0x7efed80fc7d0 msgr2=0x7efed80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 -- 172.21.15.28:0/1067841433 shutdown_connections 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.045+0000 7efedde8b640 1 -- 172.21.15.28:0/1067841433 wait complete. 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 Processor -- start 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 -- start start 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8102d90 0x7efed8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efed8103740 0x7efed8198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed8108290 0x7efed819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efed8112090 con 0x7efed8102d90 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efed8111f10 con 0x7efed8108290 2024-09-15T21:58:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efedde8b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efed8112210 con 0x7efed8103740 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.046+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8102d90 0x7efed8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8102d90 0x7efed8198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54038/0 (socket says 172.21.15.28:54038) 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed8108290 0x7efed819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed77fe640 1 -- 172.21.15.28:0/2603398298 learned_addr learned my addr 172.21.15.28:0/2603398298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed6ffd640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efed8103740 0x7efed8198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed7fff640 1 -- 172.21.15.28:0/2603398298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efed8103740 msgr2=0x7efed8198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed7fff640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efed8103740 0x7efed8198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed7fff640 1 -- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8102d90 msgr2=0x7efed8198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed7fff640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8102d90 0x7efed8198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed7fff640 1 -- 172.21.15.28:0/2603398298 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed819d500 con 0x7efed8108290 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed77fe640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8102d90 0x7efed8198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed7fff640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed8108290 0x7efed819ce20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efed002ae00 tx=0x7efed0031df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efed4ff9640 1 -- 172.21.15.28:0/2603398298 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed0003d80 con 0x7efed8108290 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.047+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed819d790 con 0x7efed8108290 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.048+0000 7efed4ff9640 1 -- 172.21.15.28:0/2603398298 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed0004520 con 0x7efed8108290 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.048+0000 7efed4ff9640 1 -- 172.21.15.28:0/2603398298 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed0030d00 con 0x7efed8108290 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.048+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed81a50d0 con 0x7efed8108290 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efed4ff9640 1 -- 172.21.15.28:0/2603398298 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7efed0046480 con 0x7efed8108290 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 Processor -- start 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 -- start start 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818e7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e7dc0 0x55fd818f2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efed4ff9640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efec00a33b0 0x7efec00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fd8184d2e0 con 0x55fd818e74a0 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fd817f4ce0 con 0x55fd818e7dc0 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.050+0000 7efee0264740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fd818492a0 con 0x55fd818f2680 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed4ff9640 1 -- 172.21.15.28:0/2603398298 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7efed002e030 con 0x7efed8108290 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818e7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818e7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54052/0 (socket says 172.21.15.28:54052) 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed6ffd640 1 -- 172.21.15.28:0/852987741 learned_addr learned my addr 172.21.15.28:0/852987741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed77fe640 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e7dc0 0x55fd818f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed6ffd640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efec00a33b0 0x7efec00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed7fff640 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed77fe640 1 -- 172.21.15.28:0/852987741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 msgr2=0x55fd818f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed77fe640 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed77fe640 1 -- 172.21.15.28:0/852987741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 msgr2=0x55fd818e7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed77fe640 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818e7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed77fe640 1 -- 172.21.15.28:0/852987741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd817d9440 con 0x55fd818e7dc0 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed77fe640 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e7dc0 0x55fd818f2140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efec80114f0 tx=0x7efec80119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=406e30ef82e5fc6b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efed6ffd640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efec00a33b0 0x7efec00a5870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efecc0093e0 tx=0x7efecc009850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.051+0000 7efebdffb640 1 -- 172.21.15.28:0/852987741 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8011020 con 0x55fd818e7dc0 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efebdffb640 1 -- 172.21.15.28:0/852987741 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec80111c0 con 0x55fd818e7dc0 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efebdffb640 1 -- 172.21.15.28:0/852987741 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec800f5c0 con 0x55fd818e7dc0 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 -- 172.21.15.28:0/852987741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e7dc0 msgr2=0x55fd818f2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e7dc0 0x55fd818f2140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efec80114f0 tx=0x7efec80119b0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 -- 172.21.15.28:0/852987741 shutdown_connections 2024-09-15T21:58:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e7dc0 0x55fd818f2140 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 --2- 172.21.15.28:0/852987741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818e7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 -- 172.21.15.28:0/852987741 >> 172.21.15.28:0/852987741 conn(0x55fd818e3a30 msgr2=0x55fd818e3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 -- 172.21.15.28:0/852987741 shutdown_connections 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.052+0000 7efee0264740 1 -- 172.21.15.28:0/852987741 wait complete. 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 -- start start 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 Processor -- start 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e74a0 0x55fd818f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e7dc0 0x55fd818ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818ef800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fd818f5250 con 0x55fd818e7dc0 2024-09-15T21:58:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fd81808050 con 0x55fd818e74a0 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fd818f53d0 con 0x55fd818f2680 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.053+0000 7efee0264740 10 client.0 _ll_register_callbacks cb 0x55fd81848160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e7dc0 0x55fd818ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e7dc0 0x55fd818ef2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54062/0 (socket says 172.21.15.28:54062) 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 -- 172.21.15.28:0/3194607307 learned_addr learned my addr 172.21.15.28:0/3194607307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed7fff640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818ef800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed77fe640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e74a0 0x55fd818f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 -- 172.21.15.28:0/3194607307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 msgr2=0x55fd818ef800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818ef800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 -- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e74a0 msgr2=0x55fd818f1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e74a0 0x55fd818f1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 -- 172.21.15.28:0/3194607307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd815be3c0 con 0x55fd818e7dc0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed7fff640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818ef800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed77fe640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e74a0 0x55fd818f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efed6ffd640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e7dc0 0x55fd818ef2a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efecc00fee0 tx=0x7efecc009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a706b846b8c10e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efeb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efeb6ffd640 1 -- 172.21.15.28:0/3194607307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc026070 con 0x55fd818e7dc0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efeb6ffd640 1 -- 172.21.15.28:0/3194607307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7efecc004220 con 0x55fd818e7dc0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.054+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efec80111c0 con 0x55fd818e7dc0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efeb6ffd640 1 -- 172.21.15.28:0/3194607307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efecc004420 con 0x55fd818e7dc0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 10 client.15990 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fd81900c20 con 0x55fd818e7dc0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efeb6ffd640 1 -- 172.21.15.28:0/3194607307 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec80111c0 con 0x55fd818e7dc0 2024-09-15T21:58:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efeb6ffd640 1 -- 172.21.15.28:0/3194607307 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7efecc004b40 con 0x55fd818e7dc0 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 10 client.15990 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 10 client.15990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 20 client.15990 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819045f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 4 client.15990 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd8185a850 con 0x55fd819021a0 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.055+0000 7efed77fe640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.056+0000 7efed77fe640 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819045f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=898871a7139ba7e4 server_cookie=fc3e9f72f5b5c51e in_seq=0 out_seq=0 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.056+0000 7efeb6ffd640 10 client.15990 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efeb6ffd640 1 -- 172.21.15.28:0/3194607307 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fd8185a850 con 0x55fd819021a0 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efeb6ffd640 10 client.15990 handle_command_reply: tid=0 2024-09-15T21:58:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 client.15990 shutdown 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e7dc0 msgr2=0x55fd818ef2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e7dc0 0x55fd818ef2a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efecc00fee0 tx=0x7efecc009370 comp rx=0 tx=0).stop 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 shutdown_connections 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819045f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efeb6ffd640 0 client.15990 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818f2680 0x55fd818ef800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e7dc0 0x55fd818ef2a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 --2- 172.21.15.28:0/3194607307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818e74a0 0x55fd818f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 >> 172.21.15.28:0/3194607307 conn(0x55fd818e3a30 msgr2=0x55fd818e9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 shutdown_connections 2024-09-15T21:58:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 1 -- 172.21.15.28:0/3194607307 wait complete. 2024-09-15T21:58:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.058+0000 7efee0264740 20 client.15990 trim_cache size 0 max 16384 2024-09-15T21:58:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 Processor -- start 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 -- start start 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd818eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd81909f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efed81aeeb0 con 0x55fd818e74a0 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efed81b0a50 con 0x55fd818f2680 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.076+0000 7efee0264740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efed81af580 con 0x55fd818e7dc0 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818fe4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54072/0 (socket says 172.21.15.28:54072) 2024-09-15T21:58:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed77fe640 1 -- 172.21.15.28:0/3060332915 learned_addr learned my addr 172.21.15.28:0/3060332915 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed7fff640 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd81909f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed6ffd640 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd818eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed7fff640 1 -- 172.21.15.28:0/3060332915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 msgr2=0x55fd818eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed7fff640 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd818eeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed7fff640 1 -- 172.21.15.28:0/3060332915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 msgr2=0x55fd818fe4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed7fff640 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818fe4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed7fff640 1 -- 172.21.15.28:0/3060332915 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd818f2360 con 0x55fd818f2680 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed77fe640 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efed7fff640 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd81909f00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efed00ee990 tx=0x7efed0009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef376cc282c7505 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efebd7fa640 1 -- 172.21.15.28:0/3060332915 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed00bd070 con 0x55fd818f2680 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efebd7fa640 1 -- 172.21.15.28:0/3060332915 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed0036040 con 0x55fd818f2680 2024-09-15T21:58:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.077+0000 7efebd7fa640 1 -- 172.21.15.28:0/3060332915 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed00d6750 con 0x55fd818f2680 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 -- 172.21.15.28:0/3060332915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 msgr2=0x55fd81909f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd81909f00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efed00ee990 tx=0x7efed0009650 comp rx=0 tx=0).stop 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 -- 172.21.15.28:0/3060332915 shutdown_connections 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd81909f00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd818eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 --2- 172.21.15.28:0/3060332915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd818fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 -- 172.21.15.28:0/3060332915 >> 172.21.15.28:0/3060332915 conn(0x55fd818e3a30 msgr2=0x55fd818ee390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 -- 172.21.15.28:0/3060332915 shutdown_connections 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.078+0000 7efee0264740 1 -- 172.21.15.28:0/3060332915 wait complete. 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.079+0000 7efee0264740 1 -- start start 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.079+0000 7efee0264740 1 Processor -- start 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efee0264740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd819654a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efee0264740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd819659e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efee0264740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd818f9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efee0264740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efed81b0320 con 0x55fd818e74a0 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efee0264740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efed81afc50 con 0x55fd818f2680 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efee0264740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efed81ace90 con 0x55fd818e7dc0 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efee0264740 10 client.0 _ll_register_callbacks cb 0x7efecc00fdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd819654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd819654a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54082/0 (socket says 172.21.15.28:54082) 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 -- 172.21.15.28:0/2042217636 learned_addr learned my addr 172.21.15.28:0/2042217636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed7fff640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd818f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed6ffd640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd819659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 -- 172.21.15.28:0/2042217636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 msgr2=0x55fd819659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd819659e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 -- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 msgr2=0x55fd818f9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd818f9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 -- 172.21.15.28:0/2042217636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd818f9b60 con 0x55fd818e74a0 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed7fff640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd818f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efed77fe640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd819654a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efec8009230 tx=0x7efec8009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d5a568f96a4837 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.080+0000 7efeb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efeb6ffd640 1 -- 172.21.15.28:0/2042217636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8003df0 con 0x55fd818e74a0 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efeb6ffd640 1 -- 172.21.15.28:0/2042217636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7efec8011040 con 0x55fd818e74a0 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efed0036040 con 0x55fd818e74a0 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efeb6ffd640 1 -- 172.21.15.28:0/2042217636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efec8039050 con 0x55fd818e74a0 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 10 client.15996 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fd818fae70 con 0x55fd818e74a0 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efeb6ffd640 1 -- 172.21.15.28:0/2042217636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed0036040 con 0x55fd818e74a0 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efeb6ffd640 1 -- 172.21.15.28:0/2042217636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7efec800fa40 con 0x55fd818e74a0 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 10 client.15996 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 20 client.15996 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819732f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 4 client.15996 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.081+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fd81854d60 con 0x55fd819021a0 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.082+0000 7efed6ffd640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.082+0000 7efeb6ffd640 10 client.15996 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.082+0000 7efed6ffd640 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819732f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ec5ae2f0d470973 server_cookie=91f690e35d6cad87 in_seq=0 out_seq=0 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.082+0000 7efeb6ffd640 1 -- 172.21.15.28:0/2042217636 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fd81854d60 con 0x55fd819021a0 2024-09-15T21:58:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.082+0000 7efeb6ffd640 10 client.15996 handle_command_reply: tid=0 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.082+0000 7efee0264740 1 client.15996 shutdown 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 msgr2=0x55fd819654a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd819654a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efec8009230 tx=0x7efec8009530 comp rx=0 tx=0).stop 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 shutdown_connections 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fd819021a0 0x55fd819732f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd818f2680 0x55fd818f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efeb6ffd640 0 client.15996 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fd818e7dc0 0x55fd819659e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 --2- 172.21.15.28:0/2042217636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fd818e74a0 0x55fd819654a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 >> 172.21.15.28:0/2042217636 conn(0x55fd818e3a30 msgr2=0x55fd818e2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 shutdown_connections 2024-09-15T21:58:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 1 -- 172.21.15.28:0/2042217636 wait complete. 2024-09-15T21:58:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.083+0000 7efee0264740 20 client.15996 trim_cache size 0 max 16384 2024-09-15T21:58:54.097 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:58:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T21:58:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T21:58:54.097 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:58:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efec00a33b0 msgr2=0x7efec00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efec00a33b0 0x7efec00a5870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efecc0093e0 tx=0x7efecc009850 comp rx=0 tx=0).stop 2024-09-15T21:58:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed8108290 msgr2=0x7efed819ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed8108290 0x7efed819ce20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efed002ae00 tx=0x7efed0031df0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 shutdown_connections 2024-09-15T21:58:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efec00a33b0 0x7efec00a5870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed8108290 0x7efed819ce20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efed8103740 0x7efed8198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 --2- 172.21.15.28:0/2603398298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8102d90 0x7efed8198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 >> 172.21.15.28:0/2603398298 conn(0x7efed80fc7d0 msgr2=0x7efed8105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 shutdown_connections 2024-09-15T21:58:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:54.084+0000 7efedde8b640 1 -- 172.21.15.28:0/2603398298 wait complete. 2024-09-15T21:58:54.114 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-09-15T21:58:54.148 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-15T21:58:54.209 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-09-15T21:58:54.268 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-09-15T21:58:54.325 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-09-15T21:58:54.484 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-09-15T21:58:54.544 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-09-15T21:58:54.600 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-09-15T21:58:54.656 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-09-15T21:58:54.711 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-09-15T21:58:54.767 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-09-15T21:58:54.789 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-09-15T21:58:54.850 INFO:tasks.workunit.client.0.smithi028.stdout: /bin/sh ./config.status 2024-09-15T21:58:54.899 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-09-15T21:58:54.911 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-09-15T21:58:54.936 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-09-15T21:58:54.947 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-09-15T21:58:54.971 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-09-15T21:58:55.035 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T21:58:55.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:58:55.152 INFO:tasks.workunit.client.0.smithi028.stdout:rm -f stamp-h1 2024-09-15T21:58:55.155 INFO:tasks.workunit.client.0.smithi028.stdout:touch config.h.in 2024-09-15T21:58:55.156 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh ./config.status config.h 2024-09-15T21:58:55.194 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-09-15T21:58:55.217 INFO:tasks.workunit.client.0.smithi028.stdout:make all-am 2024-09-15T21:58:55.223 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-15T21:58:55.223 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-15T21:58:55.355 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:55.420 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-15T21:58:55.423 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-15T21:58:55.600 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-15T21:58:55.602 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-15T21:58:55.693 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-15T21:58:55.695 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-15T21:58:55.787 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-15T21:58:55.789 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-15T21:58:55.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:55 smithi028 ceph-mon[30969]: pgmap v50: 129 pgs: 129 active+clean; 5.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 460 KiB/s wr, 172 op/s 2024-09-15T21:58:55.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:55 smithi203 ceph-mon[35546]: pgmap v50: 129 pgs: 129 active+clean; 5.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 460 KiB/s wr, 172 op/s 2024-09-15T21:58:55.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 -- 172.21.15.28:0/1712166207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 msgr2=0x7fb8bc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:55.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1712166207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc1031d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009900 tx=0x7fb8ac02a3f0 comp rx=0 tx=0).stop 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 -- 172.21.15.28:0/1712166207 shutdown_connections 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1712166207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8bc10c2f0 0x7fb8bc10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1712166207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8bc1037a0 0x7fb8bc10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1712166207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc1031d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 -- 172.21.15.28:0/1712166207 >> 172.21.15.28:0/1712166207 conn(0x7fb8bc0fc7d0 msgr2=0x7fb8bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 -- 172.21.15.28:0/1712166207 shutdown_connections 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 -- 172.21.15.28:0/1712166207 wait complete. 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.953+0000 7fb8c299a640 1 Processor -- start 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c299a640 1 -- start start 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c299a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c299a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8bc1037a0 0x7fb8bc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c299a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8bc10c2f0 0x7fb8bc19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c299a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1120f0 con 0x7fb8bc102df0 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c299a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc111f70 con 0x7fb8bc10c2f0 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c299a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc112270 con 0x7fb8bc1037a0 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54108/0 (socket says 172.21.15.28:54108) 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 -- 172.21.15.28:0/1961100214 learned_addr learned my addr 172.21.15.28:0/1961100214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bb7fe640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8bc1037a0 0x7fb8bc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8c0f37640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8bc10c2f0 0x7fb8bc19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 -- 172.21.15.28:0/1961100214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8bc1037a0 msgr2=0x7fb8bc198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8bc1037a0 0x7fb8bc198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 -- 172.21.15.28:0/1961100214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8bc10c2f0 msgr2=0x7fb8bc19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8bc10c2f0 0x7fb8bc19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.954+0000 7fb8bbfff640 1 -- 172.21.15.28:0/1961100214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc19d500 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.955+0000 7fb8bbfff640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc198550 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac005ab0 tx=0x7fb8ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.955+0000 7fb8b97fa640 1 -- 172.21.15.28:0/1961100214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac03f030 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.955+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc19d790 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.955+0000 7fb8b97fa640 1 -- 172.21.15.28:0/1961100214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8ac003ce0 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.955+0000 7fb8b97fa640 1 -- 172.21.15.28:0/1961100214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac035070 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.955+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc1a50d0 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.956+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8bc06a790 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.958+0000 7fb8b97fa640 1 -- 172.21.15.28:0/1961100214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fb8ac042030 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.959+0000 7fb8b97fa640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb8a40a33b0 0x7fb8a40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.959+0000 7fb8b97fa640 1 -- 172.21.15.28:0/1961100214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb8ac0f0a80 con 0x7fb8bc102df0 2024-09-15T21:58:55.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.959+0000 7fb8bb7fe640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb8a40a33b0 0x7fb8a40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.959+0000 7fb8bb7fe640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb8a40a33b0 0x7fb8a40a5870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0005e10 tx=0x7fb8b0005da0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:55.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:55.963+0000 7fb8b97fa640 1 -- 172.21.15.28:0/1961100214 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb8ac0f2e20 con 0x7fb8bc102df0 2024-09-15T21:58:55.976 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-15T21:58:55.980 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-15T21:58:56.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:55 smithi174 ceph-mon[35244]: pgmap v50: 129 pgs: 129 active+clean; 5.9 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 460 KiB/s wr, 172 op/s 2024-09-15T21:58:56.030 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-15T21:58:56.033 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-15T21:58:56.075 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-15T21:58:56.078 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-15T21:58:56.124 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-15T21:58:56.127 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-15T21:58:56.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.179+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8bc1031d0 con 0x7fb8bc102df0 2024-09-15T21:58:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.179+0000 7fb8b97fa640 1 -- 172.21.15.28:0/1961100214 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11989 (secure 0 0 0) 0x7fb8ac0bd0a0 con 0x7fb8bc102df0 2024-09-15T21:58:56.181 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:58:56.182 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"btime":"2024-09-15T21:58:48:502775+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:58:48.285105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:58:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-09-15T21:58:56.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.181+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb8a40a33b0 msgr2=0x7fb8a40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:56.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.181+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb8a40a33b0 0x7fb8a40a5870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0005e10 tx=0x7fb8b0005da0 comp rx=0 tx=0).stop 2024-09-15T21:58:56.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.181+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 msgr2=0x7fb8bc198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.181+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc198550 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac005ab0 tx=0x7fb8ac0042f0 comp rx=0 tx=0).stop 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 shutdown_connections 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb8a40a33b0 0x7fb8a40a5870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8bc10c2f0 0x7fb8bc19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8bc1037a0 0x7fb8bc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 --2- 172.21.15.28:0/1961100214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8bc102df0 0x7fb8bc198550 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 >> 172.21.15.28:0/1961100214 conn(0x7fb8bc0fc7d0 msgr2=0x7fb8bc0fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 shutdown_connections 2024-09-15T21:58:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:56.182+0000 7fb8c299a640 1 -- 172.21.15.28:0/1961100214 wait complete. 2024-09-15T21:58:56.345 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-15T21:58:56.349 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-15T21:58:56.577 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-15T21:58:56.580 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-15T21:58:56.749 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-15T21:58:56.752 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-15T21:58:56.819 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-15T21:58:56.822 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-15T21:58:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:56 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1961100214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:56.955 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-15T21:58:56.958 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-15T21:58:56.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:56 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1961100214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:57.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:56 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1961100214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:58:57.061 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-15T21:58:57.064 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-15T21:58:57.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T21:58:57.107 INFO:tasks.workunit.client.0.smithi028.stderr:In file included from parser.h:22, 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr: from parser.c:25: 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_filesys’: 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.108 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.109 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.110 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.111 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-15T21:58:57.112 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_tg_stats’: 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.113 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.114 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.115 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-15T21:58:57.116 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-15T21:58:57.117 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.161 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘get_optstr’: 2024-09-15T21:58:57.162 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-15T21:58:57.162 INFO:tasks.workunit.client.0.smithi028.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-15T21:58:57.162 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.162 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:200:14: note: source object declared here 2024-09-15T21:58:57.162 INFO:tasks.workunit.client.0.smithi028.stderr: 200 | char temp[BUFSIZE]; 2024-09-15T21:58:57.162 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-09-15T21:58:57.168 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘size64_convert’: 2024-09-15T21:58:57.168 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-15T21:58:57.168 INFO:tasks.workunit.client.0.smithi028.stderr: 119 | memcpy(unit, "\0", 3); 2024-09-15T21:58:57.168 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-09-15T21:58:57.379 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:57.473 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-15T21:58:57.476 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-15T21:58:57.533 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-15T21:58:57.536 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-15T21:58:57.638 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-15T21:58:57.641 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-15T21:58:57.674 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-15T21:58:57.677 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-15T21:58:57.713 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-15T21:58:57.713 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd .. 2024-09-15T21:58:57.713 INFO:tasks.workunit.client.0.smithi028.stderr:+ mkdir tmp 2024-09-15T21:58:57.715 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd tmp 2024-09-15T21:58:57.715 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in $mydir/*.ffsb 2024-09-15T21:58:57.715 INFO:tasks.workunit.client.0.smithi028.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-09-15T21:58:57.716 INFO:tasks.workunit.client.0.smithi028.stdout:FFSB version 6.0-RC2 started 2024-09-15T21:58:57.716 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout:benchmark time = 300 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout:ThreadGroup 0 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout:================ 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: num_threads = 32 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: read_random = off 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: read_size = 0 (0B) 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: read_blocksize = 0 (0B) 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: read_skip = off 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: read_skipsize = 0 (0B) 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.719 INFO:tasks.workunit.client.0.smithi028.stdout: write_random = on 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: write_size = 5242880 (5MB) 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: fsync_file = 0 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: write_blocksize = 4096 (4KB) 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: wait time = 0 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: op weights 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: read = 0 (0.00%) 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: readall = 0 (0.00%) 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: write = 1 (100.00%) 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: create = 0 (0.00%) 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: append = 0 (0.00%) 2024-09-15T21:58:57.720 INFO:tasks.workunit.client.0.smithi028.stdout: delete = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: metaop = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: createdir = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: stat = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: writeall = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: writeall_fsync = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: open_close = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: write_fsync = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: create_fsync = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: append_fsync = 0 (0.00%) 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.721 INFO:tasks.workunit.client.0.smithi028.stdout:FileSystem . 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout:========== 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: num_dirs = 0 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: starting files = 128 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: min file size = 104857600 (100MB) 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: max file size = 104857600 (100MB) 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: directio = off 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: alignedio = on 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: bufferedio = off 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: aging is off 2024-09-15T21:58:57.722 INFO:tasks.workunit.client.0.smithi028.stdout: current utilization = 0.04% 2024-09-15T21:58:57.723 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:58:57.723 INFO:tasks.workunit.client.0.smithi028.stdout:checking existing fs: . 2024-09-15T21:58:57.723 INFO:tasks.workunit.client.0.smithi028.stdout:recreating new fileset 2024-09-15T21:58:57.723 INFO:tasks.workunit.client.0.smithi028.stderr:opendir: No such file or directory 2024-09-15T21:58:57.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:57 smithi028 ceph-mon[30969]: pgmap v51: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 612 KiB/s wr, 216 op/s 2024-09-15T21:58:57.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:57 smithi203 ceph-mon[35546]: pgmap v51: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 612 KiB/s wr, 216 op/s 2024-09-15T21:58:58.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:57 smithi174 ceph-mon[35244]: pgmap v51: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 612 KiB/s wr, 216 op/s 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1553844960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc4810c180 msgr2=0x7fbc4810e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1553844960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc4810c180 0x7fbc4810e620 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009900 tx=0x7fbc4002a480 comp rx=0 tx=0).stop 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1553844960 shutdown_connections 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1553844960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc4810c180 0x7fbc4810e620 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1553844960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc48103670 0x7fbc4810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1553844960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc48102d50 0x7fbc48103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1553844960 >> 172.21.15.28:0/1553844960 conn(0x7fbc480fc7d0 msgr2=0x7fbc480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1553844960 shutdown_connections 2024-09-15T21:58:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.919+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1553844960 wait complete. 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 Processor -- start 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 -- start start 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc48102d50 0x7fbc481009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc48103670 0x7fbc48100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4810c180 0x7fbc48101430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc48111f80 con 0x7fbc48103670 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc48111e00 con 0x7fbc4810c180 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4ce5f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc48112100 con 0x7fbc48102d50 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc45d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc48103670 0x7fbc48100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc45d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc48103670 0x7fbc48100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54136/0 (socket says 172.21.15.28:54136) 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc45d9b640 1 -- 172.21.15.28:0/1534407191 learned_addr learned my addr 172.21.15.28:0/1534407191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc4659c640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc48102d50 0x7fbc481009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.920+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4810c180 0x7fbc48101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc45d9b640 1 -- 172.21.15.28:0/1534407191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc48102d50 msgr2=0x7fbc481009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc48102d50 0x7fbc481009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc45d9b640 1 -- 172.21.15.28:0/1534407191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4810c180 msgr2=0x7fbc48101430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4810c180 0x7fbc48101430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc45d9b640 1 -- 172.21.15.28:0/1534407191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc481a6970 con 0x7fbc48103670 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc4659c640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc48102d50 0x7fbc481009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4810c180 0x7fbc48101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc48103670 0x7fbc48100ef0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00eb10 tx=0x7fbc3c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc377fe640 1 -- 172.21.15.28:0/1534407191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c010430 con 0x7fbc48103670 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc481a6c60 con 0x7fbc48103670 2024-09-15T21:58:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc377fe640 1 -- 172.21.15.28:0/1534407191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc3c010ab0 con 0x7fbc48103670 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc377fe640 1 -- 172.21.15.28:0/1534407191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c018b90 con 0x7fbc48103670 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.921+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc481a71a0 con 0x7fbc48103670 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.923+0000 7fbc377fe640 1 -- 172.21.15.28:0/1534407191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fbc3c010c60 con 0x7fbc48103670 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 Processor -- start 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 -- start start 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc377fe640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbc300a33b0 0x7fbc300a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954f9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d9954f9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d995506760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d99545f2f0 con 0x55d9954f95d0 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d995406cf0 con 0x55d995504370 2024-09-15T21:58:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.924+0000 7fbc4f238740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d99545b2b0 con 0x55d9954fab40 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc377fe640 1 -- 172.21.15.28:0/1534407191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fbc3c0d0640 con 0x7fbc48103670 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbc300a33b0 0x7fbc300a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d9954f9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d9954f9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54148/0 (socket says 172.21.15.28:54148) 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc4659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954f9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 -- 172.21.15.28:0/2813755408 learned_addr learned my addr 172.21.15.28:0/2813755408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d995506760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 -- 172.21.15.28:0/2813755408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 msgr2=0x55d9954f9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954f9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 -- 172.21.15.28:0/2813755408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 msgr2=0x55d995506760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d995506760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 -- 172.21.15.28:0/2813755408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9953eb3a0 con 0x55d9954f95d0 2024-09-15T21:58:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d995506760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d9954f9a10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0a4ca0 tx=0x7fbc3c09c160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f4fd3ad46a358f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbc300a33b0 0x7fbc300a5870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc40031550 tx=0x7fbc400058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.925+0000 7fbc34ff9640 1 -- 172.21.15.28:0/2813755408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c0a0910 con 0x55d9954f95d0 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc34ff9640 1 -- 172.21.15.28:0/2813755408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc3c0a51d0 con 0x55d9954f95d0 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc34ff9640 1 -- 172.21.15.28:0/2813755408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c09b9b0 con 0x55d9954f95d0 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc4f238740 1 -- 172.21.15.28:0/2813755408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 msgr2=0x55d9954f9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc4f238740 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d9954f9a10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0a4ca0 tx=0x7fbc3c09c160 comp rx=0 tx=0).stop 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc4f238740 1 -- 172.21.15.28:0/2813755408 shutdown_connections 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc4f238740 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d995506760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc4f238740 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d9954f9a10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc4f238740 1 --2- 172.21.15.28:0/2813755408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.926+0000 7fbc4f238740 1 -- 172.21.15.28:0/2813755408 >> 172.21.15.28:0/2813755408 conn(0x55d9954c2310 msgr2=0x55d9954f5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 -- 172.21.15.28:0/2813755408 shutdown_connections 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 -- 172.21.15.28:0/2813755408 wait complete. 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 -- start start 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 Processor -- start 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954f95d0 0x55d995503cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954fab40 0x55d995500f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d9955014d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d995507010 con 0x55d9954fab40 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d99541a060 con 0x55d995504370 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d995507190 con 0x55d9954f95d0 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc4f238740 10 client.0 _ll_register_callbacks cb 0x55d99516d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc45d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954fab40 0x55d995500f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc45d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954fab40 0x55d995500f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54160/0 (socket says 172.21.15.28:54160) 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.927+0000 7fbc45d9b640 1 -- 172.21.15.28:0/752950555 learned_addr learned my addr 172.21.15.28:0/752950555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d9955014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc4659c640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954f95d0 0x55d995503cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc45d9b640 1 -- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954f95d0 msgr2=0x55d995503cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954f95d0 0x55d995503cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc45d9b640 1 -- 172.21.15.28:0/752950555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 msgr2=0x55d9955014d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d9955014d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc45d9b640 1 -- 172.21.15.28:0/752950555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9951d03c0 con 0x55d9954fab40 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d9955014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954fab40 0x55d995500f70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00cf90 tx=0x7fbc3c098e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9729f4172a0afbe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc2dffb640 1 -- 172.21.15.28:0/752950555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c09b9b0 con 0x55d9954fab40 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc3c0a51d0 con 0x55d9954fab40 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc2dffb640 1 -- 172.21.15.28:0/752950555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fbc3c09bb50 con 0x55d9954fab40 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc2dffb640 1 -- 172.21.15.28:0/752950555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbc3c083050 con 0x55d9954fab40 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc4f238740 10 client.16014 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.928+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d995512a20 con 0x55d9954fab40 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc2dffb640 1 -- 172.21.15.28:0/752950555 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc3c0a51d0 con 0x55d9954fab40 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc2dffb640 1 -- 172.21.15.28:0/752950555 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fbc3c098440 con 0x55d9954fab40 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4f238740 10 client.16014 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4f238740 10 client.16014 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4f238740 20 client.16014 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4f238740 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d995513d70 0x55d995516160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4f238740 4 client.16014 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d99546c860 con 0x55d995513d70 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4659c640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d995513d70 0x55d995516160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc2dffb640 10 client.16014 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.929+0000 7fbc4659c640 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d995513d70 0x55d995516160 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1090b170d4a0704f server_cookie=a65d62cf609134ac in_seq=0 out_seq=0 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc2dffb640 1 -- 172.21.15.28:0/752950555 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d99546c860 con 0x55d995513d70 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc2dffb640 10 client.16014 handle_command_reply: tid=0 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 client.16014 shutdown 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954fab40 msgr2=0x55d995500f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954fab40 0x55d995500f70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00cf90 tx=0x7fbc3c098e00 comp rx=0 tx=0).stop 2024-09-15T21:58:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 shutdown_connections 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d995513d70 0x55d995516160 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d9955014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954fab40 0x55d995500f70 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 --2- 172.21.15.28:0/752950555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954f95d0 0x55d995503cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc2dffb640 0 client.16014 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.930+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 >> 172.21.15.28:0/752950555 conn(0x55d9954c2310 msgr2=0x55d9954fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.931+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 shutdown_connections 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.931+0000 7fbc4f238740 1 -- 172.21.15.28:0/752950555 wait complete. 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.931+0000 7fbc4f238740 20 client.16014 trim_cache size 0 max 16384 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 Processor -- start 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 -- start start 2024-09-15T21:58:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9954f95d0 0x55d995500ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954fc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d995504370 0x55d9954fc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc481b1110 con 0x55d995504370 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc481b2cb0 con 0x55d9954f95d0 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4f238740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc481b17e0 con 0x55d9954fab40 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d995504370 0x55d9954fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc4659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9954f95d0 0x55d995500ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc45d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d995504370 0x55d9954fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54176/0 (socket says 172.21.15.28:54176) 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.944+0000 7fbc46d9d640 1 -- 172.21.15.28:0/277518404 learned_addr learned my addr 172.21.15.28:0/277518404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc46d9d640 1 -- 172.21.15.28:0/277518404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 msgr2=0x55d9954fc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/277518404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954fc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc46d9d640 1 -- 172.21.15.28:0/277518404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9954f95d0 msgr2=0x55d995500ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/277518404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9954f95d0 0x55d995500ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc46d9d640 1 -- 172.21.15.28:0/277518404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9954f6730 con 0x55d995504370 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/277518404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d995504370 0x55d9954fc660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbc4002a9a0 tx=0x7fbc4003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07ce632edea6c8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc2ffff640 1 -- 172.21.15.28:0/277518404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40035040 con 0x55d995504370 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc2ffff640 1 -- 172.21.15.28:0/277518404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc400044c0 con 0x55d995504370 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.945+0000 7fbc2ffff640 1 -- 172.21.15.28:0/277518404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400549e0 con 0x55d995504370 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 -- 172.21.15.28:0/277518404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d995504370 msgr2=0x55d9954fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 --2- 172.21.15.28:0/277518404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d995504370 0x55d9954fc660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbc4002a9a0 tx=0x7fbc4003d000 comp rx=0 tx=0).stop 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 -- 172.21.15.28:0/277518404 shutdown_connections 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 --2- 172.21.15.28:0/277518404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d995504370 0x55d9954fc660 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 --2- 172.21.15.28:0/277518404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d9954fc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 --2- 172.21.15.28:0/277518404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d9954f95d0 0x55d995500ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 -- 172.21.15.28:0/277518404 >> 172.21.15.28:0/277518404 conn(0x55d9954c2310 msgr2=0x55d995502ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 -- 172.21.15.28:0/277518404 shutdown_connections 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.946+0000 7fbc4f238740 1 -- 172.21.15.28:0/277518404 wait complete. 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 -- start start 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 Processor -- start 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d995516670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d995516bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d99557c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc481b2580 con 0x55d9954f95d0 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc481b1eb0 con 0x55d995504370 2024-09-15T21:58:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc481af0f0 con 0x55d9954fab40 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d995516670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d995516670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54190/0 (socket says 172.21.15.28:54190) 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4659c640 1 -- 172.21.15.28:0/1723689416 learned_addr learned my addr 172.21.15.28:0/1723689416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d995516bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc4f238740 10 client.0 _ll_register_callbacks cb 0x7fbc3c0cd940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.947+0000 7fbc46d9d640 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d99557c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4659c640 1 -- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 msgr2=0x55d995516bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4659c640 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d995516bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4659c640 1 -- 172.21.15.28:0/1723689416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 msgr2=0x55d99557c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4659c640 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d99557c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4659c640 1 -- 172.21.15.28:0/1723689416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d99557cbe0 con 0x55d9954f95d0 2024-09-15T21:58:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4659c640 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d995516670 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x55d9955072e0 tx=0x7fbc38010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9a638399912774 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc2dffb640 1 -- 172.21.15.28:0/1723689416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3800fd40 con 0x55d9954f95d0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc2dffb640 1 -- 172.21.15.28:0/1723689416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fbc38004030 con 0x55d9954f95d0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d995466d70 con 0x55d9954f95d0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc2dffb640 1 -- 172.21.15.28:0/1723689416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbc38030050 con 0x55d9954f95d0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4f238740 10 client.16020 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d99557def0 con 0x55d9954f95d0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.948+0000 7fbc2dffb640 1 -- 172.21.15.28:0/1723689416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d995466d70 con 0x55d9954f95d0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc2dffb640 1 -- 172.21.15.28:0/1723689416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fbc38025020 con 0x55d9954f95d0 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc4f238740 10 client.16020 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc4f238740 10 client.16020 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc4f238740 20 client.16020 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc4f238740 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d99557e5c0 0x55d9955809b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc4f238740 4 client.16020 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9953e3170 con 0x55d99557e5c0 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d99557e5c0 0x55d9955809b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc2dffb640 10 client.16020 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.949+0000 7fbc45d9b640 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d99557e5c0 0x55d9955809b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37870d4c28d8b9df server_cookie=c6184d1eb2804200 in_seq=0 out_seq=0 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc2dffb640 1 -- 172.21.15.28:0/1723689416 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9953e3170 con 0x55d99557e5c0 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc2dffb640 10 client.16020 handle_command_reply: tid=0 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 client.16020 shutdown 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 msgr2=0x55d995516670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d995516670 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x55d9955072e0 tx=0x7fbc38010d20 comp rx=0 tx=0).stop 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 shutdown_connections 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55d99557e5c0 0x55d9955809b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d995504370 0x55d99557c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9954fab40 0x55d995516bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc2dffb640 0 client.16020 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T21:58:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 --2- 172.21.15.28:0/1723689416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9954f95d0 0x55d995516670 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 >> 172.21.15.28:0/1723689416 conn(0x55d9954c2310 msgr2=0x55d9954ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:58.092 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:58:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 shutdown_connections 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 1 -- 172.21.15.28:0/1723689416 wait complete. 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.950+0000 7fbc4f238740 20 client.16020 trim_cache size 0 max 16384 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbc300a33b0 msgr2=0x7fbc300a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbc300a33b0 0x7fbc300a5870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc40031550 tx=0x7fbc400058d0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc48103670 msgr2=0x7fbc48100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc48103670 0x7fbc48100ef0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00eb10 tx=0x7fbc3c00c2d0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 shutdown_connections 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbc300a33b0 0x7fbc300a5870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4810c180 0x7fbc48101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc48103670 0x7fbc48100ef0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 --2- 172.21.15.28:0/1534407191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc48102d50 0x7fbc481009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 >> 172.21.15.28:0/1534407191 conn(0x7fbc480fc7d0 msgr2=0x7fbc480fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 shutdown_connections 2024-09-15T21:58:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:57.951+0000 7fbc4ce5f640 1 -- 172.21.15.28:0/1534407191 wait complete. 2024-09-15T21:58:58.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T21:58:59.253 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:58:59.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:58:59 smithi028 ceph-mon[30969]: pgmap v52: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 505 KiB/s wr, 206 op/s 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 -- 172.21.15.28:0/4140885776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410e4d0 msgr2=0x7f03341108e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 --2- 172.21.15.28:0/4140885776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410e4d0 0x7f03341108e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f032c009ca0 tx=0x7f032c02a8c0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 -- 172.21.15.28:0/4140885776 shutdown_connections 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 --2- 172.21.15.28:0/4140885776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033410e4d0 0x7f03341108e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 --2- 172.21.15.28:0/4140885776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334101360 0x7f033410df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 --2- 172.21.15.28:0/4140885776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03341009b0 0x7f0334100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 -- 172.21.15.28:0/4140885776 >> 172.21.15.28:0/4140885776 conn(0x7f03340fc7d0 msgr2=0x7f03340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 -- 172.21.15.28:0/4140885776 shutdown_connections 2024-09-15T21:58:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.887+0000 7f033c220640 1 -- 172.21.15.28:0/4140885776 wait complete. 2024-09-15T21:58:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.888+0000 7f033c220640 1 Processor -- start 2024-09-15T21:58:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.888+0000 7f033c220640 1 -- start start 2024-09-15T21:58:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.889+0000 7f033c220640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03341009b0 0x7f03341a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.889+0000 7f033c220640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334101360 0x7f03341a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.889+0000 7f033c220640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033410e4d0 0x7f033419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.889+0000 7f033c220640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03341142f0 con 0x7f03341009b0 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.889+0000 7f033c220640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0334114170 con 0x7f033410e4d0 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.889+0000 7f033c220640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0334114470 con 0x7f0334101360 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.889+0000 7f0339fbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03341009b0 0x7f03341a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.890+0000 7f0339fbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03341009b0 0x7f03341a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54208/0 (socket says 172.21.15.28:54208) 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.890+0000 7f0339fbc640 1 -- 172.21.15.28:0/1841735736 learned_addr learned my addr 172.21.15.28:0/1841735736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.890+0000 7f033a7bd640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033410e4d0 0x7f033419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.890+0000 7f03397bb640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334101360 0x7f03341a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.890+0000 7f0339fbc640 1 -- 172.21.15.28:0/1841735736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334101360 msgr2=0x7f03341a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.890+0000 7f0339fbc640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334101360 0x7f03341a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.891+0000 7f0339fbc640 1 -- 172.21.15.28:0/1841735736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033410e4d0 msgr2=0x7f033419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.891+0000 7f0339fbc640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033410e4d0 0x7f033419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.891+0000 7f0339fbc640 1 -- 172.21.15.28:0/1841735736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033419ad10 con 0x7f03341009b0 2024-09-15T21:58:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.891+0000 7f0339fbc640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03341009b0 0x7f03341a0420 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f033000b4a0 tx=0x7f033000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.891+0000 7f032affd640 1 -- 172.21.15.28:0/1841735736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330014070 con 0x7f03341009b0 2024-09-15T21:58:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.891+0000 7f032affd640 1 -- 172.21.15.28:0/1841735736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0330010af0 con 0x7f03341009b0 2024-09-15T21:58:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.891+0000 7f032affd640 1 -- 172.21.15.28:0/1841735736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330017d10 con 0x7f03341009b0 2024-09-15T21:58:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.892+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f033419b000 con 0x7f03341009b0 2024-09-15T21:58:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.892+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03341a72f0 con 0x7f03341009b0 2024-09-15T21:58:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.895+0000 7f032affd640 1 -- 172.21.15.28:0/1841735736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f0330017070 con 0x7f03341009b0 2024-09-15T21:58:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.895+0000 7f033e5f9740 1 Processor -- start 2024-09-15T21:58:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033e5f9740 1 -- start start 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033e5f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620749312f0 con 0x5620749cb5d0 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033e5f9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5620748d8cf0 con 0x5620749ccb40 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033e5f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56207492d2b0 con 0x5620749d6370 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f032affd640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f03397bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f03397bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:45838/0 (socket says 172.21.15.28:45838) 2024-09-15T21:58:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f03397bb640 1 -- 172.21.15.28:0/2625501128 learned_addr learned my addr 172.21.15.28:0/2625501128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f0339fbc640 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f033a7bd640 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.896+0000 7f032affd640 1 -- 172.21.15.28:0/1841735736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f03300cf260 con 0x7f03341009b0 2024-09-15T21:58:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.897+0000 7f0339fbc640 1 -- 172.21.15.28:0/2625501128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 msgr2=0x5620749d8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.897+0000 7f0339fbc640 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.897+0000 7f0339fbc640 1 -- 172.21.15.28:0/2625501128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 msgr2=0x5620749cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.897+0000 7f0339fbc640 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.897+0000 7f0339fbc640 1 -- 172.21.15.28:0/2625501128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620748bd3a0 con 0x5620749cb5d0 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.898+0000 7f0339fbc640 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749cba10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f03300cc760 tx=0x7f0330017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0fdce510a0bf42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.898+0000 7f033a7bd640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.898+0000 7f031ffff640 1 -- 172.21.15.28:0/2625501128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033001e070 con 0x5620749cb5d0 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.898+0000 7f031ffff640 1 -- 172.21.15.28:0/2625501128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033009b9f0 con 0x5620749cb5d0 2024-09-15T21:58:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.898+0000 7f031ffff640 1 -- 172.21.15.28:0/2625501128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03300a3f60 con 0x5620749cb5d0 2024-09-15T21:58:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 -- 172.21.15.28:0/2625501128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 msgr2=0x5620749cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749cba10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f03300cc760 tx=0x7f0330017730 comp rx=0 tx=0).stop 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033a7bd640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f032c006010 tx=0x7f032c0095c0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 -- 172.21.15.28:0/2625501128 shutdown_connections 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749cba10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2625501128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 -- 172.21.15.28:0/2625501128 >> 172.21.15.28:0/2625501128 conn(0x562074994310 msgr2=0x5620749c7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 -- 172.21.15.28:0/2625501128 shutdown_connections 2024-09-15T21:58:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.900+0000 7f033e5f9740 1 -- 172.21.15.28:0/2625501128 wait complete. 2024-09-15T21:58:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.901+0000 7f033e5f9740 1 -- start start 2024-09-15T21:58:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.901+0000 7f033e5f9740 1 Processor -- start 2024-09-15T21:58:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.902+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749d5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.902+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.902+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f033e5f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620749d9070 con 0x5620749cb5d0 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f03397bb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f03397bb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:51750/0 (socket says 172.21.15.28:51750) 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f03397bb640 1 -- 172.21.15.28:0/3604764569 learned_addr learned my addr 172.21.15.28:0/3604764569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f0339fbc640 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749d5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f033a7bd640 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5620748ec060 con 0x5620749ccb40 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.903+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5620749d91f0 con 0x5620749d6370 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f033e5f9740 10 client.0 _ll_register_callbacks cb 0x56207463f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f03397bb640 1 -- 172.21.15.28:0/3604764569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 msgr2=0x5620749d37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f03397bb640 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f03397bb640 1 -- 172.21.15.28:0/3604764569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 msgr2=0x5620749d5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f03397bb640 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749d5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f03397bb640 1 -- 172.21.15.28:0/3604764569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620746a23c0 con 0x5620749ccb40 2024-09-15T21:58:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f033009b9f0 con 0x5620749ccb40 2024-09-15T21:58:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f03397bb640 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d30a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f03240126b0 tx=0x7f0324012ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b693313318bc1a35 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.904+0000 7f031d7fa640 10 client.25625 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:58:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.905+0000 7f031d7fa640 1 -- 172.21.15.28:0/3604764569 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240118d0 con 0x5620749ccb40 2024-09-15T21:58:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.905+0000 7f031d7fa640 1 -- 172.21.15.28:0/3604764569 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f0324011a70 con 0x5620749ccb40 2024-09-15T21:58:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.905+0000 7f031d7fa640 1 -- 172.21.15.28:0/3604764569 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0324011cb0 con 0x5620749ccb40 2024-09-15T21:58:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.906+0000 7f033e5f9740 10 client.25625 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.906+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5620749e4b10 con 0x5620749ccb40 2024-09-15T21:58:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f031d7fa640 1 -- 172.21.15.28:0/3604764569 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f0324026470 con 0x5620749ccb40 2024-09-15T21:58:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f033e5f9740 10 client.25625 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f033e5f9740 10 client.25625 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T21:58:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f033e5f9740 20 client.25625 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5620749e5ec0 0x5620749e82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f033e5f9740 4 client.25625 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:58:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56207493e860 con 0x5620749e5ec0 2024-09-15T21:58:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.907+0000 7f0339fbc640 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5620749e5ec0 0x5620749e82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.908+0000 7f031d7fa640 10 client.25625 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.908+0000 7f0339fbc640 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5620749e5ec0 0x5620749e82b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b1450c54934790f3 server_cookie=bb3111823827a46d in_seq=0 out_seq=0 2024-09-15T21:58:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.908+0000 7f031d7fa640 1 -- 172.21.15.28:0/3604764569 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56207493e860 con 0x5620749e5ec0 2024-09-15T21:58:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.908+0000 7f031d7fa640 10 client.25625 handle_command_reply: tid=0 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.908+0000 7f033e5f9740 1 client.25625 shutdown 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 msgr2=0x5620749d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d30a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f03240126b0 tx=0x7f0324012ab0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 shutdown_connections 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749d37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d30a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5620749e5ec0 0x5620749e82b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3604764569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749d5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 >> 172.21.15.28:0/3604764569 conn(0x562074994310 msgr2=0x5620749cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f031d7fa640 0 client.25625 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 shutdown_connections 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.909+0000 7f033e5f9740 1 -- 172.21.15.28:0/3604764569 wait complete. 2024-09-15T21:58:59.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.910+0000 7f033e5f9740 20 client.25625 trim_cache size 0 max 16384 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 Processor -- start 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 -- start start 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749cb5d0 0x5620749e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749ccb40 0x5620749ce120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749d6370 0x5620749ce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03341b1280 con 0x5620749d6370 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03341b2e20 con 0x5620749cb5d0 2024-09-15T21:58:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.928+0000 7f033e5f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03341b1950 con 0x5620749ccb40 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749cb5d0 0x5620749e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749cb5d0 0x5620749e2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:51754/0 (socket says 172.21.15.28:51754) 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 -- 172.21.15.28:0/2282459778 learned_addr learned my addr 172.21.15.28:0/2282459778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f033a7bd640 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749d6370 0x5620749ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f03397bb640 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749ccb40 0x5620749ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 -- 172.21.15.28:0/2282459778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749ccb40 msgr2=0x5620749ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749ccb40 0x5620749ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 -- 172.21.15.28:0/2282459778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749d6370 msgr2=0x5620749ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749d6370 0x5620749ce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 -- 172.21.15.28:0/2282459778 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620749d15f0 con 0x5620749cb5d0 2024-09-15T21:58:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.929+0000 7f0339fbc640 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749cb5d0 0x5620749e2310 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f033009bba0 tx=0x7f03300bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32c38031371d1a2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.930+0000 7f031f7fe640 1 -- 172.21.15.28:0/2282459778 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033001e040 con 0x5620749cb5d0 2024-09-15T21:58:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.930+0000 7f031f7fe640 1 -- 172.21.15.28:0/2282459778 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03300a3f60 con 0x5620749cb5d0 2024-09-15T21:58:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.930+0000 7f031f7fe640 1 -- 172.21.15.28:0/2282459778 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330095e60 con 0x5620749cb5d0 2024-09-15T21:58:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 -- 172.21.15.28:0/2282459778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749cb5d0 msgr2=0x5620749e2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749cb5d0 0x5620749e2310 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f033009bba0 tx=0x7f03300bb040 comp rx=0 tx=0).stop 2024-09-15T21:58:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 -- 172.21.15.28:0/2282459778 shutdown_connections 2024-09-15T21:58:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749d6370 0x5620749ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749ccb40 0x5620749ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 --2- 172.21.15.28:0/2282459778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749cb5d0 0x5620749e2310 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 -- 172.21.15.28:0/2282459778 >> 172.21.15.28:0/2282459778 conn(0x562074994310 msgr2=0x5620749d2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 -- 172.21.15.28:0/2282459778 shutdown_connections 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 -- 172.21.15.28:0/2282459778 wait complete. 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 -- start start 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.931+0000 7f033e5f9740 1 Processor -- start 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749de930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f033e5f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749da1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f033e5f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03341b26f0 con 0x5620749cb5d0 2024-09-15T21:58:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f033e5f9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03341b2020 con 0x5620749ccb40 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f033e5f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03341af260 con 0x5620749d6370 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f033e5f9740 10 client.0 _ll_register_callbacks cb 0x7f0324009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f0339fbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f0339fbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749de930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54252/0 (socket says 172.21.15.28:54252) 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.932+0000 7f0339fbc640 1 -- 172.21.15.28:0/3476376990 learned_addr learned my addr 172.21.15.28:0/3476376990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f03397bb640 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f0339fbc640 1 -- 172.21.15.28:0/3476376990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 msgr2=0x5620749da1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f0339fbc640 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749da1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f0339fbc640 1 -- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 msgr2=0x5620749d9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f0339fbc640 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f0339fbc640 1 -- 172.21.15.28:0/3476376990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620749da760 con 0x5620749cb5d0 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f0339fbc640 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749de930 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f03300995a0 tx=0x7f03300ccb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7633a46a18118621 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:58:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f031d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:58:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f031d7fa640 1 -- 172.21.15.28:0/3476376990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033001e070 con 0x5620749cb5d0 2024-09-15T21:58:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.933+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03300a3f60 con 0x5620749cb5d0 2024-09-15T21:58:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.934+0000 7f031d7fa640 1 -- 172.21.15.28:0/3476376990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f03300b4040 con 0x5620749cb5d0 2024-09-15T21:58:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.934+0000 7f031d7fa640 1 -- 172.21.15.28:0/3476376990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0330095e60 con 0x5620749cb5d0 2024-09-15T21:58:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.934+0000 7f033e5f9740 10 client.16038 fetch_fsmap learned FSMap version 30 2024-09-15T21:58:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.934+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562074a50860 con 0x5620749cb5d0 2024-09-15T21:58:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.934+0000 7f031d7fa640 1 -- 172.21.15.28:0/3476376990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03300a3f60 con 0x5620749cb5d0 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.935+0000 7f031d7fa640 1 -- 172.21.15.28:0/3476376990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f033009a740 con 0x5620749cb5d0 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.935+0000 7f033e5f9740 10 client.16038 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.935+0000 7f033e5f9740 10 client.16038 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.935+0000 7f033e5f9740 20 client.16038 populate_metadata read hostname 'smithi028' 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.935+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x562074a50a30 0x562074a52e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.935+0000 7f033e5f9740 4 client.16038 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.935+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562074938d70 con 0x562074a50a30 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.936+0000 7f03397bb640 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x562074a50a30 0x562074a52e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.938+0000 7f031d7fa640 10 client.16038 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.938+0000 7f03397bb640 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x562074a50a30 0x562074a52e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7bc9be4c84b3eed5 server_cookie=708e521bdd29cbcb in_seq=0 out_seq=0 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.939+0000 7f031d7fa640 1 -- 172.21.15.28:0/3476376990 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562074938d70 con 0x562074a50a30 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.939+0000 7f031d7fa640 10 client.16038 handle_command_reply: tid=0 2024-09-15T21:58:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.939+0000 7f033e5f9740 1 client.16038 shutdown 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.939+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 msgr2=0x5620749de930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.939+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749de930 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f03300995a0 tx=0x7f03300ccb60 comp rx=0 tx=0).stop 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.940+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 shutdown_connections 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.940+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5620749d6370 0x5620749da1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.940+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5620749ccb40 0x5620749d9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.940+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x562074a50a30 0x562074a52e20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.940+0000 7f033e5f9740 1 --2- 172.21.15.28:0/3476376990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620749cb5d0 0x5620749de930 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.940+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 >> 172.21.15.28:0/3476376990 conn(0x562074994310 msgr2=0x5620749c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.940+0000 7f031d7fa640 0 client.16038 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T21:58:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.941+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 shutdown_connections 2024-09-15T21:58:59.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.942+0000 7f033e5f9740 1 -- 172.21.15.28:0/3476376990 wait complete. 2024-09-15T21:58:59.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.942+0000 7f033e5f9740 20 client.16038 trim_cache size 0 max 16384 2024-09-15T21:58:59.945 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 msgr2=0x7f03200a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f032c006010 tx=0x7f032c0095c0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03341009b0 msgr2=0x7f03341a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03341009b0 0x7f03341a0420 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f033000b4a0 tx=0x7f033000b960 comp rx=0 tx=0).stop 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 shutdown_connections 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f033410e4d0 0x7f033419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334101360 0x7f03341a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 --2- 172.21.15.28:0/1841735736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03341009b0 0x7f03341a0420 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.944+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 >> 172.21.15.28:0/1841735736 conn(0x7f03340fc7d0 msgr2=0x7f03340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.945+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 shutdown_connections 2024-09-15T21:58:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:58:59.945+0000 7f033c220640 1 -- 172.21.15.28:0/1841735736 wait complete. 2024-09-15T21:58:59.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:58:59 smithi203 ceph-mon[35546]: pgmap v52: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 505 KiB/s wr, 206 op/s 2024-09-15T21:59:00.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:58:59 smithi174 ceph-mon[35244]: pgmap v52: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 505 KiB/s wr, 206 op/s 2024-09-15T21:59:00.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T21:59:01.340 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:59:01.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:01 smithi028 ceph-mon[30969]: pgmap v53: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 505 KiB/s wr, 206 op/s 2024-09-15T21:59:01.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:01 smithi203 ceph-mon[35546]: pgmap v53: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 505 KiB/s wr, 206 op/s 2024-09-15T21:59:02.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:01 smithi174 ceph-mon[35244]: pgmap v53: 129 pgs: 129 active+clean; 7.2 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 505 KiB/s wr, 206 op/s 2024-09-15T21:59:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 -- 172.21.15.28:0/2000716706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 msgr2=0x7fa338101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/2000716706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338101200 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa328009930 tx=0x7fa32802a4d0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 -- 172.21.15.28:0/2000716706 shutdown_connections 2024-09-15T21:59:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/2000716706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa338102ef0 0x7fa33810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/2000716706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa338102570 0x7fa3381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/2000716706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338101200 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 -- 172.21.15.28:0/2000716706 >> 172.21.15.28:0/2000716706 conn(0x7fa3380fc7d0 msgr2=0x7fa3380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 -- 172.21.15.28:0/2000716706 shutdown_connections 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.166+0000 7fa33e4ea640 1 -- 172.21.15.28:0/2000716706 wait complete. 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.167+0000 7fa33e4ea640 1 Processor -- start 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.167+0000 7fa33e4ea640 1 -- start start 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa33e4ea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa33e4ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa338102570 0x7fa338198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa33e4ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa338102ef0 0x7fa33819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa33e4ea640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381120d0 con 0x7fa338102570 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa33e4ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa338111f50 con 0x7fa338102ef0 2024-09-15T21:59:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa33e4ea640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa338112250 con 0x7fa338100e20 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa337fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa338102570 0x7fa338198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa337fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:39964/0 (socket says 172.21.15.28:39964) 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa337fff640 1 -- 172.21.15.28:0/3655151061 learned_addr learned my addr 172.21.15.28:0/3655151061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa33ca87640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa338102ef0 0x7fa33819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa337fff640 1 -- 172.21.15.28:0/3655151061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa338102ef0 msgr2=0x7fa33819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.168+0000 7fa337fff640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa338102ef0 0x7fa33819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa337fff640 1 -- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa338102570 msgr2=0x7fa338198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa337fff640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa338102570 0x7fa338198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa337fff640 1 -- 172.21.15.28:0/3655151061 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33819d540 con 0x7fa338100e20 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa33ca87640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa338102ef0 0x7fa33819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa337fff640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338198590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa32802a9f0 tx=0x7fa328003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa3357fa640 1 -- 172.21.15.28:0/3655151061 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32803f070 con 0x7fa338100e20 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa3357fa640 1 -- 172.21.15.28:0/3655151061 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa328031c90 con 0x7fa338100e20 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa3357fa640 1 -- 172.21.15.28:0/3655151061 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328035030 con 0x7fa338100e20 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33819d7d0 con 0x7fa338100e20 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.169+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa33819dc60 con 0x7fa338100e20 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.172+0000 7fa3357fa640 1 -- 172.21.15.28:0/3655151061 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fa328042070 con 0x7fa338100e20 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.172+0000 7fa3357fa640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa3200a3430 0x7fa3200a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3357fa640 1 -- 172.21.15.28:0/3655151061 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa3280f0b20 con 0x7fa338100e20 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 Processor -- start 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 -- start start 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3377fe640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa3200a3430 0x7fa3200a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 0x563d5b9e98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9ea220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9f4b80 0x563d5b9f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563d5b9472f0 con 0x563d5b9f4b80 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563d5b8eecf0 con 0x563d5b9e9de0 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.173+0000 7fa3408c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563d5b9432b0 con 0x563d5b9eb350 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa337fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 0x563d5b9e98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa3200a3430 0x7fa3200a58f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3380fede0 tx=0x7fa32c00f7a0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9ea220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:36754/0 (socket says 172.21.15.28:36754) 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 -- 172.21.15.28:0/1397828677 learned_addr learned my addr 172.21.15.28:0/1397828677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 -- 172.21.15.28:0/1397828677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 msgr2=0x563d5b9e98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 --2- 172.21.15.28:0/1397828677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 0x563d5b9e98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 -- 172.21.15.28:0/1397828677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9f4b80 msgr2=0x563d5b9f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 --2- 172.21.15.28:0/1397828677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9f4b80 0x563d5b9f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.174+0000 7fa3377fe640 1 -- 172.21.15.28:0/1397828677 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d5b8d33a0 con 0x563d5b9e9de0 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.175+0000 7fa3377fe640 1 --2- 172.21.15.28:0/1397828677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9ea220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00f4c0 tx=0x7fa32c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26461c3af0eb54f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.175+0000 7fa31e7fc640 1 -- 172.21.15.28:0/1397828677 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c016070 con 0x563d5b9e9de0 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.176+0000 7fa31e7fc640 1 -- 172.21.15.28:0/1397828677 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32c01d910 con 0x563d5b9e9de0 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.176+0000 7fa31e7fc640 1 -- 172.21.15.28:0/1397828677 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c01a9a0 con 0x563d5b9e9de0 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.176+0000 7fa3408c3740 1 -- 172.21.15.28:0/1397828677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 msgr2=0x563d5b9ea220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.176+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1397828677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9ea220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00f4c0 tx=0x7fa32c013000 comp rx=0 tx=0).stop 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 -- 172.21.15.28:0/1397828677 shutdown_connections 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1397828677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9f4b80 0x563d5b9f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1397828677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9ea220 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1397828677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 0x563d5b9e98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 -- 172.21.15.28:0/1397828677 >> 172.21.15.28:0/1397828677 conn(0x563d5b9aa310 msgr2=0x563d5b9e6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 -- 172.21.15.28:0/1397828677 shutdown_connections 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 -- 172.21.15.28:0/1397828677 wait complete. 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.177+0000 7fa3408c3740 1 -- start start 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 1 Processor -- start 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9f4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9f2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563d5b9dda20 con 0x563d5b9eb350 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563d5b902060 con 0x563d5b9e9de0 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563d5b9ddba0 con 0x563d5b9f4b80 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.178+0000 7fa3408c3740 10 client.0 _ll_register_callbacks cb 0x563d5b6558c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9f4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:36768/0 (socket says 172.21.15.28:36768) 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 -- 172.21.15.28:0/75533111 learned_addr learned my addr 172.21.15.28:0/75533111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa3377fe640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa33ca87640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 -- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 msgr2=0x563d5b9f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 -- 172.21.15.28:0/75533111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 msgr2=0x563d5b9f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9f18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa337fff640 1 -- 172.21.15.28:0/75533111 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d5b6b83c0 con 0x563d5b9e9de0 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.179+0000 7fa33ca87640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa337fff640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9f4620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa328009900 tx=0x7fa32803ffb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35f438f911c09d9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa3177fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:59:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa3177fe640 1 -- 172.21.15.28:0/75533111 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328049040 con 0x563d5b9e9de0 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa3177fe640 1 -- 172.21.15.28:0/75533111 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa3280ee130 con 0x563d5b9e9de0 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa32c01d910 con 0x563d5b9e9de0 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa3177fe640 1 -- 172.21.15.28:0/75533111 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa3280da050 con 0x563d5b9e9de0 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa3408c3740 10 client.25643 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.180+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563d5ba0bb80 con 0x563d5b9e9de0 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3177fe640 1 -- 172.21.15.28:0/75533111 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32c01d910 con 0x563d5b9e9de0 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3177fe640 1 -- 172.21.15.28:0/75533111 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fa3280bdc90 con 0x563d5b9e9de0 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3408c3740 10 client.25643 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3408c3740 10 client.25643 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3408c3740 20 client.25643 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3408c3740 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba0cf70 0x563d5ba0f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3408c3740 4 client.25643 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:59:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d5b954860 con 0x563d5ba0cf70 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.181+0000 7fa3377fe640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba0cf70 0x563d5ba0f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.182+0000 7fa3177fe640 10 client.25643 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.182+0000 7fa3377fe640 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba0cf70 0x563d5ba0f3a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=755b582faab18f server_cookie=82a68169785cb68e in_seq=0 out_seq=0 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.182+0000 7fa3177fe640 1 -- 172.21.15.28:0/75533111 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563d5b954860 con 0x563d5ba0cf70 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.182+0000 7fa3177fe640 10 client.25643 handle_command_reply: tid=0 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.182+0000 7fa3408c3740 1 client.25643 shutdown 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 msgr2=0x563d5b9f4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9f4620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa328009900 tx=0x7fa32803ffb0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 shutdown_connections 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba0cf70 0x563d5ba0f3a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 --2- 172.21.15.28:0/75533111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9f4620 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3177fe640 0 client.25643 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 >> 172.21.15.28:0/75533111 conn(0x563d5b9aa310 msgr2=0x563d5b9ec170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 shutdown_connections 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 1 -- 172.21.15.28:0/75533111 wait complete. 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.183+0000 7fa3408c3740 20 client.25643 trim_cache size 0 max 16384 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 Processor -- start 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 -- start start 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9e9de0 0x563d5b9e6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 0x563d5b9f08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9f4b80 0x563d5ba14c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381ace90 con 0x563d5b9e9de0 2024-09-15T21:59:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381aea30 con 0x563d5b9f4b80 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa3408c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381ad560 con 0x563d5b9eb350 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa337fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9e9de0 0x563d5b9e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa337fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9e9de0 0x563d5b9e6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40708/0 (socket says 172.21.15.28:40708) 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.197+0000 7fa337fff640 1 -- 172.21.15.28:0/465532673 learned_addr learned my addr 172.21.15.28:0/465532673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa337fff640 1 -- 172.21.15.28:0/465532673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 msgr2=0x563d5b9f08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa337fff640 1 --2- 172.21.15.28:0/465532673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 0x563d5b9f08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa337fff640 1 -- 172.21.15.28:0/465532673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9f4b80 msgr2=0x563d5ba14c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa337fff640 1 --2- 172.21.15.28:0/465532673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9f4b80 0x563d5ba14c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa337fff640 1 -- 172.21.15.28:0/465532673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d5b9efe40 con 0x563d5b9e9de0 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa337fff640 1 --2- 172.21.15.28:0/465532673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9e9de0 0x563d5b9e6580 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa3280b9e90 tx=0x7fa328003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fdb76b0e3634ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa31dffb640 1 -- 172.21.15.28:0/465532673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328049050 con 0x563d5b9e9de0 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa31dffb640 1 -- 172.21.15.28:0/465532673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3280bda30 con 0x563d5b9e9de0 2024-09-15T21:59:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.198+0000 7fa31dffb640 1 -- 172.21.15.28:0/465532673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3280bdc90 con 0x563d5b9e9de0 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 -- 172.21.15.28:0/465532673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9e9de0 msgr2=0x563d5b9e6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 --2- 172.21.15.28:0/465532673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9e9de0 0x563d5b9e6580 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa3280b9e90 tx=0x7fa328003f90 comp rx=0 tx=0).stop 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 -- 172.21.15.28:0/465532673 shutdown_connections 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 --2- 172.21.15.28:0/465532673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9f4b80 0x563d5ba14c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 --2- 172.21.15.28:0/465532673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9eb350 0x563d5b9f08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 --2- 172.21.15.28:0/465532673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9e9de0 0x563d5b9e6580 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 -- 172.21.15.28:0/465532673 >> 172.21.15.28:0/465532673 conn(0x563d5b9aa310 msgr2=0x563d5b9f01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 -- 172.21.15.28:0/465532673 shutdown_connections 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 -- 172.21.15.28:0/465532673 wait complete. 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 -- start start 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.199+0000 7fa3408c3740 1 Processor -- start 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9e44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3408c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9e0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3408c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381ae300 con 0x563d5b9eb350 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3408c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381adc30 con 0x563d5b9e9de0 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3408c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381aae70 con 0x563d5b9f4b80 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3408c3740 10 client.0 _ll_register_callbacks cb 0x7fa328009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9dfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40718/0 (socket says 172.21.15.28:40718) 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 -- 172.21.15.28:0/1259596882 learned_addr learned my addr 172.21.15.28:0/1259596882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 -- 172.21.15.28:0/1259596882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 msgr2=0x563d5b9e0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 -- 172.21.15.28:0/1259596882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 msgr2=0x563d5b9e44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa337fff640 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9e44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9e44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.200+0000 7fa3377fe640 1 -- 172.21.15.28:0/1259596882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d5b9e0980 con 0x563d5b9eb350 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa337fff640 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9e44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3377fe640 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9dfe00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00f490 tx=0x7fa32c00dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da56c63384aac83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3177fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3177fe640 1 -- 172.21.15.28:0/1259596882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c005040 con 0x563d5b9eb350 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3177fe640 1 -- 172.21.15.28:0/1259596882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa32c0051e0 con 0x563d5b9eb350 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3280bda30 con 0x563d5b9eb350 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3177fe640 1 -- 172.21.15.28:0/1259596882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa32c0053e0 con 0x563d5b9eb350 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3408c3740 10 client.16053 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.201+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563d5ba77a70 con 0x563d5b9eb350 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3177fe640 1 -- 172.21.15.28:0/1259596882 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa32c005580 con 0x563d5b9eb350 2024-09-15T21:59:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3177fe640 1 -- 172.21.15.28:0/1259596882 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fa32c038070 con 0x563d5b9eb350 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3408c3740 10 client.16053 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3408c3740 10 client.16053 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3408c3740 20 client.16053 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba780e0 0x563d5ba7a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3408c3740 4 client.16053 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.202+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa3280bda30 con 0x563d5ba780e0 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.203+0000 7fa337fff640 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba780e0 0x563d5ba7a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.205+0000 7fa337fff640 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba780e0 0x563d5ba7a500 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=513837273e7a81d4 server_cookie=25bc4cf0c3cdf906 in_seq=0 out_seq=0 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.205+0000 7fa3177fe640 10 client.16053 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.205+0000 7fa3177fe640 1 -- 172.21.15.28:0/1259596882 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa3280bda30 con 0x563d5ba780e0 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.205+0000 7fa3177fe640 10 client.16053 handle_command_reply: tid=0 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.205+0000 7fa3408c3740 1 client.16053 shutdown 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.205+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 msgr2=0x563d5b9dfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.205+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9dfe00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00f490 tx=0x7fa32c00dfa0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 shutdown_connections 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x563d5ba780e0 0x563d5ba7a500 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d5b9f4b80 0x563d5b9e0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d5b9eb350 0x563d5b9dfe00 secure :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00f490 tx=0x7fa32c00dfa0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3177fe640 0 client.16053 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 --2- 172.21.15.28:0/1259596882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d5b9e9de0 0x563d5b9e44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 >> 172.21.15.28:0/1259596882 conn(0x563d5b9aa310 msgr2=0x563d5b9eb920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 shutdown_connections 2024-09-15T21:59:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 1 -- 172.21.15.28:0/1259596882 wait complete. 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.206+0000 7fa3408c3740 20 client.16053 trim_cache size 0 max 16384 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.207+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa3200a3430 msgr2=0x7fa3200a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa3200a3430 0x7fa3200a58f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3380fede0 tx=0x7fa32c00f7a0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 msgr2=0x7fa338198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338198590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa32802a9f0 tx=0x7fa328003b90 comp rx=0 tx=0).stop 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 shutdown_connections 2024-09-15T21:59:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa3200a3430 0x7fa3200a58f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa338102ef0 0x7fa33819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa338102570 0x7fa338198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.208+0000 7fa33e4ea640 1 --2- 172.21.15.28:0/3655151061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa338100e20 0x7fa338198590 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.209+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 >> 172.21.15.28:0/3655151061 conn(0x7fa3380fc7d0 msgr2=0x7fa3380fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.209+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 shutdown_connections 2024-09-15T21:59:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:03.209+0000 7fa33e4ea640 1 -- 172.21.15.28:0/3655151061 wait complete. 2024-09-15T21:59:03.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:03 smithi028 ceph-mon[30969]: pgmap v54: 129 pgs: 129 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 124 MiB/s wr, 316 op/s 2024-09-15T21:59:03.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:03 smithi203 ceph-mon[35546]: pgmap v54: 129 pgs: 129 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 124 MiB/s wr, 316 op/s 2024-09-15T21:59:04.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:03 smithi174 ceph-mon[35244]: pgmap v54: 129 pgs: 129 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 124 MiB/s wr, 316 op/s 2024-09-15T21:59:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 -- 172.21.15.28:0/711886162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460101ac0 msgr2=0x7f2460101f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 --2- 172.21.15.28:0/711886162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460101ac0 0x7f2460101f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2454009900 tx=0x7f245402a410 comp rx=0 tx=0).stop 2024-09-15T21:59:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 -- 172.21.15.28:0/711886162 shutdown_connections 2024-09-15T21:59:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 --2- 172.21.15.28:0/711886162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2460102460 0x7f246010e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 --2- 172.21.15.28:0/711886162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460101ac0 0x7f2460101f20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 --2- 172.21.15.28:0/711886162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460107ad0 0x7f2460107eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 -- 172.21.15.28:0/711886162 >> 172.21.15.28:0/711886162 conn(0x7f24600fd820 msgr2=0x7f24600ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 -- 172.21.15.28:0/711886162 shutdown_connections 2024-09-15T21:59:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.025+0000 7f2465b6d640 1 -- 172.21.15.28:0/711886162 wait complete. 2024-09-15T21:59:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.026+0000 7f2465b6d640 1 Processor -- start 2024-09-15T21:59:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f2465b6d640 1 -- start start 2024-09-15T21:59:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f2465b6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460101ac0 0x7f2460198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f2465b6d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460102460 0x7f24601988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f2465b6d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2460107ad0 0x7f246019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f2465b6d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2460111ed0 con 0x7f2460101ac0 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f2465b6d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2460111d50 con 0x7f2460102460 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f2465b6d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2460112050 con 0x7f2460107ad0 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460101ac0 0x7f2460198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460102460 0x7f24601988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460101ac0 0x7f2460198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40730/0 (socket says 172.21.15.28:40730) 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 -- 172.21.15.28:0/581031098 learned_addr learned my addr 172.21.15.28:0/581031098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245ffff640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2460107ad0 0x7f246019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 -- 172.21.15.28:0/581031098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2460107ad0 msgr2=0x7f246019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2460107ad0 0x7f246019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 -- 172.21.15.28:0/581031098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460102460 msgr2=0x7f24601988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460102460 0x7f24601988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245f7fe640 1 -- 172.21.15.28:0/581031098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246019d370 con 0x7f2460101ac0 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245ffff640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2460107ad0 0x7f246019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.027+0000 7f245effd640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460102460 0x7f24601988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.028+0000 7f245f7fe640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460101ac0 0x7f2460198380 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f245000d950 tx=0x7f245000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.028+0000 7f245cff9640 1 -- 172.21.15.28:0/581031098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450017030 con 0x7f2460101ac0 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.028+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246019d660 con 0x7f2460101ac0 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.028+0000 7f245cff9640 1 -- 172.21.15.28:0/581031098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2450004020 con 0x7f2460101ac0 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.028+0000 7f245cff9640 1 -- 172.21.15.28:0/581031098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450005650 con 0x7f2460101ac0 2024-09-15T21:59:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.028+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2460074420 con 0x7f2460101ac0 2024-09-15T21:59:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.031+0000 7f245cff9640 1 -- 172.21.15.28:0/581031098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f24500041d0 con 0x7f2460101ac0 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.031+0000 7f24689e3740 1 Processor -- start 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f24689e3740 1 -- start start 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f24689e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f24689e3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f24689e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f24689e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6be3282f0 con 0x55e6be3c3af0 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f24689e3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e6be2d0270 con 0x55e6be3c4410 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f24689e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e6be3242b0 con 0x55e6be3cecb0 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245cff9640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f24480a34e0 0x7f24480a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:36818/0 (socket says 172.21.15.28:36818) 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 -- 172.21.15.28:0/1515308152 learned_addr learned my addr 172.21.15.28:0/1515308152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245effd640 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245ffff640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f24480a34e0 0x7f24480a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 -- 172.21.15.28:0/1515308152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 msgr2=0x55e6be3d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 -- 172.21.15.28:0/1515308152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 msgr2=0x55e6be3c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245cff9640 1 -- 172.21.15.28:0/581031098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f24500cf840 con 0x7f2460101ac0 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245f7fe640 1 -- 172.21.15.28:0/1515308152 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6be2cf340 con 0x55e6be3c4410 2024-09-15T21:59:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.032+0000 7f245ffff640 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T21:59:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.033+0000 7f245f7fe640 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3ce770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f245009b4c0 tx=0x7f2450014f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86d8658d3686d79e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.033+0000 7f2445ffb640 1 -- 172.21.15.28:0/1515308152 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245009b120 con 0x55e6be3c4410 2024-09-15T21:59:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.033+0000 7f245ffff640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f24480a34e0 0x7f24480a59a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2458009b80 tx=0x7f2458009500 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.033+0000 7f2445ffb640 1 -- 172.21.15.28:0/1515308152 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245009bec0 con 0x55e6be3c4410 2024-09-15T21:59:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.033+0000 7f2445ffb640 1 -- 172.21.15.28:0/1515308152 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24500a4430 con 0x55e6be3c4410 2024-09-15T21:59:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 -- 172.21.15.28:0/1515308152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 msgr2=0x55e6be3ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3ce770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f245009b4c0 tx=0x7f2450014f80 comp rx=0 tx=0).stop 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 -- 172.21.15.28:0/1515308152 shutdown_connections 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3ce770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 --2- 172.21.15.28:0/1515308152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 -- 172.21.15.28:0/1515308152 >> 172.21.15.28:0/1515308152 conn(0x55e6be3c0060 msgr2=0x55e6be3c0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 -- 172.21.15.28:0/1515308152 shutdown_connections 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.034+0000 7f24689e3740 1 -- 172.21.15.28:0/1515308152 wait complete. 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.035+0000 7f24689e3740 1 -- start start 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.035+0000 7f24689e3740 1 Processor -- start 2024-09-15T21:59:04.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.035+0000 7f24689e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3c3af0 0x55e6be3cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.035+0000 7f24689e3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.035+0000 7f24689e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.035+0000 7f245effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:36830/0 (socket says 172.21.15.28:36830) 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3c3af0 0x55e6be3cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3c3af0 0x55e6be3cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:40034/0 (socket says 172.21.15.28:40034) 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40742/0 (socket says 172.21.15.28:40742) 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 -- 172.21.15.28:0/3863103140 learned_addr learned my addr 172.21.15.28:0/3863103140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6be3d1820 con 0x55e6be3cecb0 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e6be2b1290 con 0x55e6be3c4410 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e6be3d19a0 con 0x55e6be3c3af0 2024-09-15T21:59:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 -- 172.21.15.28:0/3863103140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3c3af0 msgr2=0x55e6be3cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3c3af0 0x55e6be3cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 -- 172.21.15.28:0/3863103140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3cecb0 msgr2=0x55e6be3cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 -- 172.21.15.28:0/3863103140 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6be03b210 con 0x55e6be3c4410 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245f7fe640 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3c3af0 0x55e6be3cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245ffff640 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f24689e3740 10 client.0 _ll_register_callbacks cb 0x55e6be2e2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f245effd640 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3cc0a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f245402a930 tx=0x7f2454003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=263ef2dec47aba58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245009bec0 con 0x55e6be3c4410 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.036+0000 7f243effd640 10 client.25655 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f243effd640 1 -- 172.21.15.28:0/3863103140 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454035070 con 0x55e6be3c4410 2024-09-15T21:59:04.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f243effd640 1 -- 172.21.15.28:0/3863103140 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2454003ba0 con 0x55e6be3c4410 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f243effd640 1 -- 172.21.15.28:0/3863103140 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2454003da0 con 0x55e6be3c4410 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 10 client.25655 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55e6be107620 con 0x55e6be3c4410 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f243effd640 1 -- 172.21.15.28:0/3863103140 <== mon.2 v2:172.21.15.174:3300/0 4 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2454031540 con 0x55e6be3c4410 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 10 client.25655 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 20 client.25655 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be3dea10 0x55e6be3e0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 4 client.25655 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.037+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6be335a20 con 0x55e6be3dea10 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.038+0000 7f245f7fe640 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be3dea10 0x55e6be3e0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.038+0000 7f245f7fe640 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be3dea10 0x55e6be3e0ea0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ac8c6aa9bb46290 server_cookie=c6c768147fe9efd0 in_seq=0 out_seq=0 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.038+0000 7f243effd640 10 client.25655 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f243effd640 1 -- 172.21.15.28:0/3863103140 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e6be335a20 con 0x55e6be3dea10 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f243effd640 10 client.25655 handle_command_reply: tid=0 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 client.25655 shutdown 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 msgr2=0x55e6be3cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3cc0a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f245402a930 tx=0x7f2454003820 comp rx=0 tx=0).stop 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 shutdown_connections 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be3dea10 0x55e6be3e0ea0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3cc0a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f243effd640 0 client.25655 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 --2- 172.21.15.28:0/3863103140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3c3af0 0x55e6be3cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 >> 172.21.15.28:0/3863103140 conn(0x55e6be3c0060 msgr2=0x55e6be3bf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 shutdown_connections 2024-09-15T21:59:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 1 -- 172.21.15.28:0/3863103140 wait complete. 2024-09-15T21:59:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.039+0000 7f24689e3740 20 client.25655 trim_cache size 0 max 16384 2024-09-15T21:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.086+0000 7f24689e3740 1 Processor -- start 2024-09-15T21:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.086+0000 7f24689e3740 1 -- start start 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.087+0000 7f24689e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f24689e3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3c0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f24689e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f24689e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f244c00fd60 con 0x55e6be3c3af0 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f24689e3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f244c011900 con 0x55e6be3c4410 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f245f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40758/0 (socket says 172.21.15.28:40758) 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f245f7fe640 1 -- 172.21.15.28:0/1610447958 learned_addr learned my addr 172.21.15.28:0/1610447958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:04.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.088+0000 7f24689e3740 1 -- 172.21.15.28:0/1610447958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f244c010430 con 0x55e6be3cecb0 2024-09-15T21:59:04.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.089+0000 7f245ffff640 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.090+0000 7f245effd640 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3c0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.090+0000 7f245ffff640 1 -- 172.21.15.28:0/1610447958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 msgr2=0x55e6be3c0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.090+0000 7f245ffff640 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3c0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.090+0000 7f245ffff640 1 -- 172.21.15.28:0/1610447958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 msgr2=0x55e6be3c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.090+0000 7f245ffff640 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.090+0000 7f245ffff640 1 -- 172.21.15.28:0/1610447958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6be3caaf0 con 0x55e6be3cecb0 2024-09-15T21:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.090+0000 7f245effd640 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3c0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T21:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.093+0000 7f245ffff640 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f24580099a0 tx=0x7f2458019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caef47b21ade301a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.094+0000 7f24457fa640 1 -- 172.21.15.28:0/1610447958 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458004b20 con 0x55e6be3cecb0 2024-09-15T21:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.094+0000 7f24457fa640 1 -- 172.21.15.28:0/1610447958 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2458004cc0 con 0x55e6be3cecb0 2024-09-15T21:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.094+0000 7f24457fa640 1 -- 172.21.15.28:0/1610447958 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245801d6d0 con 0x55e6be3cecb0 2024-09-15T21:59:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.096+0000 7f24689e3740 1 -- 172.21.15.28:0/1610447958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 msgr2=0x55e6be3cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.096+0000 7f24689e3740 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f24580099a0 tx=0x7f2458019000 comp rx=0 tx=0).stop 2024-09-15T21:59:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.096+0000 7f24689e3740 1 -- 172.21.15.28:0/1610447958 shutdown_connections 2024-09-15T21:59:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.096+0000 7f24689e3740 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3cc410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.096+0000 7f24689e3740 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c4410 0x55e6be3c0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.096+0000 7f24689e3740 1 --2- 172.21.15.28:0/1610447958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c3af0 0x55e6be3c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.097+0000 7f24689e3740 1 -- 172.21.15.28:0/1610447958 >> 172.21.15.28:0/1610447958 conn(0x55e6be3c0060 msgr2=0x55e6be3cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:04.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.098+0000 7f24689e3740 1 -- 172.21.15.28:0/1610447958 shutdown_connections 2024-09-15T21:59:04.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.098+0000 7f24689e3740 1 -- 172.21.15.28:0/1610447958 wait complete. 2024-09-15T21:59:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.098+0000 7f24689e3740 1 -- start start 2024-09-15T21:59:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 1 Processor -- start 2024-09-15T21:59:04.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c3af0 0x55e6be3d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c4410 0x55e6be3d2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f244c0111d0 con 0x55e6be3c4410 2024-09-15T21:59:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f244c010b00 con 0x55e6be3c3af0 2024-09-15T21:59:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f244c00dd40 con 0x55e6be3cecb0 2024-09-15T21:59:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.099+0000 7f24689e3740 10 client.0 _ll_register_callbacks cb 0x7f2454009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.100+0000 7f245effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c4410 0x55e6be3d2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.100+0000 7f245effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c4410 0x55e6be3d2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40762/0 (socket says 172.21.15.28:40762) 2024-09-15T21:59:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.100+0000 7f245effd640 1 -- 172.21.15.28:0/1403928258 learned_addr learned my addr 172.21.15.28:0/1403928258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.100+0000 7f245ffff640 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.101+0000 7f245f7fe640 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c3af0 0x55e6be3d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.101+0000 7f245ffff640 1 -- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c3af0 msgr2=0x55e6be3d28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.101+0000 7f245ffff640 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c3af0 0x55e6be3d28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.101+0000 7f245ffff640 1 -- 172.21.15.28:0/1403928258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c4410 msgr2=0x55e6be3d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.101+0000 7f245ffff640 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c4410 0x55e6be3d2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.101+0000 7f245ffff640 1 -- 172.21.15.28:0/1403928258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6be448780 con 0x55e6be3cecb0 2024-09-15T21:59:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.102+0000 7f245ffff640 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d3370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f245802a040 tx=0x7f2458004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed657aa88d1d73da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.102+0000 7f243effd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T21:59:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.102+0000 7f243effd640 1 -- 172.21.15.28:0/1403928258 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24580059d0 con 0x55e6be3cecb0 2024-09-15T21:59:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.103+0000 7f243effd640 1 -- 172.21.15.28:0/1403928258 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2458011020 con 0x55e6be3cecb0 2024-09-15T21:59:04.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.103+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2458004cc0 con 0x55e6be3cecb0 2024-09-15T21:59:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.103+0000 7f243effd640 1 -- 172.21.15.28:0/1403928258 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f245803f050 con 0x55e6be3cecb0 2024-09-15T21:59:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.104+0000 7f24689e3740 10 client.25600 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.104+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e6be452150 con 0x55e6be3cecb0 2024-09-15T21:59:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.105+0000 7f243effd640 1 -- 172.21.15.28:0/1403928258 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2458011220 con 0x55e6be3cecb0 2024-09-15T21:59:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f243effd640 1 -- 172.21.15.28:0/1403928258 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2458032980 con 0x55e6be3cecb0 2024-09-15T21:59:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f24689e3740 10 client.25600 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f24689e3740 10 client.25600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f24689e3740 20 client.25600 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f24689e3740 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be452770 0x55e6be454b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f24689e3740 4 client.25600 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:59:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f2458004cc0 con 0x55e6be452770 2024-09-15T21:59:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.106+0000 7f245f7fe640 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be452770 0x55e6be454b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.108+0000 7f245f7fe640 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be452770 0x55e6be454b90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1702da511cc5a13 server_cookie=965ab353dffe79c in_seq=0 out_seq=0 2024-09-15T21:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.109+0000 7f243effd640 10 client.25600 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.248+0000 7f243effd640 1 -- 172.21.15.28:0/1403928258 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f2458004cc0 con 0x55e6be452770 2024-09-15T21:59:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.248+0000 7f243effd640 10 client.25600 handle_command_reply: tid=0 2024-09-15T21:59:04.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.248+0000 7f24689e3740 1 client.25600 shutdown 2024-09-15T21:59:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 msgr2=0x55e6be3d3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d3370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f245802a040 tx=0x7f2458004990 comp rx=0 tx=0).stop 2024-09-15T21:59:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 shutdown_connections 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e6be452770 0x55e6be454b90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f243effd640 0 client.25600 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e6be3cecb0 0x55e6be3d3370 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6be3c4410 0x55e6be3d2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 --2- 172.21.15.28:0/1403928258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e6be3c3af0 0x55e6be3d28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 >> 172.21.15.28:0/1403928258 conn(0x55e6be3c0060 msgr2=0x55e6be3be9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 shutdown_connections 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 1 -- 172.21.15.28:0/1403928258 wait complete. 2024-09-15T21:59:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.249+0000 7f24689e3740 20 client.25600 trim_cache size 0 max 16384 2024-09-15T21:59:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T21:59:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T21:59:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T21:59:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T21:59:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f24480a34e0 msgr2=0x7f24480a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f24480a34e0 0x7f24480a59a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2458009b80 tx=0x7f2458009500 comp rx=0 tx=0).stop 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460101ac0 msgr2=0x7f2460198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460101ac0 0x7f2460198380 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f245000d950 tx=0x7f245000dd50 comp rx=0 tx=0).stop 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 shutdown_connections 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f24480a34e0 0x7f24480a59a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2460107ad0 0x7f246019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460102460 0x7f24601988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 --2- 172.21.15.28:0/581031098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2460101ac0 0x7f2460198380 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.250+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 >> 172.21.15.28:0/581031098 conn(0x7f24600fd820 msgr2=0x7f24600fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.251+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 shutdown_connections 2024-09-15T21:59:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:04.251+0000 7f2465b6d640 1 -- 172.21.15.28:0/581031098 wait complete. 2024-09-15T21:59:04.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T21:59:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:59:05.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 -- 172.21.15.28:0/2200463652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548100e20 msgr2=0x7f3548101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2200463652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548100e20 0x7f3548101200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3538009900 tx=0x7f353802a4a0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 -- 172.21.15.28:0/2200463652 shutdown_connections 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2200463652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3548102ef0 0x7f354810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2200463652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3548102570 0x7f35481029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2200463652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548100e20 0x7f3548101200 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 -- 172.21.15.28:0/2200463652 >> 172.21.15.28:0/2200463652 conn(0x7f35480fc7d0 msgr2=0x7f35480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.522+0000 7f354d8e7640 1 -- 172.21.15.28:0/2200463652 shutdown_connections 2024-09-15T21:59:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.523+0000 7f354d8e7640 1 -- 172.21.15.28:0/2200463652 wait complete. 2024-09-15T21:59:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.523+0000 7f354d8e7640 1 Processor -- start 2024-09-15T21:59:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f354d8e7640 1 -- start start 2024-09-15T21:59:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f354d8e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3548100e20 0x7f35481a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f354d8e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3548102570 0x7f35481a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f354d8e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548102ef0 0x7f354819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f354d8e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35481121d0 con 0x7f3548102ef0 2024-09-15T21:59:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f354d8e7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3548112050 con 0x7f3548102570 2024-09-15T21:59:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f354d8e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3548112350 con 0x7f3548100e20 2024-09-15T21:59:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f35477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548102ef0 0x7f354819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f35477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548102ef0 0x7f354819a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40792/0 (socket says 172.21.15.28:40792) 2024-09-15T21:59:05.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.524+0000 7f35477fe640 1 -- 172.21.15.28:0/2799526691 learned_addr learned my addr 172.21.15.28:0/2799526691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f35477fe640 1 -- 172.21.15.28:0/2799526691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3548100e20 msgr2=0x7f35481a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f35477fe640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3548100e20 0x7f35481a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f35477fe640 1 -- 172.21.15.28:0/2799526691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3548102570 msgr2=0x7f35481a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f35477fe640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3548102570 0x7f35481a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f35477fe640 1 -- 172.21.15.28:0/2799526691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354819af20 con 0x7f3548102ef0 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f35477fe640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548102ef0 0x7f354819a6b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f354000b970 tx=0x7f354000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f354c8e5640 1 -- 172.21.15.28:0/2799526691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540019070 con 0x7f3548102ef0 2024-09-15T21:59:05.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.525+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354819b210 con 0x7f3548102ef0 2024-09-15T21:59:05.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.526+0000 7f354c8e5640 1 -- 172.21.15.28:0/2799526691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354000cb40 con 0x7f3548102ef0 2024-09-15T21:59:05.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.526+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35481a7410 con 0x7f3548102ef0 2024-09-15T21:59:05.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.526+0000 7f354c8e5640 1 -- 172.21.15.28:0/2799526691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354001d630 con 0x7f3548102ef0 2024-09-15T21:59:05.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.530+0000 7f354c8e5640 1 -- 172.21.15.28:0/2799526691 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f354001c070 con 0x7f3548102ef0 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 Processor -- start 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 -- start start 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354c8e5640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f35300a3430 0x7f35300a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb53de0 0x55ec1eb54220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb60f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ec1eab12f0 con 0x55ec1eb55350 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ec1ea58cf0 con 0x55ec1eb53de0 2024-09-15T21:59:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354fcc0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ec1eaad2b0 con 0x55ec1eb5eb80 2024-09-15T21:59:05.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f35467fc640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f35300a3430 0x7f35300a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f3546ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f3546ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40802/0 (socket says 172.21.15.28:40802) 2024-09-15T21:59:05.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f3546ffd640 1 -- 172.21.15.28:0/1744848882 learned_addr learned my addr 172.21.15.28:0/1744848882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:05.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.531+0000 7f354c8e5640 1 -- 172.21.15.28:0/2799526691 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f35400cf400 con 0x7f3548102ef0 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f35467fc640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f35300a3430 0x7f35300a58f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f354819bf00 tx=0x7f353c006c40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f3546ffd640 1 -- 172.21.15.28:0/1744848882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 msgr2=0x55ec1eb60f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f3546ffd640 1 --2- 172.21.15.28:0/1744848882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb60f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f3546ffd640 1 -- 172.21.15.28:0/1744848882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb53de0 msgr2=0x55ec1eb54220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f3546ffd640 1 --2- 172.21.15.28:0/1744848882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb53de0 0x55ec1eb54220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f3546ffd640 1 -- 172.21.15.28:0/1744848882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec1ea3d3a0 con 0x55ec1eb55350 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f3546ffd640 1 --2- 172.21.15.28:0/1744848882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb538a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3538009230 tx=0x7f35380040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85a9b7987c2d780 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:05.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.532+0000 7f352dffb640 1 -- 172.21.15.28:0/1744848882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353803e030 con 0x55ec1eb55350 2024-09-15T21:59:05.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.533+0000 7f352dffb640 1 -- 172.21.15.28:0/1744848882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35380042e0 con 0x55ec1eb55350 2024-09-15T21:59:05.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.533+0000 7f352dffb640 1 -- 172.21.15.28:0/1744848882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538035040 con 0x55ec1eb55350 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 -- 172.21.15.28:0/1744848882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 msgr2=0x55ec1eb538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1744848882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb538a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3538009230 tx=0x7f35380040f0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 -- 172.21.15.28:0/1744848882 shutdown_connections 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1744848882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb60f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1744848882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb53de0 0x55ec1eb54220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1744848882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb538a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 -- 172.21.15.28:0/1744848882 >> 172.21.15.28:0/1744848882 conn(0x55ec1eb14310 msgr2=0x55ec1eb50470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 -- 172.21.15.28:0/1744848882 shutdown_connections 2024-09-15T21:59:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 -- 172.21.15.28:0/1744848882 wait complete. 2024-09-15T21:59:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.534+0000 7f354fcc0740 1 -- start start 2024-09-15T21:59:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.535+0000 7f354fcc0740 1 Processor -- start 2024-09-15T21:59:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.535+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb53de0 0x55ec1eb5e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.535+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb5b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.535+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb5be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f354fcc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ec1eb61870 con 0x55ec1eb55350 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f354fcc0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ec1ea6c060 con 0x55ec1eb5eb80 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f354fcc0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ec1eb619f0 con 0x55ec1eb53de0 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb5b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f354fcc0740 10 client.0 _ll_register_callbacks cb 0x55ec1e7bf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb5b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40806/0 (socket says 172.21.15.28:40806) 2024-09-15T21:59:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 -- 172.21.15.28:0/2333840148 learned_addr learned my addr 172.21.15.28:0/2333840148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 -- 172.21.15.28:0/2333840148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb53de0 msgr2=0x55ec1eb5e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb53de0 0x55ec1eb5e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 -- 172.21.15.28:0/2333840148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 msgr2=0x55ec1eb5be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb5be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.536+0000 7f35467fc640 1 -- 172.21.15.28:0/2333840148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec1e8223c0 con 0x55ec1eb55350 2024-09-15T21:59:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.537+0000 7f35467fc640 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb5b820 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f353c006ac0 tx=0x7f353c0157f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b19f437fdb50bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:05.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.537+0000 7f3526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:59:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.538+0000 7f3526ffd640 1 -- 172.21.15.28:0/2333840148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c015e60 con 0x55ec1eb55350 2024-09-15T21:59:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.538+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35380042e0 con 0x55ec1eb55350 2024-09-15T21:59:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.538+0000 7f3526ffd640 1 -- 172.21.15.28:0/2333840148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f353c004020 con 0x55ec1eb55350 2024-09-15T21:59:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.539+0000 7f3526ffd640 1 -- 172.21.15.28:0/2333840148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f353c029050 con 0x55ec1eb55350 2024-09-15T21:59:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.539+0000 7f354fcc0740 10 client.16083 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.539+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ec1eb6d3d0 con 0x55ec1eb55350 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.540+0000 7f3526ffd640 1 -- 172.21.15.28:0/2333840148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f353c003b70 con 0x55ec1eb55350 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.541+0000 7f3526ffd640 1 -- 172.21.15.28:0/2333840148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f353c012070 con 0x55ec1eb55350 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.541+0000 7f354fcc0740 10 client.16083 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.541+0000 7f354fcc0740 10 client.16083 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.541+0000 7f354fcc0740 20 client.16083 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.541+0000 7f354fcc0740 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1eb6ed60 0x55ec1eb711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.541+0000 7f354fcc0740 4 client.16083 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:59:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.541+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35380042e0 con 0x55ec1eb6ed60 2024-09-15T21:59:05.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.542+0000 7f3546ffd640 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1eb6ed60 0x55ec1eb711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.543+0000 7f3546ffd640 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1eb6ed60 0x55ec1eb711e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb3228db676b82de server_cookie=a67c54e4bf642f83 in_seq=0 out_seq=0 2024-09-15T21:59:05.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.543+0000 7f3526ffd640 10 client.16083 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.544+0000 7f3526ffd640 1 -- 172.21.15.28:0/2333840148 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f35380042e0 con 0x55ec1eb6ed60 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.544+0000 7f3526ffd640 10 client.16083 handle_command_reply: tid=0 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.544+0000 7f354fcc0740 1 client.16083 shutdown 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.544+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 msgr2=0x55ec1eb5b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb5b820 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f353c006ac0 tx=0x7f353c0157f0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 shutdown_connections 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1eb6ed60 0x55ec1eb711e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb5be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb55350 0x55ec1eb5b820 secure :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f353c006ac0 tx=0x7f353c0157f0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 --2- 172.21.15.28:0/2333840148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb53de0 0x55ec1eb5e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 >> 172.21.15.28:0/2333840148 conn(0x55ec1eb14310 msgr2=0x55ec1eb56140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f3526ffd640 0 client.16083 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:59:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 shutdown_connections 2024-09-15T21:59:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 1 -- 172.21.15.28:0/2333840148 wait complete. 2024-09-15T21:59:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.545+0000 7f354fcc0740 20 client.16083 trim_cache size 0 max 16384 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.569+0000 7f354fcc0740 1 Processor -- start 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f354fcc0740 1 -- start start 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb6aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb5b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5a800 0x55ec1eb76ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f3546ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb6aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f3546ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb6aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40816/0 (socket says 172.21.15.28:40816) 2024-09-15T21:59:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f354fcc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f353400ff10 con 0x55ec1eb53de0 2024-09-15T21:59:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f3546ffd640 1 -- 172.21.15.28:0/3012424614 learned_addr learned my addr 172.21.15.28:0/3012424614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f354fcc0740 1 -- 172.21.15.28:0/3012424614 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3534011ab0 con 0x55ec1eb5eb80 2024-09-15T21:59:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.570+0000 7f354fcc0740 1 -- 172.21.15.28:0/3012424614 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35340105e0 con 0x55ec1eb5a800 2024-09-15T21:59:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f3546ffd640 1 -- 172.21.15.28:0/3012424614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5a800 msgr2=0x55ec1eb76ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f3546ffd640 1 --2- 172.21.15.28:0/3012424614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5a800 0x55ec1eb76ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f3546ffd640 1 -- 172.21.15.28:0/3012424614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 msgr2=0x55ec1eb5b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f3546ffd640 1 --2- 172.21.15.28:0/3012424614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb5b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f3546ffd640 1 -- 172.21.15.28:0/3012424614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec1eb5c180 con 0x55ec1eb53de0 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f3546ffd640 1 --2- 172.21.15.28:0/3012424614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb6aba0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3538009900 tx=0x7f3538009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f83e4bb7a55abb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f352d7fa640 1 -- 172.21.15.28:0/3012424614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353803e040 con 0x55ec1eb53de0 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f352d7fa640 1 -- 172.21.15.28:0/3012424614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3538005850 con 0x55ec1eb53de0 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.571+0000 7f352d7fa640 1 -- 172.21.15.28:0/3012424614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538035040 con 0x55ec1eb53de0 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.572+0000 7f354fcc0740 1 -- 172.21.15.28:0/3012424614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 msgr2=0x55ec1eb6aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.572+0000 7f354fcc0740 1 --2- 172.21.15.28:0/3012424614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb6aba0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3538009900 tx=0x7f3538009260 comp rx=0 tx=0).stop 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.572+0000 7f354fcc0740 1 -- 172.21.15.28:0/3012424614 shutdown_connections 2024-09-15T21:59:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.572+0000 7f354fcc0740 1 --2- 172.21.15.28:0/3012424614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5a800 0x55ec1eb76ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.572+0000 7f354fcc0740 1 --2- 172.21.15.28:0/3012424614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb5b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.572+0000 7f354fcc0740 1 --2- 172.21.15.28:0/3012424614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb6aba0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.572+0000 7f354fcc0740 1 -- 172.21.15.28:0/3012424614 >> 172.21.15.28:0/3012424614 conn(0x55ec1eb14310 msgr2=0x55ec1eb5a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.573+0000 7f354fcc0740 1 -- 172.21.15.28:0/3012424614 shutdown_connections 2024-09-15T21:59:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.573+0000 7f354fcc0740 1 -- 172.21.15.28:0/3012424614 wait complete. 2024-09-15T21:59:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.573+0000 7f354fcc0740 1 -- start start 2024-09-15T21:59:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.573+0000 7f354fcc0740 1 Processor -- start 2024-09-15T21:59:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5a800 0x55ec1eb62510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f354fcc0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb62a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f354fcc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3534011380 con 0x55ec1eb53de0 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f354fcc0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3534010cb0 con 0x55ec1eb5a800 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f354fcc0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f353400def0 con 0x55ec1eb5eb80 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f35467fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5a800 0x55ec1eb62510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f35467fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5a800 0x55ec1eb62510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:36914/0 (socket says 172.21.15.28:36914) 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.574+0000 7f35467fc640 1 -- 172.21.15.28:0/1633121270 learned_addr learned my addr 172.21.15.28:0/1633121270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f35477fe640 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb62a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f354fcc0740 10 client.0 _ll_register_callbacks cb 0x7f353c009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f3546ffd640 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f35477fe640 1 -- 172.21.15.28:0/1633121270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5a800 msgr2=0x55ec1eb62510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f35477fe640 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5a800 0x55ec1eb62510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f35477fe640 1 -- 172.21.15.28:0/1633121270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 msgr2=0x55ec1eb671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f35477fe640 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.575+0000 7f35477fe640 1 -- 172.21.15.28:0/1633121270 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec1eb63010 con 0x55ec1eb5eb80 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.576+0000 7f35477fe640 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb62a50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f35400cc900 tx=0x7f354009bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2914dd5d8d77428 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.576+0000 7f3526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.576+0000 7f3526ffd640 1 -- 172.21.15.28:0/1633121270 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354009b490 con 0x55ec1eb5eb80 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.576+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3538005850 con 0x55ec1eb5eb80 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.576+0000 7f3526ffd640 1 -- 172.21.15.28:0/1633121270 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f35400cca70 con 0x55ec1eb5eb80 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.576+0000 7f3526ffd640 1 -- 172.21.15.28:0/1633121270 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f354009f840 con 0x55ec1eb5eb80 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.577+0000 7f354fcc0740 10 client.25603 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.577+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ec1ebd97d0 con 0x55ec1eb5eb80 2024-09-15T21:59:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f3526ffd640 1 -- 172.21.15.28:0/1633121270 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3538005850 con 0x55ec1eb5eb80 2024-09-15T21:59:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f3526ffd640 1 -- 172.21.15.28:0/1633121270 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f35400a3c70 con 0x55ec1eb5eb80 2024-09-15T21:59:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f354fcc0740 10 client.25603 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f354fcc0740 10 client.25603 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T21:59:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f354fcc0740 20 client.25603 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1ebd9ea0 0x55ec1ebdc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f354fcc0740 4 client.25603 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:59:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.578+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec1eabe860 con 0x55ec1ebd9ea0 2024-09-15T21:59:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.579+0000 7f3546ffd640 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1ebd9ea0 0x55ec1ebdc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.584+0000 7f3546ffd640 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1ebd9ea0 0x55ec1ebdc290 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f7c5a3c6c1f3fcd0 server_cookie=870888efe68cbb6e in_seq=0 out_seq=0 2024-09-15T21:59:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.584+0000 7f3526ffd640 10 client.25603 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T21:59:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.584+0000 7f3526ffd640 1 -- 172.21.15.28:0/1633121270 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec1eabe860 con 0x55ec1ebd9ea0 2024-09-15T21:59:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.584+0000 7f3526ffd640 10 client.25603 handle_command_reply: tid=0 2024-09-15T21:59:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.585+0000 7f354fcc0740 1 client.25603 shutdown 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 msgr2=0x55ec1eb62a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb62a50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f35400cc900 tx=0x7f354009bb90 comp rx=0 tx=0).stop 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 shutdown_connections 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55ec1ebd9ea0 0x55ec1ebdc290 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ec1eb5eb80 0x55ec1eb62a50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec1eb5a800 0x55ec1eb62510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 --2- 172.21.15.28:0/1633121270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec1eb53de0 0x55ec1eb671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 >> 172.21.15.28:0/1633121270 conn(0x55ec1eb14310 msgr2=0x55ec1eb464e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.586+0000 7f3526ffd640 0 client.25603 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T21:59:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.587+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 shutdown_connections 2024-09-15T21:59:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.587+0000 7f354fcc0740 1 -- 172.21.15.28:0/1633121270 wait complete. 2024-09-15T21:59:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.587+0000 7f354fcc0740 20 client.25603 trim_cache size 0 max 16384 2024-09-15T21:59:05.590 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.588+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f35300a3430 msgr2=0x7f35300a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f35300a3430 0x7f35300a58f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f354819bf00 tx=0x7f353c006c40 comp rx=0 tx=0).stop 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548102ef0 msgr2=0x7f354819a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548102ef0 0x7f354819a6b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f354000b970 tx=0x7f354000be30 comp rx=0 tx=0).stop 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 shutdown_connections 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f35300a3430 0x7f35300a58f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3548102ef0 0x7f354819a6b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3548102570 0x7f35481a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 --2- 172.21.15.28:0/2799526691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3548100e20 0x7f35481a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.589+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 >> 172.21.15.28:0/2799526691 conn(0x7f35480fc7d0 msgr2=0x7f3548105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.590+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 shutdown_connections 2024-09-15T21:59:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:05.590+0000 7f354d8e7640 1 -- 172.21.15.28:0/2799526691 wait complete. 2024-09-15T21:59:05.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:05 smithi028 ceph-mon[30969]: pgmap v55: 129 pgs: 129 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 124 MiB/s wr, 154 op/s 2024-09-15T21:59:05.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:05 smithi203 ceph-mon[35546]: pgmap v55: 129 pgs: 129 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 124 MiB/s wr, 154 op/s 2024-09-15T21:59:06.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:05 smithi174 ceph-mon[35244]: pgmap v55: 129 pgs: 129 active+clean; 1.5 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 124 MiB/s wr, 154 op/s 2024-09-15T21:59:06.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T21:59:06.944 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.685+0000 7fd665d31640 1 -- 172.21.15.28:0/594040692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd660100a80 msgr2=0x7fd660100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.685+0000 7fd665d31640 1 --2- 172.21.15.28:0/594040692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd660100a80 0x7fd660100ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd654014ce0 tx=0x7fd654032710 comp rx=0 tx=0).stop 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.687+0000 7fd665d31640 1 -- 172.21.15.28:0/594040692 shutdown_connections 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.687+0000 7fd665d31640 1 --2- 172.21.15.28:0/594040692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd660101420 0x7fd66010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.687+0000 7fd665d31640 1 --2- 172.21.15.28:0/594040692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd660100a80 0x7fd660100ee0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.687+0000 7fd665d31640 1 --2- 172.21.15.28:0/594040692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 0x7fd660106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.687+0000 7fd665d31640 1 -- 172.21.15.28:0/594040692 >> 172.21.15.28:0/594040692 conn(0x7fd6600fc7a0 msgr2=0x7fd6600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.687+0000 7fd665d31640 1 -- 172.21.15.28:0/594040692 shutdown_connections 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.687+0000 7fd665d31640 1 -- 172.21.15.28:0/594040692 wait complete. 2024-09-15T21:59:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.688+0000 7fd665d31640 1 Processor -- start 2024-09-15T21:59:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.689+0000 7fd665d31640 1 -- start start 2024-09-15T21:59:07.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:07 smithi028 ceph-mon[30969]: pgmap v56: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 486 MiB/s wr, 333 op/s 2024-09-15T21:59:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.690+0000 7fd665d31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd660100a80 0x7fd660073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd665d31640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd660101420 0x7fd660073e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd665d31640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 0x7fd66006cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd665d31640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd660120d80 con 0x7fd660100a80 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd665d31640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd660120c00 con 0x7fd660101420 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd665d31640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd660120f00 con 0x7fd660106a80 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd65ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 0x7fd66006cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd65ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 0x7fd66006cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:40126/0 (socket says 172.21.15.28:40126) 2024-09-15T21:59:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.691+0000 7fd65ffff640 1 -- 172.21.15.28:0/352105619 learned_addr learned my addr 172.21.15.28:0/352105619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.693+0000 7fd65ffff640 1 -- 172.21.15.28:0/352105619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd660101420 msgr2=0x7fd660073e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:59:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.693+0000 7fd65ffff640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd660101420 0x7fd660073e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.693+0000 7fd65ffff640 1 -- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd660100a80 msgr2=0x7fd660073940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.693+0000 7fd65ffff640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd660100a80 0x7fd660073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.693+0000 7fd65ffff640 1 -- 172.21.15.28:0/352105619 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66006d470 con 0x7fd660106a80 2024-09-15T21:59:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.694+0000 7fd65ffff640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 0x7fd66006cbb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d680 tx=0x7fd658007590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.694+0000 7fd65cff9640 1 -- 172.21.15.28:0/352105619 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658019020 con 0x7fd660106a80 2024-09-15T21:59:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.695+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66006d700 con 0x7fd660106a80 2024-09-15T21:59:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.695+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6600716f0 con 0x7fd660106a80 2024-09-15T21:59:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.695+0000 7fd65cff9640 1 -- 172.21.15.28:0/352105619 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65800ae10 con 0x7fd660106a80 2024-09-15T21:59:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.695+0000 7fd65cff9640 1 -- 172.21.15.28:0/352105619 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658015660 con 0x7fd660106a80 2024-09-15T21:59:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.701+0000 7fd65cff9640 1 -- 172.21.15.28:0/352105619 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd65800c030 con 0x7fd660106a80 2024-09-15T21:59:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.703+0000 7fd66810a740 1 Processor -- start 2024-09-15T21:59:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.703+0000 7fd66810a740 1 -- start start 2024-09-15T21:59:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.704+0000 7fd66810a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465245090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.704+0000 7fd66810a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x560465245a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.704+0000 7fd66810a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x560465252760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.704+0000 7fd66810a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604651ab2f0 con 0x560465246b40 2024-09-15T21:59:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.704+0000 7fd66810a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560465152cf0 con 0x560465250370 2024-09-15T21:59:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.704+0000 7fd66810a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604651a72b0 con 0x5604652455d0 2024-09-15T21:59:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.706+0000 7fd65effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x560465252760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.706+0000 7fd65effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465245090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.706+0000 7fd65effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465245090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40844/0 (socket says 172.21.15.28:40844) 2024-09-15T21:59:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.706+0000 7fd65effd640 1 -- 172.21.15.28:0/2522025675 learned_addr learned my addr 172.21.15.28:0/2522025675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.708+0000 7fd65cff9640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6480a3430 0x7fd6480a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.708+0000 7fd65cff9640 1 -- 172.21.15.28:0/352105619 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd6580cfeb0 con 0x7fd660106a80 2024-09-15T21:59:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.708+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x560465245a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.708+0000 7fd65ffff640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6480a3430 0x7fd6480a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.708+0000 7fd65effd640 1 -- 172.21.15.28:0/2522025675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 msgr2=0x560465245a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.709+0000 7fd65effd640 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x560465245a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.709+0000 7fd65effd640 1 -- 172.21.15.28:0/2522025675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 msgr2=0x560465252760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.709+0000 7fd65effd640 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x560465252760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.709+0000 7fd65effd640 1 -- 172.21.15.28:0/2522025675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604651373a0 con 0x560465246b40 2024-09-15T21:59:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.709+0000 7fd65effd640 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465245090 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd654032c30 tx=0x7fd654000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8a93b415a63818 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.709+0000 7fd65ffff640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6480a3430 0x7fd6480a58f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd65809bdf0 tx=0x7fd658005af0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.710+0000 7fd645ffb640 1 -- 172.21.15.28:0/2522025675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654014040 con 0x560465246b40 2024-09-15T21:59:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.710+0000 7fd645ffb640 1 -- 172.21.15.28:0/2522025675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654039400 con 0x560465246b40 2024-09-15T21:59:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.710+0000 7fd645ffb640 1 -- 172.21.15.28:0/2522025675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654003810 con 0x560465246b40 2024-09-15T21:59:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.711+0000 7fd66810a740 1 -- 172.21.15.28:0/2522025675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 msgr2=0x560465245090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.711+0000 7fd66810a740 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465245090 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd654032c30 tx=0x7fd654000bc0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.712+0000 7fd66810a740 1 -- 172.21.15.28:0/2522025675 shutdown_connections 2024-09-15T21:59:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.712+0000 7fd66810a740 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x560465252760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.712+0000 7fd66810a740 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x560465245a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.712+0000 7fd66810a740 1 --2- 172.21.15.28:0/2522025675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465245090 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.712+0000 7fd66810a740 1 -- 172.21.15.28:0/2522025675 >> 172.21.15.28:0/2522025675 conn(0x56046520e310 msgr2=0x560465241c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.712+0000 7fd66810a740 1 -- 172.21.15.28:0/2522025675 shutdown_connections 2024-09-15T21:59:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.712+0000 7fd66810a740 1 -- 172.21.15.28:0/2522025675 wait complete. 2024-09-15T21:59:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.713+0000 7fd66810a740 1 -- start start 2024-09-15T21:59:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.713+0000 7fd66810a740 1 Processor -- start 2024-09-15T21:59:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd66810a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604652455d0 0x56046524fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd66810a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560465246b40 0x56046524cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd66810a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465250370 0x56046524d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd66810a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560465253000 con 0x560465250370 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd66810a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560465166060 con 0x5604652455d0 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd66810a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560465253180 con 0x560465246b40 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd66810a740 10 client.0 _ll_register_callbacks cb 0x560464eb98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd65effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465250370 0x56046524d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd65effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465250370 0x56046524d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40856/0 (socket says 172.21.15.28:40856) 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd65effd640 1 -- 172.21.15.28:0/1767251638 learned_addr learned my addr 172.21.15.28:0/1767251638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd65effd640 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604652455d0 0x56046524fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.714+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560465246b40 0x56046524cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.715+0000 7fd65effd640 1 -- 172.21.15.28:0/1767251638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560465246b40 msgr2=0x56046524cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.715+0000 7fd65effd640 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560465246b40 0x56046524cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.715+0000 7fd65effd640 1 -- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604652455d0 msgr2=0x56046524fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.715+0000 7fd65effd640 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604652455d0 0x56046524fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.715+0000 7fd65effd640 1 -- 172.21.15.28:0/1767251638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560464f1c3c0 con 0x560465250370 2024-09-15T21:59:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.716+0000 7fd65effd640 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465250370 0x56046524d570 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd6540141b0 tx=0x7fd654005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88c90e5a6ca0ceb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.716+0000 7fd63effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:59:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.717+0000 7fd63effd640 1 -- 172.21.15.28:0/1767251638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6540399c0 con 0x560465250370 2024-09-15T21:59:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.717+0000 7fd63effd640 1 -- 172.21.15.28:0/1767251638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd654003da0 con 0x560465250370 2024-09-15T21:59:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.717+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd654039400 con 0x560465250370 2024-09-15T21:59:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.718+0000 7fd63effd640 1 -- 172.21.15.28:0/1767251638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd654003fa0 con 0x560465250370 2024-09-15T21:59:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.718+0000 7fd66810a740 10 client.16098 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.718+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560465253000 con 0x560465250370 2024-09-15T21:59:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.718+0000 7fd63effd640 1 -- 172.21.15.28:0/1767251638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd654039400 con 0x560465250370 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd63effd640 1 -- 172.21.15.28:0/1767251638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd654009b70 con 0x560465250370 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd66810a740 10 client.16098 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd66810a740 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd66810a740 20 client.16098 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd66810a740 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56046525fc60 0x560465262050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd66810a740 4 client.16098 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604651b8860 con 0x56046525fc60 2024-09-15T21:59:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.719+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56046525fc60 0x560465262050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.720+0000 7fd63effd640 10 client.16098 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.720+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56046525fc60 0x560465262050 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b1b1c57f55b908 server_cookie=408e4ecafb75aed9 in_seq=0 out_seq=0 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.720+0000 7fd63effd640 1 -- 172.21.15.28:0/1767251638 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604651b8860 con 0x56046525fc60 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd63effd640 10 client.16098 handle_command_reply: tid=0 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 client.16098 shutdown 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465250370 msgr2=0x56046524d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465250370 0x56046524d570 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd6540141b0 tx=0x7fd654005f90 comp rx=0 tx=0).stop 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 shutdown_connections 2024-09-15T21:59:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56046525fc60 0x560465262050 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465250370 0x56046524d570 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560465246b40 0x56046524cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 --2- 172.21.15.28:0/1767251638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604652455d0 0x56046524fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 >> 172.21.15.28:0/1767251638 conn(0x56046520e310 msgr2=0x560465247930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd63effd640 0 client.16098 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 shutdown_connections 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 1 -- 172.21.15.28:0/1767251638 wait complete. 2024-09-15T21:59:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.721+0000 7fd66810a740 20 client.16098 trim_cache size 0 max 16384 2024-09-15T21:59:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.737+0000 7fd66810a740 1 Processor -- start 2024-09-15T21:59:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd66810a740 1 -- start start 2024-09-15T21:59:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd66810a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x56046525c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd66810a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465241700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd65f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465241700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd65f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465241700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40866/0 (socket says 172.21.15.28:40866) 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2236503227 learned_addr learned my addr 172.21.15.28:0/2236503227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd66810a740 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x560465241c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c00fec0 con 0x560465246b40 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c011a60 con 0x560465250370 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.738+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c010590 con 0x5604652455d0 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.739+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2236503227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 msgr2=0x56046525c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.739+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x56046525c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.739+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2236503227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 msgr2=0x560465241c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T21:59:07.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.739+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x560465241c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.739+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2236503227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560465242180 con 0x560465246b40 2024-09-15T21:59:07.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.739+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465241700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x560465242ac0 tx=0x7fd650005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93857f9af4e596ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:07.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.740+0000 7fd6457fa640 1 -- 172.21.15.28:0/2236503227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650005250 con 0x560465246b40 2024-09-15T21:59:07.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.740+0000 7fd6457fa640 1 -- 172.21.15.28:0/2236503227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd650004c80 con 0x560465246b40 2024-09-15T21:59:07.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.740+0000 7fd6457fa640 1 -- 172.21.15.28:0/2236503227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650003d00 con 0x560465246b40 2024-09-15T21:59:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 msgr2=0x560465241700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465241700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x560465242ac0 tx=0x7fd650005a20 comp rx=0 tx=0).stop 2024-09-15T21:59:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 shutdown_connections 2024-09-15T21:59:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x560465241c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465241700 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 --2- 172.21.15.28:0/2236503227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x56046525c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 >> 172.21.15.28:0/2236503227 conn(0x56046520e310 msgr2=0x56046524cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 shutdown_connections 2024-09-15T21:59:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.741+0000 7fd66810a740 1 -- 172.21.15.28:0/2236503227 wait complete. 2024-09-15T21:59:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.742+0000 7fd66810a740 1 -- start start 2024-09-15T21:59:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.743+0000 7fd66810a740 1 Processor -- start 2024-09-15T21:59:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.743+0000 7fd66810a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x560465258a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.743+0000 7fd66810a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465249ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd65f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465249ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd65f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465249ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40876/0 (socket says 172.21.15.28:40876) 2024-09-15T21:59:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd66810a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x56046524a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2492692873 learned_addr learned my addr 172.21.15.28:0/2492692873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c011330 con 0x560465246b40 2024-09-15T21:59:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c010c60 con 0x560465250370 2024-09-15T21:59:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c00dea0 con 0x5604652455d0 2024-09-15T21:59:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.744+0000 7fd66810a740 10 client.0 _ll_register_callbacks cb 0x7fd654014b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:07.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2492692873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 msgr2=0x560465258a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:07.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x560465258a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2492692873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 msgr2=0x56046524a020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x56046524a020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd65f7fe640 1 -- 172.21.15.28:0/2492692873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56046524a660 con 0x560465246b40 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd650004c80 con 0x560465246b40 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd65f7fe640 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465249ae0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd6500092a0 tx=0x7fd6500041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fcc71f76fa6893 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.745+0000 7fd63effd640 10 client.16104 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd63effd640 1 -- 172.21.15.28:0/2492692873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650033050 con 0x560465246b40 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd63effd640 1 -- 172.21.15.28:0/2492692873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd650005050 con 0x560465246b40 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd63effd640 1 -- 172.21.15.28:0/2492692873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd6500052d0 con 0x560465246b40 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd66810a740 10 client.16104 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5604652ca490 con 0x560465246b40 2024-09-15T21:59:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd63effd640 1 -- 172.21.15.28:0/2492692873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd650036020 con 0x560465246b40 2024-09-15T21:59:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd66810a740 10 client.16104 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd66810a740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd66810a740 20 client.16104 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.746+0000 7fd66810a740 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604652caab0 0x5604652ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.747+0000 7fd66810a740 4 client.16104 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T21:59:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.747+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604651b2d70 con 0x5604652caab0 2024-09-15T21:59:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.748+0000 7fd65effd640 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604652caab0 0x5604652ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.752+0000 7fd63effd640 10 client.16104 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.752+0000 7fd65effd640 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604652caab0 0x5604652ccf10 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76c9812bf35a9ccf server_cookie=cc21603695e51521 in_seq=0 out_seq=0 2024-09-15T21:59:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.753+0000 7fd63effd640 1 -- 172.21.15.28:0/2492692873 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604651b2d70 con 0x5604652caab0 2024-09-15T21:59:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.753+0000 7fd63effd640 10 client.16104 handle_command_reply: tid=0 2024-09-15T21:59:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.753+0000 7fd66810a740 1 client.16104 shutdown 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.753+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 msgr2=0x560465249ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.753+0000 7fd66810a740 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465249ae0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd6500092a0 tx=0x7fd6500041f0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 shutdown_connections 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604652caab0 0x5604652ccf10 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560465250370 0x56046524a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560465246b40 0x560465249ae0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 --2- 172.21.15.28:0/2492692873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604652455d0 0x560465258a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 >> 172.21.15.28:0/2492692873 conn(0x56046520e310 msgr2=0x56046524e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd63effd640 0 client.16104 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 shutdown_connections 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 1 -- 172.21.15.28:0/2492692873 wait complete. 2024-09-15T21:59:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.754+0000 7fd66810a740 20 client.16104 trim_cache size 0 max 16384 2024-09-15T21:59:07.757 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6480a3430 msgr2=0x7fd6480a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6480a3430 0x7fd6480a58f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd65809bdf0 tx=0x7fd658005af0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 msgr2=0x7fd66006cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 0x7fd66006cbb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d680 tx=0x7fd658007590 comp rx=0 tx=0).stop 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 shutdown_connections 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6480a3430 0x7fd6480a58f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd660106a80 0x7fd66006cbb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd660101420 0x7fd660073e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 --2- 172.21.15.28:0/352105619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd660100a80 0x7fd660073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.756+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 >> 172.21.15.28:0/352105619 conn(0x7fd6600fc7a0 msgr2=0x7fd6600fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:07.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.758+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 shutdown_connections 2024-09-15T21:59:07.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:07.759+0000 7fd665d31640 1 -- 172.21.15.28:0/352105619 wait complete. 2024-09-15T21:59:07.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:07 smithi203 ceph-mon[35546]: pgmap v56: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 486 MiB/s wr, 333 op/s 2024-09-15T21:59:08.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:07 smithi174 ceph-mon[35244]: pgmap v56: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 486 MiB/s wr, 333 op/s 2024-09-15T21:59:09.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:09 smithi028 ceph-mon[30969]: pgmap v57: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 485 MiB/s wr, 289 op/s 2024-09-15T21:59:09.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:09 smithi203 ceph-mon[35546]: pgmap v57: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 485 MiB/s wr, 289 op/s 2024-09-15T21:59:10.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:09 smithi174 ceph-mon[35244]: pgmap v57: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 485 MiB/s wr, 289 op/s 2024-09-15T21:59:10.969 INFO:tasks.workunit.client.0.smithi028.stdout:fs setup took 13 secs 2024-09-15T21:59:12.002 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T21:59:12.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T21:59:12.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:11 smithi028 ceph-mon[30969]: pgmap v58: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 485 MiB/s wr, 289 op/s 2024-09-15T21:59:12.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:11 smithi203 ceph-mon[35546]: pgmap v58: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 485 MiB/s wr, 289 op/s 2024-09-15T21:59:12.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:11 smithi174 ceph-mon[35244]: pgmap v58: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 485 MiB/s wr, 289 op/s 2024-09-15T21:59:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:59:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.361+0000 7f122b059640 1 -- 172.21.15.28:0/1460203659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224106f20 msgr2=0x7f122410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 --2- 172.21.15.28:0/1460203659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224106f20 0x7f122410eaa0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1214009900 tx=0x7f121402a410 comp rx=0 tx=0).stop 2024-09-15T21:59:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 -- 172.21.15.28:0/1460203659 shutdown_connections 2024-09-15T21:59:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 --2- 172.21.15.28:0/1460203659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224106f20 0x7f122410eaa0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 --2- 172.21.15.28:0/1460203659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12241027a0 0x7f12241067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 --2- 172.21.15.28:0/1460203659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224101df0 0x7f12241021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 -- 172.21.15.28:0/1460203659 >> 172.21.15.28:0/1460203659 conn(0x7f12240fd8b0 msgr2=0x7f12240ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 -- 172.21.15.28:0/1460203659 shutdown_connections 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.362+0000 7f122b059640 1 -- 172.21.15.28:0/1460203659 wait complete. 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 Processor -- start 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 -- start start 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224101df0 0x7f1224198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12241027a0 0x7f1224198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224106f20 0x7f122419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1224112020 con 0x7f1224101df0 2024-09-15T21:59:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1224111ea0 con 0x7f12241027a0 2024-09-15T21:59:13.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f122b059640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f12241121a0 con 0x7f1224106f20 2024-09-15T21:59:13.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f1228df5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224101df0 0x7f1224198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f1228df5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224101df0 0x7f1224198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44826/0 (socket says 172.21.15.28:44826) 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.363+0000 7f1228df5640 1 -- 172.21.15.28:0/3640907131 learned_addr learned my addr 172.21.15.28:0/3640907131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f12295f6640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224106f20 0x7f122419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f1223fff640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12241027a0 0x7f1224198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f1228df5640 1 -- 172.21.15.28:0/3640907131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224106f20 msgr2=0x7f122419cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f1228df5640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224106f20 0x7f122419cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f1228df5640 1 -- 172.21.15.28:0/3640907131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12241027a0 msgr2=0x7f1224198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f1228df5640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12241027a0 0x7f1224198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f1228df5640 1 -- 172.21.15.28:0/3640907131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122419d5d0 con 0x7f1224101df0 2024-09-15T21:59:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.364+0000 7f1228df5640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224101df0 0x7f1224198620 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f121c00ba50 tx=0x7f121c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.365+0000 7f1221ffb640 1 -- 172.21.15.28:0/3640907131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c019070 con 0x7f1224101df0 2024-09-15T21:59:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.365+0000 7f1221ffb640 1 -- 172.21.15.28:0/3640907131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f121c00cbe0 con 0x7f1224101df0 2024-09-15T21:59:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.365+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122419d8c0 con 0x7f1224101df0 2024-09-15T21:59:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.365+0000 7f1221ffb640 1 -- 172.21.15.28:0/3640907131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c0076e0 con 0x7f1224101df0 2024-09-15T21:59:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.365+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12241a51a0 con 0x7f1224101df0 2024-09-15T21:59:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.368+0000 7f122d432740 1 Processor -- start 2024-09-15T21:59:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.368+0000 7f122d432740 1 -- start start 2024-09-15T21:59:13.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f122d432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 0x5635f60b9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f122d432740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 0x5635f60b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f122d432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f122d432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635f600f280 con 0x5635f60b3d60 2024-09-15T21:59:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f122d432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5635f5fb6b10 con 0x5635f60b8da0 2024-09-15T21:59:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f122d432740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5635f600b240 con 0x5635f60b96c0 2024-09-15T21:59:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44838/0 (socket says 172.21.15.28:44838) 2024-09-15T21:59:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 -- 172.21.15.28:0/2515284212 learned_addr learned my addr 172.21.15.28:0/2515284212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 -- 172.21.15.28:0/2515284212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 msgr2=0x5635f60b36f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:59:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 --2- 172.21.15.28:0/2515284212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 0x5635f60b36f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 -- 172.21.15.28:0/2515284212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 msgr2=0x5635f60b9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 --2- 172.21.15.28:0/2515284212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 0x5635f60b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.369+0000 7f1223fff640 1 -- 172.21.15.28:0/2515284212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635f5fc31f0 con 0x5635f60b3d60 2024-09-15T21:59:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.370+0000 7f1223fff640 1 --2- 172.21.15.28:0/2515284212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b41e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f121000b7f0 tx=0x7f121000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf56e3060576b26 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.370+0000 7f121affd640 1 -- 172.21.15.28:0/2515284212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121000d950 con 0x5635f60b3d60 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.370+0000 7f1221ffb640 1 -- 172.21.15.28:0/3640907131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f121c01c070 con 0x7f1224101df0 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.371+0000 7f121affd640 1 -- 172.21.15.28:0/2515284212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1210018070 con 0x5635f60b3d60 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.371+0000 7f121affd640 1 -- 172.21.15.28:0/2515284212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12100123e0 con 0x5635f60b3d60 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f1221ffb640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f12080a3580 0x7f12080a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 -- 172.21.15.28:0/2515284212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 msgr2=0x5635f60b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 --2- 172.21.15.28:0/2515284212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b41e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f121000b7f0 tx=0x7f121000bcb0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f1221ffb640 1 -- 172.21.15.28:0/3640907131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f121c0cf530 con 0x7f1224101df0 2024-09-15T21:59:13.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f12295f6640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f12080a3580 0x7f12080a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 -- 172.21.15.28:0/2515284212 shutdown_connections 2024-09-15T21:59:13.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 --2- 172.21.15.28:0/2515284212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b41e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 --2- 172.21.15.28:0/2515284212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 0x5635f60b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 --2- 172.21.15.28:0/2515284212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 0x5635f60b9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 -- 172.21.15.28:0/2515284212 >> 172.21.15.28:0/2515284212 conn(0x5635f60a5ba0 msgr2=0x5635f60a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:13.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.372+0000 7f122d432740 1 -- 172.21.15.28:0/2515284212 shutdown_connections 2024-09-15T21:59:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.374+0000 7f122d432740 1 -- 172.21.15.28:0/2515284212 wait complete. 2024-09-15T21:59:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.374+0000 7f122d432740 1 -- start start 2024-09-15T21:59:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 1 Processor -- start 2024-09-15T21:59:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f12295f6640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f12080a3580 0x7f12080a5a40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1214031550 tx=0x7f1214005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b3d60 0x5635f60bcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 0x5635f60bd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b96c0 0x5635f60bd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635f60c0130 con 0x5635f60b96c0 2024-09-15T21:59:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5635f5fc9f10 con 0x5635f60b3d60 2024-09-15T21:59:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5635f60c02b0 con 0x5635f60b8da0 2024-09-15T21:59:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.375+0000 7f122d432740 10 client.0 _ll_register_callbacks cb 0x5635f5f92a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.376+0000 7f12295f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b96c0 0x5635f60bd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.376+0000 7f12295f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b96c0 0x5635f60bd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44854/0 (socket says 172.21.15.28:44854) 2024-09-15T21:59:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.376+0000 7f12295f6640 1 -- 172.21.15.28:0/2905709446 learned_addr learned my addr 172.21.15.28:0/2905709446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.377+0000 7f12295f6640 1 -- 172.21.15.28:0/2905709446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 msgr2=0x5635f60bd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.377+0000 7f12295f6640 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 0x5635f60bd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.377+0000 7f12295f6640 1 -- 172.21.15.28:0/2905709446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b3d60 msgr2=0x5635f60bcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.377+0000 7f12295f6640 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b3d60 0x5635f60bcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.377+0000 7f12295f6640 1 -- 172.21.15.28:0/2905709446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635f5d806a0 con 0x5635f60b96c0 2024-09-15T21:59:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.377+0000 7f12295f6640 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b96c0 0x5635f60bd690 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1214009900 tx=0x7f1214004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5be74937edf72a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.377+0000 7f1207fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.378+0000 7f1207fff640 1 -- 172.21.15.28:0/2905709446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121404c070 con 0x5635f60b96c0 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.378+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1210018070 con 0x5635f60b96c0 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.378+0000 7f1207fff640 1 -- 172.21.15.28:0/2905709446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f1214003800 con 0x5635f60b96c0 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f1207fff640 1 -- 172.21.15.28:0/2905709446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1214058050 con 0x5635f60b96c0 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f122d432740 10 client.16116 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5635f60cbef0 con 0x5635f60b96c0 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f1207fff640 1 -- 172.21.15.28:0/2905709446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1210018070 con 0x5635f60b96c0 2024-09-15T21:59:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f1207fff640 1 -- 172.21.15.28:0/2905709446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f1214048c40 con 0x5635f60b96c0 2024-09-15T21:59:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f122d432740 10 client.16116 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f122d432740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f122d432740 20 client.16116 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.379+0000 7f122d432740 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f60cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.380+0000 7f122d432740 4 client.16116 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:59:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.380+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635f601c7f0 con 0x5635f60cd3c0 2024-09-15T21:59:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.381+0000 7f1228df5640 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f60cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.385+0000 7f1228df5640 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f60cf860 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d70b690a8010299b server_cookie=eae1eb3e00b27cfd in_seq=0 out_seq=0 2024-09-15T21:59:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.385+0000 7f1207fff640 10 client.16116 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f1207fff640 1 -- 172.21.15.28:0/2905709446 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5635f601c7f0 con 0x5635f60cd3c0 2024-09-15T21:59:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f1207fff640 10 client.16116 handle_command_reply: tid=0 2024-09-15T21:59:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 client.16116 shutdown 2024-09-15T21:59:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b96c0 msgr2=0x5635f60bd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b96c0 0x5635f60bd690 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1214009900 tx=0x7f1214004270 comp rx=0 tx=0).stop 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 shutdown_connections 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f60cf860 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f1207fff640 0 client.16116 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b96c0 0x5635f60bd690 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 0x5635f60bd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 --2- 172.21.15.28:0/2905709446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b3d60 0x5635f60bcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.387+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 >> 172.21.15.28:0/2905709446 conn(0x5635f60a5ba0 msgr2=0x5635f60a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.388+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 shutdown_connections 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.388+0000 7f122d432740 1 -- 172.21.15.28:0/2905709446 wait complete. 2024-09-15T21:59:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.388+0000 7f122d432740 20 client.16116 trim_cache size 0 max 16384 2024-09-15T21:59:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 Processor -- start 2024-09-15T21:59:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 -- start start 2024-09-15T21:59:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 0x5635f60a5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b96c0 0x5635f60d5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12241ace20 con 0x5635f60b3d60 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12241ae9c0 con 0x5635f60b96c0 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f122d432740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f12241ad4f0 con 0x5635f60b8da0 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44868/0 (socket says 172.21.15.28:44868) 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 -- 172.21.15.28:0/3899619144 learned_addr learned my addr 172.21.15.28:0/3899619144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 -- 172.21.15.28:0/3899619144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 msgr2=0x5635f60a5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 --2- 172.21.15.28:0/3899619144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 0x5635f60a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 -- 172.21.15.28:0/3899619144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b96c0 msgr2=0x5635f60d5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 --2- 172.21.15.28:0/3899619144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b96c0 0x5635f60d5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.420+0000 7f1228df5640 1 -- 172.21.15.28:0/3899619144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635f60bbc00 con 0x5635f60b3d60 2024-09-15T21:59:13.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.421+0000 7f1228df5640 1 --2- 172.21.15.28:0/3899619144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b36f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f121c00ba20 tx=0x7f121c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd44a2fb554121e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.421+0000 7f121a7fc640 1 -- 172.21.15.28:0/3899619144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c09f910 con 0x5635f60b3d60 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.421+0000 7f121a7fc640 1 -- 172.21.15.28:0/3899619144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f121c0a41d0 con 0x5635f60b3d60 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.421+0000 7f121a7fc640 1 -- 172.21.15.28:0/3899619144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c09b7c0 con 0x5635f60b3d60 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 -- 172.21.15.28:0/3899619144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 msgr2=0x5635f60b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 --2- 172.21.15.28:0/3899619144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b36f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f121c00ba20 tx=0x7f121c004b50 comp rx=0 tx=0).stop 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 -- 172.21.15.28:0/3899619144 shutdown_connections 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 --2- 172.21.15.28:0/3899619144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b96c0 0x5635f60d5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 --2- 172.21.15.28:0/3899619144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b8da0 0x5635f60a5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 --2- 172.21.15.28:0/3899619144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60b36f0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 -- 172.21.15.28:0/3899619144 >> 172.21.15.28:0/3899619144 conn(0x5635f60a5ba0 msgr2=0x5635f60a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:13.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 -- 172.21.15.28:0/3899619144 shutdown_connections 2024-09-15T21:59:13.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.422+0000 7f122d432740 1 -- 172.21.15.28:0/3899619144 wait complete. 2024-09-15T21:59:13.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.423+0000 7f122d432740 1 -- start start 2024-09-15T21:59:13.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.423+0000 7f122d432740 1 Processor -- start 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.424+0000 7f122d432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.424+0000 7f122d432740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 0x5635f60c2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.424+0000 7f122d432740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 0x5635f60c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.424+0000 7f122d432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12241ae290 con 0x5635f60b3d60 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.424+0000 7f122d432740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12241adbc0 con 0x5635f60b8da0 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.424+0000 7f122d432740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f12241aae00 con 0x5635f60b96c0 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.424+0000 7f122d432740 10 client.0 _ll_register_callbacks cb 0x7f121403f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44876/0 (socket says 172.21.15.28:44876) 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 -- 172.21.15.28:0/1769450167 learned_addr learned my addr 172.21.15.28:0/1769450167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 -- 172.21.15.28:0/1769450167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 msgr2=0x5635f60c2ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T21:59:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 0x5635f60c2ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 -- 172.21.15.28:0/1769450167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 msgr2=0x5635f60c2560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 0x5635f60c2560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 -- 172.21.15.28:0/1769450167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635f6138150 con 0x5635f60b3d60 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1228df5640 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60c5ea0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f121c004a80 tx=0x7f121c09fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259e1ebb496f3cf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.425+0000 7f1207fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f1207fff640 1 -- 172.21.15.28:0/1769450167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c0a9060 con 0x5635f60b3d60 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f1207fff640 1 -- 172.21.15.28:0/1769450167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f121c0ccb40 con 0x5635f60b3d60 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f121c0a41d0 con 0x5635f60b3d60 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f1207fff640 1 -- 172.21.15.28:0/1769450167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f121c0b8070 con 0x5635f60b3d60 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f122d432740 10 client.16122 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5635f61392f0 con 0x5635f60b3d60 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f1207fff640 1 -- 172.21.15.28:0/1769450167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f121c0a41d0 con 0x5635f60b3d60 2024-09-15T21:59:13.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.426+0000 7f1207fff640 1 -- 172.21.15.28:0/1769450167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f121c0b4390 con 0x5635f60b3d60 2024-09-15T21:59:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.427+0000 7f122d432740 10 client.16122 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.427+0000 7f122d432740 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.427+0000 7f122d432740 20 client.16122 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.427+0000 7f122d432740 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f613ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.427+0000 7f122d432740 4 client.16122 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T21:59:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.427+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5635f6016d00 con 0x5635f60cd3c0 2024-09-15T21:59:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.429+0000 7f1223fff640 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f613ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.434+0000 7f1223fff640 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f613ba00 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=960ee2f07e71107c server_cookie=524cc2d511e4ab96 in_seq=0 out_seq=0 2024-09-15T21:59:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.434+0000 7f1207fff640 10 client.16122 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:13.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f1207fff640 1 -- 172.21.15.28:0/1769450167 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5635f6016d00 con 0x5635f60cd3c0 2024-09-15T21:59:13.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f1207fff640 10 client.16122 handle_command_reply: tid=0 2024-09-15T21:59:13.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 client.16122 shutdown 2024-09-15T21:59:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 msgr2=0x5635f60c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60c5ea0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f121c004a80 tx=0x7f121c09fa00 comp rx=0 tx=0).stop 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 shutdown_connections 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635f60cd3c0 0x5635f613ba00 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635f60b96c0 0x5635f60c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635f60b8da0 0x5635f60c2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f1207fff640 0 client.16122 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 --2- 172.21.15.28:0/1769450167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635f60b3d60 0x5635f60c5ea0 secure :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f121c004a80 tx=0x7f121c09fa00 comp rx=0 tx=0).stop 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.436+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 >> 172.21.15.28:0/1769450167 conn(0x5635f60a5ba0 msgr2=0x5635f60a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.437+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 shutdown_connections 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.437+0000 7f122d432740 1 -- 172.21.15.28:0/1769450167 wait complete. 2024-09-15T21:59:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.437+0000 7f122d432740 20 client.16122 trim_cache size 0 max 16384 2024-09-15T21:59:13.440 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:59:13.440 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T21:59:13.440 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "05471dec-68f4-4426-9081-c55daaa5a549", 2024-09-15T21:59:13.440 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T21:59:13.440 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:59:13.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f12080a3580 msgr2=0x7f12080a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f12080a3580 0x7f12080a5a40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1214031550 tx=0x7f1214005a20 comp rx=0 tx=0).stop 2024-09-15T21:59:13.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224101df0 msgr2=0x7f1224198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:13.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224101df0 0x7f1224198620 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f121c00ba50 tx=0x7f121c00bf10 comp rx=0 tx=0).stop 2024-09-15T21:59:13.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 shutdown_connections 2024-09-15T21:59:13.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f12080a3580 0x7f12080a5a40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1224106f20 0x7f122419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12241027a0 0x7f1224198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 --2- 172.21.15.28:0/3640907131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1224101df0 0x7f1224198620 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 >> 172.21.15.28:0/3640907131 conn(0x7f12240fd8b0 msgr2=0x7f12240fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 shutdown_connections 2024-09-15T21:59:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:13.438+0000 7f122b059640 1 -- 172.21.15.28:0/3640907131 wait complete. 2024-09-15T21:59:14.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:13 smithi174 ceph-mon[35244]: pgmap v59: 129 pgs: 129 active+clean; 9.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 841 MiB/s wr, 421 op/s 2024-09-15T21:59:14.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:13 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T21:59:14.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:13 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:13 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T21:59:14.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:13 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:13 smithi028 ceph-mon[30969]: pgmap v59: 129 pgs: 129 active+clean; 9.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 841 MiB/s wr, 421 op/s 2024-09-15T21:59:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:13 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T21:59:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:13 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:13 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T21:59:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:13 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:13 smithi203 ceph-mon[35546]: pgmap v59: 129 pgs: 129 active+clean; 9.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 841 MiB/s wr, 421 op/s 2024-09-15T21:59:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:13 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T21:59:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:13 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:13 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T21:59:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:13 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.250 INFO:tasks.workunit.client.0.smithi028.stdout:Syncing()...3 sec 2024-09-15T21:59:14.251 INFO:tasks.workunit.client.0.smithi028.stdout:Starting Actual Benchmark At: Sun Sep 15 21:59:14 2024 2024-09-15T21:59:14.251 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T21:59:14.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T21:59:14.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:14 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T21:59:14.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:14 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:14 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T21:59:14.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:14 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T21:59:14.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:14 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:14.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:14 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:14 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T21:59:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:14 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:14 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T21:59:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:14 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T21:59:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:14 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:15.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:14 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:14 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T21:59:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:14 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:14 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T21:59:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:14 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T21:59:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:14 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:15.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:14 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T21:59:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:59:16.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:15 smithi028 ceph-mon[30969]: pgmap v60: 129 pgs: 129 active+clean; 9.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 717 MiB/s wr, 311 op/s 2024-09-15T21:59:16.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:15 smithi203 ceph-mon[35546]: pgmap v60: 129 pgs: 129 active+clean; 9.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 717 MiB/s wr, 311 op/s 2024-09-15T21:59:16.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:15 smithi174 ceph-mon[35244]: pgmap v60: 129 pgs: 129 active+clean; 9.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 717 MiB/s wr, 311 op/s 2024-09-15T21:59:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.416+0000 7fa762bb0640 1 -- 172.21.15.28:0/2740305719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c101370 msgr2=0x7fa75c10d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.416+0000 7fa762bb0640 1 --2- 172.21.15.28:0/2740305719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c101370 0x7fa75c10d8a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa74c00d940 tx=0x7fa74c02c3e0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.416+0000 7fa762bb0640 1 -- 172.21.15.28:0/2740305719 shutdown_connections 2024-09-15T21:59:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.416+0000 7fa762bb0640 1 --2- 172.21.15.28:0/2740305719 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c101370 0x7fa75c10d8a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.416+0000 7fa762bb0640 1 --2- 172.21.15.28:0/2740305719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 0x7fa75c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.416+0000 7fa762bb0640 1 --2- 172.21.15.28:0/2740305719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa75c1069e0 0x7fa75c106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.416+0000 7fa762bb0640 1 -- 172.21.15.28:0/2740305719 >> 172.21.15.28:0/2740305719 conn(0x7fa75c0fc6b0 msgr2=0x7fa75c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.421+0000 7fa762bb0640 1 -- 172.21.15.28:0/2740305719 shutdown_connections 2024-09-15T21:59:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.422+0000 7fa762bb0640 1 -- 172.21.15.28:0/2740305719 wait complete. 2024-09-15T21:59:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.423+0000 7fa762bb0640 1 Processor -- start 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.425+0000 7fa762bb0640 1 -- start start 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa762bb0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 0x7fa75c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa762bb0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa75c101370 0x7fa75c1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa762bb0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c1069e0 0x7fa75c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa762bb0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c10fdc0 con 0x7fa75c1009d0 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa762bb0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c10fc40 con 0x7fa75c101370 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa762bb0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c10ff40 con 0x7fa75c1069e0 2024-09-15T21:59:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa76094c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 0x7fa75c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.426+0000 7fa76094c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 0x7fa75c1a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44886/0 (socket says 172.21.15.28:44886) 2024-09-15T21:59:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.427+0000 7fa76094c640 1 -- 172.21.15.28:0/299386442 learned_addr learned my addr 172.21.15.28:0/299386442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.427+0000 7fa75bfff640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa75c101370 0x7fa75c1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.428+0000 7fa76114d640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c1069e0 0x7fa75c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.429+0000 7fa76094c640 1 -- 172.21.15.28:0/299386442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c1069e0 msgr2=0x7fa75c19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.429+0000 7fa76094c640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c1069e0 0x7fa75c19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.429+0000 7fa76094c640 1 -- 172.21.15.28:0/299386442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa75c101370 msgr2=0x7fa75c1a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.429+0000 7fa76094c640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa75c101370 0x7fa75c1a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.429+0000 7fa76094c640 1 -- 172.21.15.28:0/299386442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c19efb0 con 0x7fa75c1009d0 2024-09-15T21:59:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.432+0000 7fa76094c640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 0x7fa75c1a46b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa75400ba50 tx=0x7fa75400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.435+0000 7fa759ffb640 1 -- 172.21.15.28:0/299386442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754019070 con 0x7fa75c1009d0 2024-09-15T21:59:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.435+0000 7fa759ffb640 1 -- 172.21.15.28:0/299386442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75400cc10 con 0x7fa75c1009d0 2024-09-15T21:59:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.435+0000 7fa759ffb640 1 -- 172.21.15.28:0/299386442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754007790 con 0x7fa75c1009d0 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.436+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75c19f2a0 con 0x7fa75c1009d0 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.436+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75c06bc10 con 0x7fa75c1009d0 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 Processor -- start 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 -- start start 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efa4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 0x558e8efc0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efc30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e8ef092f0 con 0x558e8efa52d0 2024-09-15T21:59:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e8eeb1270 con 0x558e8efa49b0 2024-09-15T21:59:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa765a25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558e8ef052b0 con 0x558e8efc0cc0 2024-09-15T21:59:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.440+0000 7fa76114d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efa4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.441+0000 7fa76114d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efa4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56914/0 (socket says 172.21.15.28:56914) 2024-09-15T21:59:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.441+0000 7fa76114d640 1 -- 172.21.15.28:0/3250116467 learned_addr learned my addr 172.21.15.28:0/3250116467 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.441+0000 7fa75bfff640 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efc30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.442+0000 7fa75bfff640 1 -- 172.21.15.28:0/3250116467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 msgr2=0x558e8efa4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.442+0000 7fa75bfff640 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efa4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.442+0000 7fa75bfff640 1 -- 172.21.15.28:0/3250116467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 msgr2=0x558e8efc0780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:59:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.443+0000 7fa75bfff640 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 0x558e8efc0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.443+0000 7fa75bfff640 1 -- 172.21.15.28:0/3250116467 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e8eeb0340 con 0x558e8efc0cc0 2024-09-15T21:59:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.445+0000 7fa759ffb640 1 -- 172.21.15.28:0/299386442 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fa754014a40 con 0x7fa75c1009d0 2024-09-15T21:59:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.446+0000 7fa759ffb640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7440a3690 0x7fa7440a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.446+0000 7fa759ffb640 1 -- 172.21.15.28:0/299386442 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa7540cf660 con 0x7fa75c1009d0 2024-09-15T21:59:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.449+0000 7fa76114d640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7440a3690 0x7fa7440a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.449+0000 7fa75bfff640 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efc30b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa75c1a0050 tx=0x7fa75000ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a194df4de374105 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.452+0000 7fa742ffd640 1 -- 172.21.15.28:0/3250116467 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750015070 con 0x558e8efc0cc0 2024-09-15T21:59:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.452+0000 7fa742ffd640 1 -- 172.21.15.28:0/3250116467 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75000cb40 con 0x558e8efc0cc0 2024-09-15T21:59:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.453+0000 7fa765a25740 1 -- 172.21.15.28:0/3250116467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 msgr2=0x558e8efc30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.453+0000 7fa765a25740 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efc30b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa75c1a0050 tx=0x7fa75000ac60 comp rx=0 tx=0).stop 2024-09-15T21:59:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.456+0000 7fa76114d640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7440a3690 0x7fa7440a5b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0049f0 tx=0x7fa74c02c8e0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.456+0000 7fa765a25740 1 -- 172.21.15.28:0/3250116467 shutdown_connections 2024-09-15T21:59:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.456+0000 7fa765a25740 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efc30b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.456+0000 7fa765a25740 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 0x558e8efc0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.456+0000 7fa765a25740 1 --2- 172.21.15.28:0/3250116467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efa4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.456+0000 7fa765a25740 1 -- 172.21.15.28:0/3250116467 >> 172.21.15.28:0/3250116467 conn(0x558e8efa1060 msgr2=0x558e8efa1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.459+0000 7fa765a25740 1 -- 172.21.15.28:0/3250116467 shutdown_connections 2024-09-15T21:59:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.460+0000 7fa765a25740 1 -- 172.21.15.28:0/3250116467 wait complete. 2024-09-15T21:59:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.461+0000 7fa765a25740 1 -- start start 2024-09-15T21:59:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.461+0000 7fa765a25740 1 Processor -- start 2024-09-15T21:59:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa765a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efbdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa765a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 0x558e8efbe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa765a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efbe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa765a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e8efc38e0 con 0x558e8efa52d0 2024-09-15T21:59:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa765a25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e8ee92290 con 0x558e8efa49b0 2024-09-15T21:59:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa76114d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efbe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa76114d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efbe660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:46830/0 (socket says 172.21.15.28:46830) 2024-09-15T21:59:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa76114d640 1 -- 172.21.15.28:0/4047659323 learned_addr learned my addr 172.21.15.28:0/4047659323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.463+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558e8efc3a60 con 0x558e8efc0cc0 2024-09-15T21:59:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.464+0000 7fa765a25740 10 client.0 _ll_register_callbacks cb 0x558e8eec3140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.464+0000 7fa76114d640 1 -- 172.21.15.28:0/4047659323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 msgr2=0x558e8efbdbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:59:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.464+0000 7fa76114d640 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efbdbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.464+0000 7fa76114d640 1 -- 172.21.15.28:0/4047659323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 msgr2=0x558e8efbe120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:59:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.464+0000 7fa76114d640 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 0x558e8efbe120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.464+0000 7fa76114d640 1 -- 172.21.15.28:0/4047659323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e8ec1c210 con 0x558e8efc0cc0 2024-09-15T21:59:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.466+0000 7fa76114d640 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efbe660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa74c034000 tx=0x7fa74c02c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d7229e32e72143 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.468+0000 7fa73bfff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T21:59:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.468+0000 7fa73bfff640 1 -- 172.21.15.28:0/4047659323 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c04d070 con 0x558e8efc0cc0 2024-09-15T21:59:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.468+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa75000cb40 con 0x558e8efc0cc0 2024-09-15T21:59:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.468+0000 7fa73bfff640 1 -- 172.21.15.28:0/4047659323 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa74c034ba0 con 0x558e8efc0cc0 2024-09-15T21:59:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.469+0000 7fa73bfff640 1 -- 172.21.15.28:0/4047659323 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa74c059050 con 0x558e8efc0cc0 2024-09-15T21:59:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.469+0000 7fa765a25740 10 client.25627 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.469+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e8ece8620 con 0x558e8efc0cc0 2024-09-15T21:59:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.470+0000 7fa73bfff640 1 -- 172.21.15.28:0/4047659323 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75000cb40 con 0x558e8efc0cc0 2024-09-15T21:59:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.471+0000 7fa73bfff640 1 -- 172.21.15.28:0/4047659323 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fa74c048070 con 0x558e8efc0cc0 2024-09-15T21:59:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.471+0000 7fa765a25740 10 client.25627 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.471+0000 7fa765a25740 10 client.25627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.471+0000 7fa765a25740 20 client.25627 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.471+0000 7fa765a25740 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8efd1230 0x558e8efd36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.471+0000 7fa765a25740 4 client.25627 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:59:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.471+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e8ef16a20 con 0x558e8efd1230 2024-09-15T21:59:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.474+0000 7fa76094c640 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8efd1230 0x558e8efd36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.485+0000 7fa73bfff640 10 client.25627 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.485+0000 7fa76094c640 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8efd1230 0x558e8efd36f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8796494918f5ac5 server_cookie=60e9d35eaac22078 in_seq=0 out_seq=0 2024-09-15T21:59:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.488+0000 7fa73bfff640 1 -- 172.21.15.28:0/4047659323 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e8ef16a20 con 0x558e8efd1230 2024-09-15T21:59:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.488+0000 7fa73bfff640 10 client.25627 handle_command_reply: tid=0 2024-09-15T21:59:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.488+0000 7fa765a25740 1 client.25627 shutdown 2024-09-15T21:59:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.492+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 msgr2=0x558e8efbe660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.492+0000 7fa765a25740 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efbe660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa74c034000 tx=0x7fa74c02c3e0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.494+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 shutdown_connections 2024-09-15T21:59:17.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.494+0000 7fa765a25740 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8efd1230 0x558e8efd36f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.494+0000 7fa765a25740 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efc0cc0 0x558e8efbe660 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.494+0000 7fa765a25740 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efa52d0 0x558e8efbe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.494+0000 7fa765a25740 1 --2- 172.21.15.28:0/4047659323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa49b0 0x558e8efbdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.494+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 >> 172.21.15.28:0/4047659323 conn(0x558e8efa1060 msgr2=0x558e8efa6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:17.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.494+0000 7fa73bfff640 0 client.25627 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.500+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 shutdown_connections 2024-09-15T21:59:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.505+0000 7fa765a25740 1 -- 172.21.15.28:0/4047659323 wait complete. 2024-09-15T21:59:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.509+0000 7fa765a25740 20 client.25627 trim_cache size 0 max 16384 2024-09-15T21:59:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.541+0000 7fa765a25740 1 Processor -- start 2024-09-15T21:59:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.542+0000 7fa765a25740 1 -- start start 2024-09-15T21:59:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.542+0000 7fa765a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efbfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.542+0000 7fa765a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efd9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.542+0000 7fa765a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efd9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.543+0000 7fa765a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c196950 con 0x558e8efc0cc0 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.543+0000 7fa76114d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efd9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.543+0000 7fa76114d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efd9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44900/0 (socket says 172.21.15.28:44900) 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.543+0000 7fa76114d640 1 -- 172.21.15.28:0/2096041415 learned_addr learned my addr 172.21.15.28:0/2096041415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.543+0000 7fa765a25740 1 -- 172.21.15.28:0/2096041415 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1984f0 con 0x558e8efa52d0 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.543+0000 7fa76094c640 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efbfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.543+0000 7fa765a25740 1 -- 172.21.15.28:0/2096041415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c197020 con 0x558e8efa49b0 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.544+0000 7fa76114d640 1 -- 172.21.15.28:0/2096041415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 msgr2=0x558e8efbfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:17.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.544+0000 7fa76114d640 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efbfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.544+0000 7fa76114d640 1 -- 172.21.15.28:0/2096041415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 msgr2=0x558e8efd9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.544+0000 7fa76114d640 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efd9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.544+0000 7fa76114d640 1 -- 172.21.15.28:0/2096041415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e8efcf8c0 con 0x558e8efc0cc0 2024-09-15T21:59:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.544+0000 7fa76114d640 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efd9540 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa74c006010 tx=0x7fa74c0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e0d19c20421f5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.545+0000 7fa7427fc640 1 -- 172.21.15.28:0/2096041415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74c04d070 con 0x558e8efc0cc0 2024-09-15T21:59:17.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.545+0000 7fa7427fc640 1 -- 172.21.15.28:0/2096041415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74c043070 con 0x558e8efc0cc0 2024-09-15T21:59:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.546+0000 7fa765a25740 1 -- 172.21.15.28:0/2096041415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 msgr2=0x558e8efd9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:17.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.546+0000 7fa765a25740 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efd9540 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa74c006010 tx=0x7fa74c0058b0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.550+0000 7fa765a25740 1 -- 172.21.15.28:0/2096041415 shutdown_connections 2024-09-15T21:59:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.550+0000 7fa765a25740 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efd9540 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.550+0000 7fa765a25740 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efd9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.550+0000 7fa765a25740 1 --2- 172.21.15.28:0/2096041415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efbfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.550+0000 7fa765a25740 1 -- 172.21.15.28:0/2096041415 >> 172.21.15.28:0/2096041415 conn(0x558e8efa1060 msgr2=0x558e8efbf440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.553+0000 7fa765a25740 1 -- 172.21.15.28:0/2096041415 shutdown_connections 2024-09-15T21:59:17.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.553+0000 7fa765a25740 1 -- 172.21.15.28:0/2096041415 wait complete. 2024-09-15T21:59:17.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.553+0000 7fa765a25740 1 -- start start 2024-09-15T21:59:17.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.553+0000 7fa765a25740 1 Processor -- start 2024-09-15T21:59:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.554+0000 7fa765a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efc9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.554+0000 7fa765a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efc4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.554+0000 7fa765a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efc4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.554+0000 7fa765a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c197dc0 con 0x558e8efc0cc0 2024-09-15T21:59:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.554+0000 7fa765a25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c1976f0 con 0x558e8efa52d0 2024-09-15T21:59:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.554+0000 7fa765a25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c100e30 con 0x558e8efa49b0 2024-09-15T21:59:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.554+0000 7fa765a25740 10 client.0 _ll_register_callbacks cb 0x7fa74c00d790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efc4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efc4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56940/0 (socket says 172.21.15.28:56940) 2024-09-15T21:59:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 -- 172.21.15.28:0/3095970168 learned_addr learned my addr 172.21.15.28:0/3095970168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa76094c640 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efc9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 -- 172.21.15.28:0/3095970168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 msgr2=0x558e8efc9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:17.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efc9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 -- 172.21.15.28:0/3095970168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 msgr2=0x558e8efc4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efc4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.555+0000 7fa75bfff640 1 -- 172.21.15.28:0/3095970168 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e8efc5040 con 0x558e8efa52d0 2024-09-15T21:59:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.556+0000 7fa75bfff640 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efc4540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa750017000 tx=0x7fa750007fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d03a75669afcc935 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.556+0000 7fa73bfff640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:59:17.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.557+0000 7fa73bfff640 1 -- 172.21.15.28:0/3095970168 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750015070 con 0x558e8efa52d0 2024-09-15T21:59:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.557+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa74c043070 con 0x558e8efa52d0 2024-09-15T21:59:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.557+0000 7fa73bfff640 1 -- 172.21.15.28:0/3095970168 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fa750004450 con 0x558e8efa52d0 2024-09-15T21:59:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.558+0000 7fa73bfff640 1 -- 172.21.15.28:0/3095970168 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa7500036b0 con 0x558e8efa52d0 2024-09-15T21:59:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.558+0000 7fa765a25740 10 client.25679 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.558+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558e8f03bd70 con 0x558e8efa52d0 2024-09-15T21:59:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.558+0000 7fa73bfff640 1 -- 172.21.15.28:0/3095970168 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74c043070 con 0x558e8efa52d0 2024-09-15T21:59:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.559+0000 7fa73bfff640 1 -- 172.21.15.28:0/3095970168 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fa750021070 con 0x558e8efa52d0 2024-09-15T21:59:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.559+0000 7fa765a25740 10 client.25679 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.559+0000 7fa765a25740 10 client.25679 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.559+0000 7fa765a25740 20 client.25679 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.559+0000 7fa765a25740 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8f03c450 0x558e8f03e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.559+0000 7fa765a25740 4 client.25679 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T21:59:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.559+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e8ef10d70 con 0x558e8f03c450 2024-09-15T21:59:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.560+0000 7fa76094c640 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8f03c450 0x558e8f03e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.568+0000 7fa76094c640 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8f03c450 0x558e8f03e870 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7d6788e43b342fd server_cookie=3cac8ecdd6f5f67 in_seq=0 out_seq=0 2024-09-15T21:59:17.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T21:59:17.569+0000 7fa73bfff640 10 client.25679 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:17 smithi028 ceph-mon[30969]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 822 MiB/s wr, 1.16k op/s 2024-09-15T21:59:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:17 smithi203 ceph-mon[35546]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 822 MiB/s wr, 1.16k op/s 2024-09-15T21:59:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:17 smithi174 ceph-mon[35244]: pgmap v61: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 822 MiB/s wr, 1.16k op/s 2024-09-15T21:59:20.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:19 smithi028 ceph-mon[30969]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 590 MiB/s wr, 1.67k op/s 2024-09-15T21:59:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:19 smithi203 ceph-mon[35546]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 590 MiB/s wr, 1.67k op/s 2024-09-15T21:59:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:19 smithi174 ceph-mon[35244]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 590 MiB/s wr, 1.67k op/s 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.071+0000 7fd2dca3a640 1 -- 172.21.15.28:0/1032134154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 msgr2=0x7fd2d8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.071+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/1032134154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d8101200 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009cd0 tx=0x7fd2c802cbd0 comp rx=0 tx=0).stop 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.072+0000 7fd2dca3a640 1 -- 172.21.15.28:0/1032134154 shutdown_connections 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.072+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/1032134154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d8102ef0 0x7fd2d810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.072+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/1032134154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8102570 0x7fd2d81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.072+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/1032134154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d8101200 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.072+0000 7fd2dca3a640 1 -- 172.21.15.28:0/1032134154 >> 172.21.15.28:0/1032134154 conn(0x7fd2d80fc7d0 msgr2=0x7fd2d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.075+0000 7fd2dca3a640 1 -- 172.21.15.28:0/1032134154 shutdown_connections 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.075+0000 7fd2dca3a640 1 -- 172.21.15.28:0/1032134154 wait complete. 2024-09-15T21:59:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.077+0000 7fd2dca3a640 1 Processor -- start 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.079+0000 7fd2dca3a640 1 -- start start 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.079+0000 7fd2dca3a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d810c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.079+0000 7fd2dca3a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8102570 0x7fd2d8073c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.079+0000 7fd2dca3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d8102ef0 0x7fd2d810c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.079+0000 7fd2dca3a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810f910 con 0x7fd2d8102ef0 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.079+0000 7fd2dca3a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810f790 con 0x7fd2d8100e20 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.079+0000 7fd2dca3a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810fa90 con 0x7fd2d8102570 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d810c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d810c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:50902/0 (socket says 172.21.15.28:50902) 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 -- 172.21.15.28:0/3013335179 learned_addr learned my addr 172.21.15.28:0/3013335179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 -- 172.21.15.28:0/3013335179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8102570 msgr2=0x7fd2d8073c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8102570 0x7fd2d8073c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 -- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d8102ef0 msgr2=0x7fd2d810c990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d8102ef0 0x7fd2d810c990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 -- 172.21.15.28:0/3013335179 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d81b3550 con 0x7fd2d8100e20 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.080+0000 7fd2d659c640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d810c450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009ca0 tx=0x7fd2c802ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.083+0000 7fd2cf7fe640 1 -- 172.21.15.28:0/3013335179 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8044020 con 0x7fd2d8100e20 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.083+0000 7fd2cf7fe640 1 -- 172.21.15.28:0/3013335179 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2c803fd30 con 0x7fd2d8100e20 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.083+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d81b37e0 con 0x7fd2d8100e20 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.084+0000 7fd2cf7fe640 1 -- 172.21.15.28:0/3013335179 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8040810 con 0x7fd2d8100e20 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.085+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d81b3c30 con 0x7fd2d8100e20 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.089+0000 7fd2cf7fe640 1 -- 172.21.15.28:0/3013335179 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd2c8047020 con 0x7fd2d8100e20 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.091+0000 7fd2dee13740 1 Processor -- start 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.091+0000 7fd2cf7fe640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2c40a3420 0x7fd2c40a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.091+0000 7fd2cf7fe640 1 -- 172.21.15.28:0/3013335179 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd2c80f4170 con 0x7fd2d8100e20 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2dee13740 1 -- start start 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c34a0 0x559d774c3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2dee13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559d774292e0 con 0x559d774c34a0 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2dee13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559d773d0ce0 con 0x559d774ce680 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2dee13740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559d774252a0 con 0x559d774c3dc0 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2c40a3420 0x7fd2c40a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:52056/0 (socket says 172.21.15.28:52056) 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 -- 172.21.15.28:0/600737057 learned_addr learned my addr 172.21.15.28:0/600737057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 -- 172.21.15.28:0/600737057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 msgr2=0x559d774d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 -- 172.21.15.28:0/600737057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c34a0 msgr2=0x559d774c3880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c34a0 0x559d774c3880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 -- 172.21.15.28:0/600737057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d773b5440 con 0x559d774c3dc0 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.097+0000 7fd2cffff640 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774ce140 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009b40 tx=0x7fd2c000e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa84d63564071f12 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.104+0000 7fd2ccff9640 1 -- 172.21.15.28:0/600737057 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c000ed80 con 0x559d774c3dc0 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.104+0000 7fd2ccff9640 1 -- 172.21.15.28:0/600737057 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2c0005500 con 0x559d774c3dc0 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.104+0000 7fd2ccff9640 1 -- 172.21.15.28:0/600737057 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0010690 con 0x559d774c3dc0 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.105+0000 7fd2dee13740 1 -- 172.21.15.28:0/600737057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 msgr2=0x559d774ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.105+0000 7fd2dee13740 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774ce140 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009b40 tx=0x7fd2c000e350 comp rx=0 tx=0).stop 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.105+0000 7fd2dee13740 1 -- 172.21.15.28:0/600737057 shutdown_connections 2024-09-15T21:59:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.105+0000 7fd2dee13740 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.105+0000 7fd2dee13740 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774ce140 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.105+0000 7fd2dee13740 1 --2- 172.21.15.28:0/600737057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c34a0 0x559d774c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.105+0000 7fd2dee13740 1 -- 172.21.15.28:0/600737057 >> 172.21.15.28:0/600737057 conn(0x559d774bfa30 msgr2=0x559d774bfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.106+0000 7fd2dee13740 1 -- 172.21.15.28:0/600737057 shutdown_connections 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.106+0000 7fd2dee13740 1 -- 172.21.15.28:0/600737057 wait complete. 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.107+0000 7fd2dee13740 1 -- start start 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.107+0000 7fd2dee13740 1 Processor -- start 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c34a0 0x559d774cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c3dc0 0x559d774cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774cb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2dee13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559d774e2260 con 0x559d774c3dc0 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2dee13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559d773e4050 con 0x559d774ce680 2024-09-15T21:59:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2dee13740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559d774e23e0 con 0x559d774c34a0 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2dee13740 10 client.0 _ll_register_callbacks cb 0x559d77424160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c34a0 0x559d774cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:50918/0 (socket says 172.21.15.28:50918) 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/2142695621 learned_addr learned my addr 172.21.15.28:0/2142695621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/2142695621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c34a0 msgr2=0x559d774cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c34a0 0x559d774cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/2142695621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c3dc0 msgr2=0x559d774cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c3dc0 0x559d774cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.108+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/2142695621 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d7719a3c0 con 0x559d774ce680 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.109+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774cb780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0015de0 tx=0x7fd2d0013650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=263257d04d7d7c9a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.109+0000 7fd2bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:59:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.109+0000 7fd2bdffb640 1 -- 172.21.15.28:0/2142695621 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000f070 con 0x559d774ce680 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.109+0000 7fd2bdffb640 1 -- 172.21.15.28:0/2142695621 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd2d0009cb0 con 0x559d774ce680 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.109+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2c0005500 con 0x559d774ce680 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.109+0000 7fd2bdffb640 1 -- 172.21.15.28:0/2142695621 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd2d0018480 con 0x559d774ce680 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 10 client.25691 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559d774edc70 con 0x559d774ce680 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2bdffb640 1 -- 172.21.15.28:0/2142695621 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d0018b60 con 0x559d774ce680 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2bdffb640 1 -- 172.21.15.28:0/2142695621 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd2d0023c90 con 0x559d774ce680 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 10 client.25691 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 10 client.25691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 20 client.25691 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d774ef1f0 0x559d774f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 4 client.25691 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T21:59:22.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.110+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd2c0005500 con 0x559d774ef1f0 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.113+0000 7fd2d659c640 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d774ef1f0 0x559d774f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.116+0000 7fd2d659c640 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d774ef1f0 0x559d774f16b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ded7e33263adf92b server_cookie=2c29a8a11555d223 in_seq=0 out_seq=0 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.116+0000 7fd2bdffb640 10 client.25691 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.117+0000 7fd2bdffb640 1 -- 172.21.15.28:0/2142695621 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fd2c0005500 con 0x559d774ef1f0 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.117+0000 7fd2bdffb640 10 client.25691 handle_command_reply: tid=0 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.117+0000 7fd2dee13740 1 client.25691 shutdown 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 msgr2=0x559d774cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774cb780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0015de0 tx=0x7fd2d0013650 comp rx=0 tx=0).stop 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 shutdown_connections 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d774ef1f0 0x559d774f16b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774ce680 0x559d774cb780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774c3dc0 0x559d774cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 --2- 172.21.15.28:0/2142695621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c34a0 0x559d774cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 >> 172.21.15.28:0/2142695621 conn(0x559d774bfa30 msgr2=0x559d774c5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2bdffb640 0 client.25691 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 shutdown_connections 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.118+0000 7fd2dee13740 1 -- 172.21.15.28:0/2142695621 wait complete. 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.120+0000 7fd2dee13740 20 client.25691 trim_cache size 0 max 16384 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.122+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2c40a3420 0x7fd2c40a58e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0018fd0 tx=0x7fd2d0012c90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.170+0000 7fd2dee13740 1 Processor -- start 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.173+0000 7fd2dee13740 1 -- start start 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.173+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.173+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.173+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d774f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.173+0000 7fd2dee13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81b52c0 con 0x559d774ce680 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.173+0000 7fd2dee13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81b6e60 con 0x559d774c34a0 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.173+0000 7fd2dee13740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81b5990 con 0x559d774c3dc0 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.176+0000 7fd2d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d774f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.176+0000 7fd2d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d774f6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57578/0 (socket says 172.21.15.28:57578) 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.176+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.176+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/1621098362 learned_addr learned my addr 172.21.15.28:0/1621098362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.177+0000 7fd2d659c640 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.177+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/1621098362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 msgr2=0x559d774c9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.177+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774c9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.178+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/1621098362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 msgr2=0x559d774eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.178+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774eb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.178+0000 7fd2d6d9d640 1 -- 172.21.15.28:0/1621098362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d774c32d0 con 0x559d774ce680 2024-09-15T21:59:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.179+0000 7fd2d6d9d640 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d774f6fc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0023fb0 tx=0x7fd2d002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295fa421e6a38591 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.180+0000 7fd2bffff640 1 -- 172.21.15.28:0/1621098362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0032070 con 0x559d774ce680 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.180+0000 7fd2bffff640 1 -- 172.21.15.28:0/1621098362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d0012dc0 con 0x559d774ce680 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.180+0000 7fd2dee13740 1 -- 172.21.15.28:0/1621098362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 msgr2=0x559d774f6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.180+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d774f6fc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0023fb0 tx=0x7fd2d002f000 comp rx=0 tx=0).stop 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.184+0000 7fd2dee13740 1 -- 172.21.15.28:0/1621098362 shutdown_connections 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.184+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d774f6fc0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.184+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.184+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1621098362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.184+0000 7fd2dee13740 1 -- 172.21.15.28:0/1621098362 >> 172.21.15.28:0/1621098362 conn(0x559d774bfa30 msgr2=0x559d774cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.185+0000 7fd2dee13740 1 -- 172.21.15.28:0/1621098362 shutdown_connections 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.185+0000 7fd2dee13740 1 -- 172.21.15.28:0/1621098362 wait complete. 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.185+0000 7fd2dee13740 1 -- start start 2024-09-15T21:59:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.185+0000 7fd2dee13740 1 Processor -- start 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.189+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.189+0000 7fd2d659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.189+0000 7fd2d659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:50936/0 (socket says 172.21.15.28:50936) 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774e5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2dee13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d77551fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2dee13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81b6730 con 0x559d774ce680 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2dee13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81b6060 con 0x559d774c34a0 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2dee13740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81021b0 con 0x559d774c3dc0 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2dee13740 10 client.0 _ll_register_callbacks cb 0x7fd2d0009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2d659c640 1 -- 172.21.15.28:0/1326371738 learned_addr learned my addr 172.21.15.28:0/1326371738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2cffff640 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774e5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2d659c640 1 -- 172.21.15.28:0/1326371738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 msgr2=0x559d774e5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2d659c640 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774e5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2d659c640 1 -- 172.21.15.28:0/1326371738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 msgr2=0x559d77551fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2d659c640 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d77551fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.190+0000 7fd2d659c640 1 -- 172.21.15.28:0/1326371738 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d77552610 con 0x559d774c34a0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.191+0000 7fd2d659c640 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774e4d30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2c802eef0 tx=0x7fd2c802cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ad2d8b065d6bac0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.191+0000 7fd2bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.192+0000 7fd2bdffb640 1 -- 172.21.15.28:0/1326371738 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80f5060 con 0x559d774c34a0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.192+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2d0012dc0 con 0x559d774c34a0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.192+0000 7fd2bdffb640 1 -- 172.21.15.28:0/1326371738 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd2c80ee4f0 con 0x559d774c34a0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.192+0000 7fd2bdffb640 1 -- 172.21.15.28:0/1326371738 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd2c80bc940 con 0x559d774c34a0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.193+0000 7fd2dee13740 10 client.25697 fetch_fsmap learned FSMap version 30 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.193+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559d77559a20 con 0x559d774c34a0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.193+0000 7fd2bdffb640 1 -- 172.21.15.28:0/1326371738 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d0012dc0 con 0x559d774c34a0 2024-09-15T21:59:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.193+0000 7fd2bdffb640 1 -- 172.21.15.28:0/1326371738 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd2c803d340 con 0x559d774c34a0 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.194+0000 7fd2dee13740 10 client.25697 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.194+0000 7fd2dee13740 10 client.25697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.194+0000 7fd2dee13740 20 client.25697 populate_metadata read hostname 'smithi028' 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.194+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d7755a220 0x559d7755c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.194+0000 7fd2dee13740 4 client.25697 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.194+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559d77436850 con 0x559d7755a220 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.194+0000 7fd2cffff640 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d7755a220 0x559d7755c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.195+0000 7fd2cffff640 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d7755a220 0x559d7755c640 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8672155cbdead7b0 server_cookie=99c966f903eb9844 in_seq=0 out_seq=0 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.195+0000 7fd2bdffb640 10 client.25697 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.199+0000 7fd2bdffb640 1 -- 172.21.15.28:0/1326371738 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559d77436850 con 0x559d7755a220 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.199+0000 7fd2bdffb640 10 client.25697 handle_command_reply: tid=0 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.199+0000 7fd2dee13740 1 client.25697 shutdown 2024-09-15T21:59:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.199+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 msgr2=0x559d774e4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.199+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774e4d30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2c802eef0 tx=0x7fd2c802cbd0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stdout: "05471dec-68f4-4426-9081-c55daaa5a549": { 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "05471dec-68f4-4426-9081-c55daaa5a549", 2024-09-15T21:59:22.012 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.200+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 shutdown_connections 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.200+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559d7755a220 0x559d7755c640 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.200+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d774ce680 0x559d77551fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.200+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559d774c3dc0 0x559d774e5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.200+0000 7fd2dee13740 1 --2- 172.21.15.28:0/1326371738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d774c34a0 0x559d774e4d30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.200+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 >> 172.21.15.28:0/1326371738 conn(0x559d774bfa30 msgr2=0x559d774be350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.200+0000 7fd2bdffb640 0 client.25697 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.204+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 shutdown_connections 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.204+0000 7fd2dee13740 1 -- 172.21.15.28:0/1326371738 wait complete. 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.204+0000 7fd2dee13740 20 client.25697 trim_cache size 0 max 16384 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2c40a3420 msgr2=0x7fd2c40a58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2c40a3420 0x7fd2c40a58e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0018fd0 tx=0x7fd2d0012c90 comp rx=0 tx=0).stop 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 msgr2=0x7fd2d810c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d810c450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009ca0 tx=0x7fd2c802ec90 comp rx=0 tx=0).stop 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 shutdown_connections 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2c40a3420 0x7fd2c40a58e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d8102ef0 0x7fd2d810c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8102570 0x7fd2d8073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 --2- 172.21.15.28:0/3013335179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d8100e20 0x7fd2d810c450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 >> 172.21.15.28:0/3013335179 conn(0x7fd2d80fc7d0 msgr2=0x7fd2d81067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.209+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 shutdown_connections 2024-09-15T21:59:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:21.210+0000 7fd2dca3a640 1 -- 172.21.15.28:0/3013335179 wait complete. 2024-09-15T21:59:22.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:21 smithi028 ceph-mon[30969]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 590 MiB/s wr, 1.67k op/s 2024-09-15T21:59:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:21 smithi203 ceph-mon[35546]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 590 MiB/s wr, 1.67k op/s 2024-09-15T21:59:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:21 smithi174 ceph-mon[35244]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 590 MiB/s wr, 1.67k op/s 2024-09-15T21:59:24.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:23 smithi028 ceph-mon[30969]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 627 MiB/s wr, 7.38k op/s 2024-09-15T21:59:24.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:23 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T21:59:24.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:23 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T21:59:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:23 smithi203 ceph-mon[35546]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 627 MiB/s wr, 7.38k op/s 2024-09-15T21:59:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:23 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T21:59:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:23 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T21:59:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:23 smithi174 ceph-mon[35244]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 627 MiB/s wr, 7.38k op/s 2024-09-15T21:59:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:23 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T21:59:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:23 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T21:59:25.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:24 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T21:59:25.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:24 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T21:59:25.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:24 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T21:59:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:25 smithi028 ceph-mon[30969]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 271 MiB/s wr, 7.25k op/s 2024-09-15T21:59:26.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:25 smithi203 ceph-mon[35546]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 271 MiB/s wr, 7.25k op/s 2024-09-15T21:59:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:25 smithi174 ceph-mon[35244]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 271 MiB/s wr, 7.25k op/s 2024-09-15T21:59:28.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:27 smithi028 ceph-mon[30969]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 277 MiB/s wr, 8.13k op/s 2024-09-15T21:59:28.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:27 smithi203 ceph-mon[35546]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 277 MiB/s wr, 8.13k op/s 2024-09-15T21:59:28.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:27 smithi174 ceph-mon[35244]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 277 MiB/s wr, 8.13k op/s 2024-09-15T21:59:30.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:29 smithi028 ceph-mon[30969]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 194 MiB/s wr, 10.44k op/s 2024-09-15T21:59:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:29 smithi203 ceph-mon[35546]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 194 MiB/s wr, 10.44k op/s 2024-09-15T21:59:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:29 smithi174 ceph-mon[35244]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 194 MiB/s wr, 10.44k op/s 2024-09-15T21:59:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:31 smithi028 ceph-mon[30969]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 64 MiB/s wr, 9.75k op/s 2024-09-15T21:59:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:31 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:59:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:31 smithi203 ceph-mon[35546]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 64 MiB/s wr, 9.75k op/s 2024-09-15T21:59:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:31 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:59:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:31 smithi174 ceph-mon[35244]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 64 MiB/s wr, 9.75k op/s 2024-09-15T21:59:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:31 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T21:59:32.353 INFO:tasks.cephfs.filesystem:scrub status for tag:05471dec-68f4-4426-9081-c55daaa5a549 - {'path': '/', 'tag': '05471dec-68f4-4426-9081-c55daaa5a549', 'options': 'recursive,force'} 2024-09-15T21:59:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:32 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:59:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:32 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:59:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:32 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:59:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:32 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:59:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:32 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:59:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:32 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:59:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:32 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T21:59:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:32 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T21:59:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:32 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T21:59:34.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:33 smithi028 ceph-mon[30969]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 99 MiB/s wr, 14.35k op/s 2024-09-15T21:59:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:33 smithi203 ceph-mon[35546]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 99 MiB/s wr, 14.35k op/s 2024-09-15T21:59:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:33 smithi174 ceph-mon[35244]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 99 MiB/s wr, 14.35k op/s 2024-09-15T21:59:36.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:35 smithi028 ceph-mon[30969]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 62 MiB/s wr, 8.64k op/s 2024-09-15T21:59:36.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:35 smithi203 ceph-mon[35546]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 62 MiB/s wr, 8.64k op/s 2024-09-15T21:59:36.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:35 smithi174 ceph-mon[35244]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 62 MiB/s wr, 8.64k op/s 2024-09-15T21:59:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:37 smithi028 ceph-mon[30969]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 65 MiB/s wr, 8.98k op/s 2024-09-15T21:59:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:37 smithi203 ceph-mon[35546]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 65 MiB/s wr, 8.98k op/s 2024-09-15T21:59:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:37 smithi174 ceph-mon[35244]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 65 MiB/s wr, 8.98k op/s 2024-09-15T21:59:38.866 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T21:59:38.866 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-15T21:59:38.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T21:59:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T21:59:40.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:39 smithi028 ceph-mon[30969]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 90 MiB/s wr, 11.94k op/s 2024-09-15T21:59:40.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:39 smithi203 ceph-mon[35546]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 90 MiB/s wr, 11.94k op/s 2024-09-15T21:59:40.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:39 smithi174 ceph-mon[35244]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 90 MiB/s wr, 11.94k op/s 2024-09-15T21:59:42.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:41 smithi028 ceph-mon[30969]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 68 MiB/s wr, 8.79k op/s 2024-09-15T21:59:42.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:41 smithi203 ceph-mon[35546]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 68 MiB/s wr, 8.79k op/s 2024-09-15T21:59:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:41 smithi174 ceph-mon[35244]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 68 MiB/s wr, 8.79k op/s 2024-09-15T21:59:44.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:43 smithi028 ceph-mon[30969]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 98 MiB/s wr, 12.76k op/s 2024-09-15T21:59:44.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:43 smithi203 ceph-mon[35546]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 98 MiB/s wr, 12.76k op/s 2024-09-15T21:59:44.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:43 smithi174 ceph-mon[35244]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 98 MiB/s wr, 12.76k op/s 2024-09-15T21:59:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:45 smithi028 ceph-mon[30969]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 69 MiB/s wr, 8.81k op/s 2024-09-15T21:59:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:45 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/430591572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:59:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:45 smithi203 ceph-mon[35546]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 69 MiB/s wr, 8.81k op/s 2024-09-15T21:59:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:45 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/430591572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:59:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:45 smithi174 ceph-mon[35244]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 69 MiB/s wr, 8.81k op/s 2024-09-15T21:59:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:45 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/430591572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T21:59:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 -- 172.21.15.28:0/1600606384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e101091f0 msgr2=0x7f2e101095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 --2- 172.21.15.28:0/1600606384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e101091f0 0x7f2e101095d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009860 tx=0x7f2e0002a3d0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 -- 172.21.15.28:0/1600606384 shutdown_connections 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 --2- 172.21.15.28:0/1600606384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e10114ab0 0x7f2e10116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 --2- 172.21.15.28:0/1600606384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10109b10 0x7f2e10114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 --2- 172.21.15.28:0/1600606384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e101091f0 0x7f2e101095d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 -- 172.21.15.28:0/1600606384 >> 172.21.15.28:0/1600606384 conn(0x7f2e1006e140 msgr2=0x7f2e1006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 -- 172.21.15.28:0/1600606384 shutdown_connections 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.879+0000 7f2e14d28640 1 -- 172.21.15.28:0/1600606384 wait complete. 2024-09-15T21:59:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.880+0000 7f2e14d28640 1 Processor -- start 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.880+0000 7f2e14d28640 1 -- start start 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e14d28640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e101091f0 0x7f2e101144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e14d28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e10109b10 0x7f2e10111550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e14d28640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10114ab0 0x7f2e10111b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e14d28640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1011c9d0 con 0x7f2e10109b10 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e14d28640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1011c850 con 0x7f2e10114ab0 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e14d28640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1011cb50 con 0x7f2e101091f0 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e101091f0 0x7f2e101144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e101091f0 0x7f2e101144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:36432/0 (socket says 172.21.15.28:36432) 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10114ab0 0x7f2e10111b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 -- 172.21.15.28:0/430591572 learned_addr learned my addr 172.21.15.28:0/430591572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0dd9b640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e10109b10 0x7f2e10111550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 -- 172.21.15.28:0/430591572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10114ab0 msgr2=0x7f2e10111b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10114ab0 0x7f2e10111b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 -- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e10109b10 msgr2=0x7f2e10111550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e10109b10 0x7f2e10111550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.881+0000 7f2e0e59c640 1 -- 172.21.15.28:0/430591572 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e101b36d0 con 0x7f2e101091f0 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.882+0000 7f2e0e59c640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e101091f0 0x7f2e101144a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e0002a8f0 tx=0x7f2e00030650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.882+0000 7f2dff7fe640 1 -- 172.21.15.28:0/430591572 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00040020 con 0x7f2e101091f0 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.882+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e101b38a0 con 0x7f2e101091f0 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.882+0000 7f2dff7fe640 1 -- 172.21.15.28:0/430591572 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e00004440 con 0x7f2e101091f0 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.882+0000 7f2dff7fe640 1 -- 172.21.15.28:0/430591572 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e000046c0 con 0x7f2e101091f0 2024-09-15T21:59:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.882+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e101b3d50 con 0x7f2e101091f0 2024-09-15T21:59:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.884+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e10073a40 con 0x7f2e101091f0 2024-09-15T21:59:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.885+0000 7f2dff7fe640 1 -- 172.21.15.28:0/430591572 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f2e00043020 con 0x7f2e101091f0 2024-09-15T21:59:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.886+0000 7f2dff7fe640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2df80a3430 0x7f2df80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T21:59:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.887+0000 7f2e0dd9b640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2df80a3430 0x7f2df80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T21:59:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.887+0000 7f2dff7fe640 1 -- 172.21.15.28:0/430591572 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2e000f0f60 con 0x7f2e101091f0 2024-09-15T21:59:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.887+0000 7f2e0dd9b640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2df80a3430 0x7f2df80a58f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e10112d10 tx=0x7f2e04005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T21:59:46.293 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"btime":"2024-09-15T21:58:48:502775+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:58:48.285105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:44.891+0000 7f2dff7fe640 1 -- 172.21.15.28:0/430591572 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2e0004a020 con 0x7f2e101091f0 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.320+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e1011ad30 con 0x7f2e101091f0 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.324+0000 7f2dff7fe640 1 -- 172.21.15.28:0/430591572 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11989 (secure 0 0 0) 0x7f2e000bd5e0 con 0x7f2e101091f0 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2df80a3430 msgr2=0x7f2df80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2df80a3430 0x7f2df80a58f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e10112d10 tx=0x7f2e04005f90 comp rx=0 tx=0).stop 2024-09-15T21:59:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e101091f0 msgr2=0x7f2e101144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e101091f0 0x7f2e101144a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e0002a8f0 tx=0x7f2e00030650 comp rx=0 tx=0).stop 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 shutdown_connections 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2df80a3430 0x7f2df80a58f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e10114ab0 0x7f2e10111b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e10109b10 0x7f2e10111550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 --2- 172.21.15.28:0/430591572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e101091f0 0x7f2e101144a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.328+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 >> 172.21.15.28:0/430591572 conn(0x7f2e1006e140 msgr2=0x7f2e100709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.329+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 shutdown_connections 2024-09-15T21:59:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T21:59:45.329+0000 7f2e14d28640 1 -- 172.21.15.28:0/430591572 wait complete. 2024-09-15T21:59:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:47 smithi028 ceph-mon[30969]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 71 MiB/s wr, 9.07k op/s 2024-09-15T21:59:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:47 smithi203 ceph-mon[35546]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 71 MiB/s wr, 9.07k op/s 2024-09-15T21:59:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:47 smithi174 ceph-mon[35244]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 71 MiB/s wr, 9.07k op/s 2024-09-15T21:59:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:49 smithi028 ceph-mon[30969]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 98 MiB/s wr, 12.72k op/s 2024-09-15T21:59:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:49 smithi203 ceph-mon[35546]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 98 MiB/s wr, 12.72k op/s 2024-09-15T21:59:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:49 smithi174 ceph-mon[35244]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 98 MiB/s wr, 12.72k op/s 2024-09-15T21:59:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:51 smithi028 ceph-mon[30969]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.87k op/s 2024-09-15T21:59:52.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:51 smithi203 ceph-mon[35546]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.87k op/s 2024-09-15T21:59:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:51 smithi174 ceph-mon[35244]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.87k op/s 2024-09-15T21:59:54.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:53 smithi028 ceph-mon[30969]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 92 MiB/s wr, 12.33k op/s 2024-09-15T21:59:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:53 smithi203 ceph-mon[35546]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 92 MiB/s wr, 12.33k op/s 2024-09-15T21:59:54.259 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:53 smithi174 ceph-mon[35244]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 92 MiB/s wr, 12.33k op/s 2024-09-15T21:59:56.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:55 smithi028 ceph-mon[30969]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.88k op/s 2024-09-15T21:59:56.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:55 smithi203 ceph-mon[35546]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.88k op/s 2024-09-15T21:59:56.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:55 smithi174 ceph-mon[35244]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.88k op/s 2024-09-15T21:59:57.887 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-15T21:59:57.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-15T21:59:58.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:57 smithi028 ceph-mon[30969]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 63 MiB/s wr, 8.50k op/s 2024-09-15T21:59:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:57 smithi203 ceph-mon[35546]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 63 MiB/s wr, 8.50k op/s 2024-09-15T21:59:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:57 smithi174 ceph-mon[35244]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 63 MiB/s wr, 8.50k op/s 2024-09-15T21:59:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:00:00.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 21:59:59 smithi028 ceph-mon[30969]: pgmap v82: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 80 MiB/s wr, 11.31k op/s 2024-09-15T22:00:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 21:59:59 smithi203 ceph-mon[35546]: pgmap v82: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 80 MiB/s wr, 11.31k op/s 2024-09-15T22:00:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 21:59:59 smithi174 ceph-mon[35244]: pgmap v82: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 80 MiB/s wr, 11.31k op/s 2024-09-15T22:00:01.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:00 smithi203 ceph-mon[35546]: overall HEALTH_OK 2024-09-15T22:00:01.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:00 smithi174 ceph-mon[35244]: overall HEALTH_OK 2024-09-15T22:00:01.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:00 smithi028 ceph-mon[30969]: overall HEALTH_OK 2024-09-15T22:00:02.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:01 smithi203 ceph-mon[35546]: pgmap v83: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 50 MiB/s wr, 7.32k op/s 2024-09-15T22:00:02.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:01 smithi174 ceph-mon[35244]: pgmap v83: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 50 MiB/s wr, 7.32k op/s 2024-09-15T22:00:02.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:00:02.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:01 smithi028 ceph-mon[30969]: pgmap v83: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 50 MiB/s wr, 7.32k op/s 2024-09-15T22:00:04.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:03 smithi028 ceph-mon[30969]: pgmap v84: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 66 MiB/s wr, 10.28k op/s 2024-09-15T22:00:04.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:03 smithi203 ceph-mon[35546]: pgmap v84: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 66 MiB/s wr, 10.28k op/s 2024-09-15T22:00:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:03 smithi174 ceph-mon[35244]: pgmap v84: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 66 MiB/s wr, 10.28k op/s 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.983+0000 7f66108d0640 1 -- 172.21.15.28:0/416835958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c101460 msgr2=0x7f660c105930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.983+0000 7f66108d0640 1 --2- 172.21.15.28:0/416835958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c101460 0x7f660c105930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6604009900 tx=0x7f660402a4a0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.984+0000 7f66108d0640 1 -- 172.21.15.28:0/416835958 shutdown_connections 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.984+0000 7f66108d0640 1 --2- 172.21.15.28:0/416835958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 0x7f660c1063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.984+0000 7f66108d0640 1 --2- 172.21.15.28:0/416835958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c101460 0x7f660c105930 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6604009900 tx=0x7f660402a4a0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.984+0000 7f66108d0640 1 --2- 172.21.15.28:0/416835958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660c100b40 0x7f660c100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.984+0000 7f66108d0640 1 -- 172.21.15.28:0/416835958 >> 172.21.15.28:0/416835958 conn(0x7f660c0fc7d0 msgr2=0x7f660c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.987+0000 7f66108d0640 1 -- 172.21.15.28:0/416835958 shutdown_connections 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.987+0000 7f66108d0640 1 -- 172.21.15.28:0/416835958 wait complete. 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.988+0000 7f66108d0640 1 Processor -- start 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.988+0000 7f66108d0640 1 -- start start 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f66108d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c100b40 0x7f660c1a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f66108d0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 0x7f660c1a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f66108d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660c19eb20 0x7f660c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 0x7f660c1a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 0x7f660c1a4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:52066/0 (socket says 172.21.15.28:52066) 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f6609d9b640 1 -- 172.21.15.28:0/2434986218 learned_addr learned my addr 172.21.15.28:0/2434986218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f66108d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1185e0 con 0x7f660c19eb20 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f66108d0640 1 -- 172.21.15.28:0/2434986218 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f660c118460 con 0x7f660c100b40 2024-09-15T22:00:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f66108d0640 1 -- 172.21.15.28:0/2434986218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f660c118760 con 0x7f660c105f50 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.989+0000 7f660a59c640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c100b40 0x7f660c1a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.990+0000 7f660ad9d640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660c19eb20 0x7f660c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.991+0000 7f6609d9b640 1 -- 172.21.15.28:0/2434986218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c100b40 msgr2=0x7f660c1a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.991+0000 7f6609d9b640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c100b40 0x7f660c1a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.991+0000 7f6609d9b640 1 -- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660c19eb20 msgr2=0x7f660c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.991+0000 7f6609d9b640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660c19eb20 0x7f660c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.991+0000 7f6609d9b640 1 -- 172.21.15.28:0/2434986218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c19f6d0 con 0x7f660c105f50 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.992+0000 7f6609d9b640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 0x7f660c1a4f90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6604005dc0 tx=0x7f6604003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.993+0000 7f66037fe640 1 -- 172.21.15.28:0/2434986218 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660403f020 con 0x7f660c105f50 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.993+0000 7f66037fe640 1 -- 172.21.15.28:0/2434986218 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6604031c90 con 0x7f660c105f50 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.993+0000 7f66037fe640 1 -- 172.21.15.28:0/2434986218 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604035070 con 0x7f660c105f50 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.994+0000 7f66108d0640 1 -- 172.21.15.28:0/2434986218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c19f960 con 0x7f660c105f50 2024-09-15T22:00:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.994+0000 7f66108d0640 1 -- 172.21.15.28:0/2434986218 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c1aba30 con 0x7f660c105f50 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:03.998+0000 7f6612ca9740 1 Processor -- start 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.000+0000 7f6612ca9740 1 -- start start 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb74180 0x56153bb72950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f66037fe640 1 -- 172.21.15.28:0/2434986218 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6604042020 con 0x7f660c105f50 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb7fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f6612ca9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56153bad7160 con 0x56153bb7d7f0 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f6612ca9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56153ba7e720 con 0x56153bb72e90 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f6612ca9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56153bad3120 con 0x56153bb74180 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f660ad9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb74180 0x56153bb72950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f660ad9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb74180 0x56153bb72950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:52072/0 (socket says 172.21.15.28:52072) 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f660ad9d640 1 -- 172.21.15.28:0/3305872451 learned_addr learned my addr 172.21.15.28:0/3305872451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f660a59c640 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.001+0000 7f6609d9b640 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb7fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.003+0000 7f660ad9d640 1 -- 172.21.15.28:0/3305872451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 msgr2=0x56153bb732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.003+0000 7f660ad9d640 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.003+0000 7f660ad9d640 1 -- 172.21.15.28:0/3305872451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 msgr2=0x56153bb7fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.005+0000 7f66037fe640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65f80a35c0 0x7f65f80a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.006+0000 7f660ad9d640 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb7fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.006+0000 7f660ad9d640 1 -- 172.21.15.28:0/3305872451 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56153b848ba0 con 0x56153bb74180 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.006+0000 7f66037fe640 1 -- 172.21.15.28:0/2434986218 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f66040f0770 con 0x7f660c105f50 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.006+0000 7f660a59c640 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.006+0000 7f660ad9d640 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb74180 0x56153bb72950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f65f400b700 tx=0x7f65f400bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c3b01ef5b111a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.007+0000 7f6609d9b640 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb7fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:00:05.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.008+0000 7f660a59c640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65f80a35c0 0x7f65f80a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.008+0000 7f66108d0640 1 -- 172.21.15.28:0/3305872451 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4003dd0 con 0x56153bb74180 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.008+0000 7f66108d0640 1 -- 172.21.15.28:0/3305872451 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f4015070 con 0x56153bb74180 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.008+0000 7f66108d0640 1 -- 172.21.15.28:0/3305872451 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4010730 con 0x56153bb74180 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.009+0000 7f6612ca9740 1 -- 172.21.15.28:0/3305872451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb74180 msgr2=0x56153bb72950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.009+0000 7f6612ca9740 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb74180 0x56153bb72950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f65f400b700 tx=0x7f65f400bbc0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.009+0000 7f660a59c640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65f80a35c0 0x7f65f80a5a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x56153bb6ff40 tx=0x7f65fc0096b0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.010+0000 7f6612ca9740 1 -- 172.21.15.28:0/3305872451 shutdown_connections 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.010+0000 7f6612ca9740 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb7fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.010+0000 7f6612ca9740 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.010+0000 7f6612ca9740 1 --2- 172.21.15.28:0/3305872451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb74180 0x56153bb72950 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.010+0000 7f6612ca9740 1 -- 172.21.15.28:0/3305872451 >> 172.21.15.28:0/3305872451 conn(0x56153bb6f0e0 msgr2=0x56153bb6f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.010+0000 7f6612ca9740 1 -- 172.21.15.28:0/3305872451 shutdown_connections 2024-09-15T22:00:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.011+0000 7f6612ca9740 1 -- 172.21.15.28:0/3305872451 wait complete. 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 -- start start 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 Processor -- start 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb72e90 0x56153bb7a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb74180 0x56153bb7aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb7b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56153bb80660 con 0x56153bb74180 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56153ba5f900 con 0x56153bb7d7f0 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56153bb807e0 con 0x56153bb72e90 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.012+0000 7f6612ca9740 10 client.0 _ll_register_callbacks cb 0x56153ba5d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb74180 0x56153bb7aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb74180 0x56153bb7aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53900/0 (socket says 172.21.15.28:53900) 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 -- 172.21.15.28:0/1970286129 learned_addr learned my addr 172.21.15.28:0/1970286129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f660a59c640 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb72e90 0x56153bb7a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 -- 172.21.15.28:0/1970286129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb72e90 msgr2=0x56153bb7a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb72e90 0x56153bb7a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 -- 172.21.15.28:0/1970286129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 msgr2=0x56153bb7b1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb7b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.013+0000 7f6609d9b640 1 -- 172.21.15.28:0/1970286129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56153ba9fc50 con 0x56153bb74180 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f6609d9b640 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb74180 0x56153bb7aca0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f66040bca10 tx=0x7f6604005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cdd5d859efa920 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f65f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f65f1ffb640 1 -- 172.21.15.28:0/1970286129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604031e40 con 0x56153bb74180 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f65f1ffb640 1 -- 172.21.15.28:0/1970286129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f66040edd80 con 0x56153bb74180 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f4015070 con 0x56153bb74180 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f65f1ffb640 1 -- 172.21.15.28:0/1970286129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f66040bd180 con 0x56153bb74180 2024-09-15T22:00:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f6612ca9740 10 client.16143 fetch_fsmap learned FSMap version 30 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.014+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56153bb8c340 con 0x56153bb74180 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f65f1ffb640 1 -- 172.21.15.28:0/1970286129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f4015070 con 0x56153bb74180 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f65f1ffb640 1 -- 172.21.15.28:0/1970286129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f66040d8030 con 0x56153bb74180 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f6612ca9740 10 client.16143 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f6612ca9740 10 client.16143 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f6612ca9740 20 client.16143 populate_metadata read hostname 'smithi028' 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bb8d6e0 0x56153bb8fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f6612ca9740 4 client.16143 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.015+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56153bae4890 con 0x56153bb8d6e0 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.016+0000 7f660a59c640 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bb8d6e0 0x56153bb8fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.018+0000 7f660a59c640 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bb8d6e0 0x56153bb8fb50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f51624412cf7ccf8 server_cookie=af9d2cde3167537f in_seq=0 out_seq=0 2024-09-15T22:00:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.018+0000 7f65f1ffb640 10 client.16143 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.020+0000 7f65f1ffb640 1 -- 172.21.15.28:0/1970286129 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56153bae4890 con 0x56153bb8d6e0 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.020+0000 7f65f1ffb640 10 client.16143 handle_command_reply: tid=0 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.021+0000 7f6612ca9740 1 client.16143 shutdown 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb74180 msgr2=0x56153bb7aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb74180 0x56153bb7aca0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f66040bca10 tx=0x7f6604005a20 comp rx=0 tx=0).stop 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 shutdown_connections 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bb8d6e0 0x56153bb8fb50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb7b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb74180 0x56153bb7aca0 secure :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f66040bca10 tx=0x7f6604005a20 comp rx=0 tx=0).stop 2024-09-15T22:00:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1970286129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb72e90 0x56153bb7a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 >> 172.21.15.28:0/1970286129 conn(0x56153bb6f0e0 msgr2=0x56153bb745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.022+0000 7f65f1ffb640 0 client.16143 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.023+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 shutdown_connections 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.023+0000 7f6612ca9740 1 -- 172.21.15.28:0/1970286129 wait complete. 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.023+0000 7f6612ca9740 20 client.16143 trim_cache size 0 max 16384 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.057+0000 7f6612ca9740 1 Processor -- start 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.058+0000 7f6612ca9740 1 -- start start 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.058+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb72e90 0x56153bb7c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.059+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb95460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.059+0000 7f660a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb72e90 0x56153bb7c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.059+0000 7f660a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb72e90 0x56153bb7c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53910/0 (socket says 172.21.15.28:53910) 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.059+0000 7f660a59c640 1 -- 172.21.15.28:0/1687531029 learned_addr learned my addr 172.21.15.28:0/1687531029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.059+0000 7f6609d9b640 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb95460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.060+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb76140 0x56153bb959a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.060+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b5b50 con 0x56153bb72e90 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.060+0000 7f660ad9d640 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb76140 0x56153bb959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.060+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b76f0 con 0x56153bb7d7f0 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.060+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b6220 con 0x56153bb76140 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.061+0000 7f6609d9b640 1 -- 172.21.15.28:0/1687531029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb76140 msgr2=0x56153bb959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.061+0000 7f6609d9b640 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb76140 0x56153bb959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.061+0000 7f6609d9b640 1 -- 172.21.15.28:0/1687531029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb72e90 msgr2=0x56153bb7c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.061+0000 7f6609d9b640 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb72e90 0x56153bb7c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.061+0000 7f6609d9b640 1 -- 172.21.15.28:0/1687531029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56153bb80540 con 0x56153bb7d7f0 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.062+0000 7f6609d9b640 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb95460 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x56153bb7bd10 tx=0x7f66040baf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527cf8c4cb6e1328 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.062+0000 7f65f3fff640 1 -- 172.21.15.28:0/1687531029 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604031e40 con 0x56153bb7d7f0 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.063+0000 7f65f3fff640 1 -- 172.21.15.28:0/1687531029 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66040bca80 con 0x56153bb7d7f0 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.063+0000 7f65f3fff640 1 -- 172.21.15.28:0/1687531029 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66040c5260 con 0x56153bb7d7f0 2024-09-15T22:00:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 msgr2=0x56153bb95460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb95460 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x56153bb7bd10 tx=0x7f66040baf90 comp rx=0 tx=0).stop 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 shutdown_connections 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb76140 0x56153bb959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb7d7f0 0x56153bb95460 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 --2- 172.21.15.28:0/1687531029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb72e90 0x56153bb7c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 >> 172.21.15.28:0/1687531029 conn(0x56153bb6f0e0 msgr2=0x56153bb7bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.064+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 shutdown_connections 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.065+0000 7f6612ca9740 1 -- 172.21.15.28:0/1687531029 wait complete. 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.065+0000 7f6612ca9740 1 -- start start 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.065+0000 7f6612ca9740 1 Processor -- start 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb86230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb82570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6612ca9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb82ab0 0x56153bb82f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6612ca9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b6fc0 con 0x56153bb7d7f0 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6612ca9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b68f0 con 0x56153bb72e90 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6612ca9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1b3b30 con 0x56153bb82ab0 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6612ca9740 10 client.0 _ll_register_callbacks cb 0x7f660402aa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb82570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb82570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53916/0 (socket says 172.21.15.28:53916) 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 -- 172.21.15.28:0/85841046 learned_addr learned my addr 172.21.15.28:0/85841046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f660a59c640 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb86230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f660ad9d640 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb82ab0 0x56153bb82f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 -- 172.21.15.28:0/85841046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb82ab0 msgr2=0x56153bb82f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb82ab0 0x56153bb82f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 -- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 msgr2=0x56153bb86230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb86230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.066+0000 7f6609d9b640 1 -- 172.21.15.28:0/85841046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56153bbf85c0 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.067+0000 7f6609d9b640 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb82570 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f66040b9190 tx=0x7f6604005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883592b8c9932a17 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.067+0000 7f65f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.068+0000 7f65f1ffb640 1 -- 172.21.15.28:0/85841046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66040bc5c0 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.068+0000 7f65f1ffb640 1 -- 172.21.15.28:0/85841046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f66040d0bb0 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.068+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66040bca80 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.068+0000 7f65f1ffb640 1 -- 172.21.15.28:0/85841046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f66040f0560 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.068+0000 7f6612ca9740 10 client.16149 fetch_fsmap learned FSMap version 30 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.068+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56153bbf85c0 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f65f1ffb640 1 -- 172.21.15.28:0/85841046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66040bca80 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f65f1ffb640 1 -- 172.21.15.28:0/85841046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f66040d8070 con 0x56153bb7d7f0 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f6612ca9740 10 client.16149 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f6612ca9740 10 client.16149 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:00:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f6612ca9740 20 client.16149 populate_metadata read hostname 'smithi028' 2024-09-15T22:00:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f6612ca9740 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bbf9be0 0x56153bbfc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f6612ca9740 4 client.16149 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-15T22:00:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x56153badebe0 con 0x56153bbf9be0 2024-09-15T22:00:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.069+0000 7f660a59c640 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bbf9be0 0x56153bbfc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.070+0000 7f65f1ffb640 10 client.16149 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:04.071+0000 7f660a59c640 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bbf9be0 0x56153bbfc000 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6784c5575d5e4293 server_cookie=d0a2c51119e81069 in_seq=0 out_seq=0 2024-09-15T22:00:05.386 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:00:06.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:05 smithi203 ceph-mon[35546]: pgmap v85: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 44 MiB/s wr, 7.20k op/s 2024-09-15T22:00:06.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:05 smithi174 ceph-mon[35244]: pgmap v85: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 44 MiB/s wr, 7.20k op/s 2024-09-15T22:00:06.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:05 smithi028 ceph-mon[30969]: pgmap v85: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 44 MiB/s wr, 7.20k op/s 2024-09-15T22:00:08.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:07 smithi203 ceph-mon[35546]: pgmap v86: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 41 MiB/s wr, 6.84k op/s 2024-09-15T22:00:08.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:07 smithi174 ceph-mon[35244]: pgmap v86: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 41 MiB/s wr, 6.84k op/s 2024-09-15T22:00:08.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:07 smithi028 ceph-mon[30969]: pgmap v86: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 41 MiB/s wr, 6.84k op/s 2024-09-15T22:00:10.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:09 smithi203 ceph-mon[35546]: pgmap v87: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 63 MiB/s wr, 9.47k op/s 2024-09-15T22:00:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:09 smithi174 ceph-mon[35244]: pgmap v87: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 63 MiB/s wr, 9.47k op/s 2024-09-15T22:00:10.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:09 smithi028 ceph-mon[30969]: pgmap v87: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 63 MiB/s wr, 9.47k op/s 2024-09-15T22:00:11.993 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:00:11.994 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:00:11.994 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:00:11.994 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.693+0000 7f21a8621640 1 -- 172.21.15.28:0/1715197919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0101420 msgr2=0x7f21a010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.693+0000 7f21a8621640 1 --2- 172.21.15.28:0/1715197919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0101420 0x7f21a010d950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2198009cd0 tx=0x7f219802ea20 comp rx=0 tx=0).stop 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.694+0000 7f21a8621640 1 -- 172.21.15.28:0/1715197919 shutdown_connections 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.694+0000 7f21a8621640 1 --2- 172.21.15.28:0/1715197919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0101420 0x7f21a010d950 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.694+0000 7f21a8621640 1 --2- 172.21.15.28:0/1715197919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a0100a80 0x7f21a0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.694+0000 7f21a8621640 1 --2- 172.21.15.28:0/1715197919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a0106a80 0x7f21a0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.694+0000 7f21a8621640 1 -- 172.21.15.28:0/1715197919 >> 172.21.15.28:0/1715197919 conn(0x7f21a00fc7a0 msgr2=0x7f21a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.698+0000 7f21a8621640 1 -- 172.21.15.28:0/1715197919 shutdown_connections 2024-09-15T22:00:11.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.698+0000 7f21a8621640 1 -- 172.21.15.28:0/1715197919 wait complete. 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 Processor -- start 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 -- start start 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a0100a80 0x7f21a006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a0101420 0x7f21a006e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0106a80 0x7f21a00733f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21a006c220 con 0x7f21a0101420 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21a006c0a0 con 0x7f21a0100a80 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.699+0000 7f21a8621640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21a006c3a0 con 0x7f21a0106a80 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0106a80 0x7f21a00733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0106a80 0x7f21a00733f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:55670/0 (socket says 172.21.15.28:55670) 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 -- 172.21.15.28:0/2034589443 learned_addr learned my addr 172.21.15.28:0/2034589443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a63bd640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a0100a80 0x7f21a006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a5bbc640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a0101420 0x7f21a006e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 -- 172.21.15.28:0/2034589443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a0100a80 msgr2=0x7f21a006dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a0100a80 0x7f21a006dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 -- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a0101420 msgr2=0x7f21a006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a0101420 0x7f21a006e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.701+0000 7f21a6bbe640 1 -- 172.21.15.28:0/2034589443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a0073c60 con 0x7f21a0106a80 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.702+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0106a80 0x7f21a00733f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2198033870 tx=0x7f21980338a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.704+0000 7f21977fe640 1 -- 172.21.15.28:0/2034589443 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198040070 con 0x7f21a0106a80 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.704+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a0073ef0 con 0x7f21a0106a80 2024-09-15T22:00:11.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.704+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a01ab5e0 con 0x7f21a0106a80 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.704+0000 7f21977fe640 1 -- 172.21.15.28:0/2034589443 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2198033b40 con 0x7f21a0106a80 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.704+0000 7f21977fe640 1 -- 172.21.15.28:0/2034589443 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198037070 con 0x7f21a0106a80 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.708+0000 7f21977fe640 1 -- 172.21.15.28:0/2034589443 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f2198043070 con 0x7f21a0106a80 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.710+0000 7f21977fe640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f218c0a35c0 0x7f218c0a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.711+0000 7f21977fe640 1 -- 172.21.15.28:0/2034589443 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f21980f0730 con 0x7f21a0106a80 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.711+0000 7f21a63bd640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f218c0a35c0 0x7f218c0a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.711+0000 7f21a63bd640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f218c0a35c0 0x7f218c0a5a80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f219c006fd0 tx=0x7f219c0090e0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.715+0000 7f21aa9fa740 1 Processor -- start 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.716+0000 7f21aa9fa740 1 -- start start 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a54880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a5f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a61a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed239ba2e0 con 0x55ed23a544a0 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ed23961ce0 con 0x55ed23a5f680 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21a6bbe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a61a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21a6bbe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a61a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56810/0 (socket says 172.21.15.28:56810) 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.717+0000 7f21a6bbe640 1 -- 172.21.15.28:0/1009506687 learned_addr learned my addr 172.21.15.28:0/1009506687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.718+0000 7f21aa9fa740 1 -- 172.21.15.28:0/1009506687 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ed239b62a0 con 0x55ed23a54dc0 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.718+0000 7f21a6bbe640 1 -- 172.21.15.28:0/1009506687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 msgr2=0x55ed23a5f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.718+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/1009506687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a5f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.718+0000 7f21a6bbe640 1 -- 172.21.15.28:0/1009506687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 msgr2=0x55ed23a54880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.718+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/1009506687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a54880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:11.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.719+0000 7f21a6bbe640 1 -- 172.21.15.28:0/1009506687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed23946440 con 0x55ed23a5f680 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.721+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/1009506687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a61a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f21980bc5e0 tx=0x7f21980098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8405f5c936ba69b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.722+0000 7f2194ff9640 1 -- 172.21.15.28:0/1009506687 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219804a040 con 0x55ed23a5f680 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.722+0000 7f2194ff9640 1 -- 172.21.15.28:0/1009506687 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21980bc650 con 0x55ed23a5f680 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.722+0000 7f2194ff9640 1 -- 172.21.15.28:0/1009506687 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21980edd40 con 0x55ed23a5f680 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.723+0000 7f21aa9fa740 1 -- 172.21.15.28:0/1009506687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 msgr2=0x55ed23a61a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.723+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/1009506687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a61a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f21980bc5e0 tx=0x7f21980098d0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.723+0000 7f21aa9fa740 1 -- 172.21.15.28:0/1009506687 shutdown_connections 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.723+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/1009506687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a61a70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.723+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/1009506687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a5f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.723+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/1009506687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a54880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.724+0000 7f21aa9fa740 1 -- 172.21.15.28:0/1009506687 >> 172.21.15.28:0/1009506687 conn(0x55ed23a50a30 msgr2=0x55ed23a50e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.725+0000 7f21aa9fa740 1 -- 172.21.15.28:0/1009506687 shutdown_connections 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.725+0000 7f21aa9fa740 1 -- 172.21.15.28:0/1009506687 wait complete. 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.726+0000 7f21aa9fa740 1 -- start start 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.726+0000 7f21aa9fa740 1 Processor -- start 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.727+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.727+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a5c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.727+0000 7f21a63bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.727+0000 7f21a63bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41768/0 (socket says 172.21.15.28:41768) 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.727+0000 7f21a63bd640 1 -- 172.21.15.28:0/4198424223 learned_addr learned my addr 172.21.15.28:0/4198424223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.727+0000 7f21a5bbc640 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a5c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.727+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a5cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed23a62010 con 0x55ed23a544a0 2024-09-15T22:00:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ed23975050 con 0x55ed23a5f680 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ed23a62190 con 0x55ed23a54dc0 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21a63bd640 1 -- 172.21.15.28:0/4198424223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 msgr2=0x55ed23a5c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21a63bd640 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a5c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21a63bd640 1 -- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 msgr2=0x55ed23a5cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21a63bd640 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a5cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21a63bd640 1 -- 172.21.15.28:0/4198424223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed2372b3c0 con 0x55ed23a544a0 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.728+0000 7f21aa9fa740 10 client.0 _ll_register_callbacks cb 0x55ed239b5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.729+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21980bc650 con 0x55ed23a544a0 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.737+0000 7f21a63bd640 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5c480 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f219c015730 tx=0x7f219c015b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a962efbc997a53 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.739+0000 7f2189ffb640 10 client.16155 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:00:12.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.739+0000 7f2189ffb640 1 -- 172.21.15.28:0/4198424223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0138d0 con 0x55ed23a544a0 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.739+0000 7f2189ffb640 1 -- 172.21.15.28:0/4198424223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f219c013a70 con 0x55ed23a544a0 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.739+0000 7f2189ffb640 1 -- 172.21.15.28:0/4198424223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f219c013c70 con 0x55ed23a544a0 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.740+0000 7f21aa9fa740 10 client.16155 fetch_fsmap learned FSMap version 30 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.740+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55ed23a6daf0 con 0x55ed23a544a0 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.753+0000 7f2189ffb640 1 -- 172.21.15.28:0/4198424223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f219c029450 con 0x55ed23a544a0 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.754+0000 7f21aa9fa740 10 client.16155 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.754+0000 7f21aa9fa740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.754+0000 7f21aa9fa740 20 client.16155 populate_metadata read hostname 'smithi028' 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.754+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23a71390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.754+0000 7f21aa9fa740 4 client.16155 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:00:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.754+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed239c7850 con 0x55ed23a6eef0 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.760+0000 7f21a5bbc640 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23a71390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.768+0000 7f21a5bbc640 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23a71390 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dcb8cb4c5d8b221 server_cookie=ce00e1e5ffdd6248 in_seq=0 out_seq=0 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.768+0000 7f2189ffb640 10 client.16155 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.769+0000 7f2189ffb640 1 -- 172.21.15.28:0/4198424223 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ed239c7850 con 0x55ed23a6eef0 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.769+0000 7f2189ffb640 10 client.16155 handle_command_reply: tid=0 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.769+0000 7f21aa9fa740 1 client.16155 shutdown 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.771+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 msgr2=0x55ed23a5c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.772+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5c480 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f219c015730 tx=0x7f219c015b30 comp rx=0 tx=0).stop 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.774+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 shutdown_connections 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.774+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23a71390 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.774+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a5cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.774+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a5c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.774+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/4198424223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5c480 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.774+0000 7f2189ffb640 0 client.16155 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.774+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 >> 172.21.15.28:0/4198424223 conn(0x55ed23a50a30 msgr2=0x55ed23a56760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.778+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 shutdown_connections 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.779+0000 7f21aa9fa740 1 -- 172.21.15.28:0/4198424223 wait complete. 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.779+0000 7f21aa9fa740 20 client.16155 trim_cache size 0 max 16384 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.819+0000 7f21aa9fa740 1 Processor -- start 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.819+0000 7f21aa9fa740 1 -- start start 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.820+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.820+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a76ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.820+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a771e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.820+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f218400feb0 con 0x55ed23a544a0 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.820+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2184011a50 con 0x55ed23a5f680 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.820+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2184010580 con 0x55ed23a54dc0 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.822+0000 7f21a6bbe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a771e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.822+0000 7f21a5bbc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a76ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.822+0000 7f21a6bbe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a771e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56834/0 (socket says 172.21.15.28:56834) 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.823+0000 7f21a6bbe640 1 -- 172.21.15.28:0/373106774 learned_addr learned my addr 172.21.15.28:0/373106774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.823+0000 7f21a6bbe640 1 -- 172.21.15.28:0/373106774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 msgr2=0x55ed23a76ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.823+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/373106774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a76ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.823+0000 7f21a6bbe640 1 -- 172.21.15.28:0/373106774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 msgr2=0x55ed23a5cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.823+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/373106774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.823+0000 7f21a6bbe640 1 -- 172.21.15.28:0/373106774 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed23a51ba0 con 0x55ed23a5f680 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.826+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/373106774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a771e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21980ba900 tx=0x7f21980bbfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f8dea9a9e153ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.828+0000 7f218bfff640 1 -- 172.21.15.28:0/373106774 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219804a040 con 0x55ed23a5f680 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.829+0000 7f218bfff640 1 -- 172.21.15.28:0/373106774 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2198004040 con 0x55ed23a5f680 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.830+0000 7f21aa9fa740 1 -- 172.21.15.28:0/373106774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 msgr2=0x55ed23a771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.830+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/373106774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a771e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21980ba900 tx=0x7f21980bbfb0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.835+0000 7f21aa9fa740 1 -- 172.21.15.28:0/373106774 shutdown_connections 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.835+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/373106774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a771e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.835+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/373106774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a54dc0 0x55ed23a76ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.835+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/373106774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a544a0 0x55ed23a5cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.835+0000 7f21aa9fa740 1 -- 172.21.15.28:0/373106774 >> 172.21.15.28:0/373106774 conn(0x55ed23a50a30 msgr2=0x55ed23a56760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.837+0000 7f21aa9fa740 1 -- 172.21.15.28:0/373106774 shutdown_connections 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.837+0000 7f21aa9fa740 1 -- 172.21.15.28:0/373106774 wait complete. 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 -- start start 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 Processor -- start 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a544a0 0x55ed23a67940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a54dc0 0x55ed23a643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2184011320 con 0x55ed23a54dc0 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2184010c50 con 0x55ed23a5f680 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.838+0000 7f21aa9fa740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f218400de90 con 0x55ed23a544a0 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.839+0000 7f21aa9fa740 10 client.0 _ll_register_callbacks cb 0x7f219c0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.839+0000 7f21a5bbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a54dc0 0x55ed23a643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.839+0000 7f21a5bbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a54dc0 0x55ed23a643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41778/0 (socket says 172.21.15.28:41778) 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.839+0000 7f21a5bbc640 1 -- 172.21.15.28:0/3392831710 learned_addr learned my addr 172.21.15.28:0/3392831710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.841+0000 7f21a63bd640 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a544a0 0x55ed23a67940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.841+0000 7f21a6bbe640 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.841+0000 7f21a5bbc640 1 -- 172.21.15.28:0/3392831710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a544a0 msgr2=0x55ed23a67940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.841+0000 7f21a5bbc640 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a544a0 0x55ed23a67940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.841+0000 7f21a5bbc640 1 -- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 msgr2=0x55ed23a648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.841+0000 7f21a5bbc640 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a648f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.841+0000 7f21a5bbc640 1 -- 172.21.15.28:0/3392831710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed23a64f30 con 0x55ed23a54dc0 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.842+0000 7f21a5bbc640 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a54dc0 0x55ed23a643b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55ed23a51890 tx=0x7f219000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7687627c0a89adab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.843+0000 7f2189ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:00:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.843+0000 7f2189ffb640 1 -- 172.21.15.28:0/3392831710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190027060 con 0x55ed23a54dc0 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.843+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2198004040 con 0x55ed23a54dc0 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.843+0000 7f2189ffb640 1 -- 172.21.15.28:0/3392831710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2190014d30 con 0x55ed23a54dc0 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.852+0000 7f2189ffb640 1 -- 172.21.15.28:0/3392831710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2190005990 con 0x55ed23a54dc0 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.854+0000 7f21aa9fa740 10 client.16158 fetch_fsmap learned FSMap version 30 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.854+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ed23adab10 con 0x55ed23a54dc0 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.855+0000 7f2189ffb640 1 -- 172.21.15.28:0/3392831710 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2198004040 con 0x55ed23a54dc0 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.856+0000 7f2189ffb640 1 -- 172.21.15.28:0/3392831710 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f2190021640 con 0x55ed23a54dc0 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.856+0000 7f21aa9fa740 10 client.16158 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.856+0000 7f21aa9fa740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.856+0000 7f21aa9fa740 20 client.16158 populate_metadata read hostname 'smithi028' 2024-09-15T22:00:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.856+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23add310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.857+0000 7f21aa9fa740 4 client.16158 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.857+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ed239c1d60 con 0x55ed23a6eef0 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.858+0000 7f21a63bd640 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23add310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.867+0000 7f2189ffb640 10 client.16158 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.867+0000 7f21a63bd640 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23add310 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28411c26f43ac4dd server_cookie=312ea9149803132c in_seq=0 out_seq=0 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.870+0000 7f2189ffb640 1 -- 172.21.15.28:0/3392831710 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ed239c1d60 con 0x55ed23a6eef0 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.870+0000 7f2189ffb640 10 client.16158 handle_command_reply: tid=0 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.871+0000 7f21aa9fa740 1 client.16158 shutdown 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a54dc0 msgr2=0x55ed23a643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a54dc0 0x55ed23a643b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55ed23a51890 tx=0x7f219000bf60 comp rx=0 tx=0).stop 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 shutdown_connections 2024-09-15T22:00:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ed23a6eef0 0x55ed23add310 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ed23a5f680 0x55ed23a648f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed23a54dc0 0x55ed23a643b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 --2- 172.21.15.28:0/3392831710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed23a544a0 0x55ed23a67940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f2189ffb640 0 client.16158 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.875+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 >> 172.21.15.28:0/3392831710 conn(0x55ed23a50a30 msgr2=0x55ed23a57050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.876+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 shutdown_connections 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.876+0000 7f21aa9fa740 1 -- 172.21.15.28:0/3392831710 wait complete. 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.877+0000 7f21aa9fa740 20 client.16158 trim_cache size 0 max 16384 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.877+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f218c0a35c0 msgr2=0x7f218c0a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.877+0000 7f21a8621640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f218c0a35c0 0x7f218c0a5a80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f219c006fd0 tx=0x7f219c0090e0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0106a80 msgr2=0x7f21a00733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0106a80 0x7f21a00733f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2198033870 tx=0x7f21980338a0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 shutdown_connections 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f218c0a35c0 0x7f218c0a5a80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21a0106a80 0x7f21a00733f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a0101420 0x7f21a006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 --2- 172.21.15.28:0/2034589443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a0100a80 0x7f21a006dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 >> 172.21.15.28:0/2034589443 conn(0x7f21a00fc7a0 msgr2=0x7f21a00fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 shutdown_connections 2024-09-15T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:10.878+0000 7f21a8621640 1 -- 172.21.15.28:0/2034589443 wait complete. 2024-09-15T22:00:12.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:11 smithi203 ceph-mon[35546]: pgmap v88: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 44 MiB/s wr, 6.39k op/s 2024-09-15T22:00:12.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:11 smithi174 ceph-mon[35244]: pgmap v88: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 44 MiB/s wr, 6.39k op/s 2024-09-15T22:00:12.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:11 smithi028 ceph-mon[30969]: pgmap v88: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 44 MiB/s wr, 6.39k op/s 2024-09-15T22:00:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:13 smithi203 ceph-mon[35546]: pgmap v89: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 61 MiB/s wr, 8.48k op/s 2024-09-15T22:00:14.259 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:13 smithi174 ceph-mon[35244]: pgmap v89: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 61 MiB/s wr, 8.48k op/s 2024-09-15T22:00:14.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:13 smithi028 ceph-mon[30969]: pgmap v89: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 61 MiB/s wr, 8.48k op/s 2024-09-15T22:00:16.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:15 smithi203 ceph-mon[35546]: pgmap v90: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 48 MiB/s wr, 6.08k op/s 2024-09-15T22:00:16.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:15 smithi174 ceph-mon[35244]: pgmap v90: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 48 MiB/s wr, 6.08k op/s 2024-09-15T22:00:16.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:15 smithi028 ceph-mon[30969]: pgmap v90: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 48 MiB/s wr, 6.08k op/s 2024-09-15T22:00:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:17 smithi028 ceph-mon[30969]: pgmap v91: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 48 MiB/s wr, 5.99k op/s 2024-09-15T22:00:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:17 smithi203 ceph-mon[35546]: pgmap v91: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 48 MiB/s wr, 5.99k op/s 2024-09-15T22:00:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:17 smithi174 ceph-mon[35244]: pgmap v91: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 48 MiB/s wr, 5.99k op/s 2024-09-15T22:00:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:19 smithi203 ceph-mon[35546]: pgmap v92: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 82 MiB/s wr, 10.18k op/s 2024-09-15T22:00:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:19 smithi174 ceph-mon[35244]: pgmap v92: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 82 MiB/s wr, 10.18k op/s 2024-09-15T22:00:20.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:19 smithi028 ceph-mon[30969]: pgmap v92: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 82 MiB/s wr, 10.18k op/s 2024-09-15T22:00:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:21 smithi203 ceph-mon[35546]: pgmap v93: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 59 MiB/s wr, 7.29k op/s 2024-09-15T22:00:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:21 smithi174 ceph-mon[35244]: pgmap v93: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 59 MiB/s wr, 7.29k op/s 2024-09-15T22:00:22.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:21 smithi028 ceph-mon[30969]: pgmap v93: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 59 MiB/s wr, 7.29k op/s 2024-09-15T22:00:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:23 smithi203 ceph-mon[35546]: pgmap v94: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 82 MiB/s wr, 10.20k op/s 2024-09-15T22:00:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:23 smithi174 ceph-mon[35244]: pgmap v94: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 82 MiB/s wr, 10.20k op/s 2024-09-15T22:00:24.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:23 smithi028 ceph-mon[30969]: pgmap v94: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 82 MiB/s wr, 10.20k op/s 2024-09-15T22:00:24.480 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:00:24.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:00:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:25 smithi174 ceph-mon[35244]: pgmap v95: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 73 MiB/s wr, 9.03k op/s 2024-09-15T22:00:26.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:25 smithi028 ceph-mon[30969]: pgmap v95: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 73 MiB/s wr, 9.03k op/s 2024-09-15T22:00:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:25 smithi203 ceph-mon[35546]: pgmap v95: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 73 MiB/s wr, 9.03k op/s 2024-09-15T22:00:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.659+0000 7f65f1ffb640 1 -- 172.21.15.28:0/85841046 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56153badebe0 con 0x56153bbf9be0 2024-09-15T22:00:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.659+0000 7f65f1ffb640 10 client.16149 handle_command_reply: tid=0 2024-09-15T22:00:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.659+0000 7f6612ca9740 1 client.16149 shutdown 2024-09-15T22:00:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 msgr2=0x56153bb82570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb82570 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f66040b9190 tx=0x7f6604005a20 comp rx=0 tx=0).stop 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 shutdown_connections 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56153bbf9be0 0x56153bbfc000 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56153bb82ab0 0x56153bb82f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56153bb7d7f0 0x56153bb82570 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 --2- 172.21.15.28:0/85841046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56153bb72e90 0x56153bb86230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 >> 172.21.15.28:0/85841046 conn(0x56153bb6f0e0 msgr2=0x56153bb7af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f65f1ffb640 0 client.16149 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 shutdown_connections 2024-09-15T22:00:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.660+0000 7f6612ca9740 1 -- 172.21.15.28:0/85841046 wait complete. 2024-09-15T22:00:26.666 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:00:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 30, 2024-09-15T22:00:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T22:00:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 6, 2024-09-15T22:00:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "1ebb5d00": { 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 6, 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 90.0, 2024-09-15T22:00:26.667 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 139.6, 2024-09-15T22:00:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-09-15T22:00:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-09-15T22:00:26.668 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:00:26.669 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:00:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.661+0000 7f6612ca9740 20 client.16149 trim_cache size 0 max 16384 2024-09-15T22:00:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.662+0000 7f66017fa640 1 -- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65f80a35c0 msgr2=0x7f65f80a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.662+0000 7f66017fa640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65f80a35c0 0x7f65f80a5a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x56153bb6ff40 tx=0x7f65fc0096b0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.662+0000 7f66017fa640 1 -- 172.21.15.28:0/2434986218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 msgr2=0x7f660c1a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.662+0000 7f66017fa640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 0x7f660c1a4f90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6604005dc0 tx=0x7f6604003b90 comp rx=0 tx=0).stop 2024-09-15T22:00:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 -- 172.21.15.28:0/2434986218 shutdown_connections 2024-09-15T22:00:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65f80a35c0 0x7f65f80a5a80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f660c19eb20 0x7f660c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f660c105f50 0x7f660c1a4f90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 --2- 172.21.15.28:0/2434986218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f660c100b40 0x7f660c1a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 -- 172.21.15.28:0/2434986218 >> 172.21.15.28:0/2434986218 conn(0x7f660c0fc7d0 msgr2=0x7f660c0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 -- 172.21.15.28:0/2434986218 shutdown_connections 2024-09-15T22:00:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:26.664+0000 7f66017fa640 1 -- 172.21.15.28:0/2434986218 wait complete. 2024-09-15T22:00:28.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:27 smithi174 ceph-mon[35244]: pgmap v96: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 70 MiB/s wr, 8.61k op/s 2024-09-15T22:00:28.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:27 smithi028 ceph-mon[30969]: pgmap v96: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 70 MiB/s wr, 8.61k op/s 2024-09-15T22:00:28.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:27 smithi203 ceph-mon[35546]: pgmap v96: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 70 MiB/s wr, 8.61k op/s 2024-09-15T22:00:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:29 smithi174 ceph-mon[35244]: pgmap v97: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 74 MiB/s wr, 9.19k op/s 2024-09-15T22:00:30.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:29 smithi028 ceph-mon[30969]: pgmap v97: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 74 MiB/s wr, 9.19k op/s 2024-09-15T22:00:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:29 smithi203 ceph-mon[35546]: pgmap v97: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 74 MiB/s wr, 9.19k op/s 2024-09-15T22:00:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:31 smithi174 ceph-mon[35244]: pgmap v98: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 45 MiB/s wr, 5.65k op/s 2024-09-15T22:00:32.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:31 smithi028 ceph-mon[30969]: pgmap v98: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 45 MiB/s wr, 5.65k op/s 2024-09-15T22:00:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:31 smithi203 ceph-mon[35546]: pgmap v98: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 45 MiB/s wr, 5.65k op/s 2024-09-15T22:00:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:33 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:00:33.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:33 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:00:33.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:33 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:00:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:34 smithi174 ceph-mon[35244]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 76 MiB/s wr, 10.27k op/s 2024-09-15T22:00:34.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:34 smithi028 ceph-mon[30969]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 76 MiB/s wr, 10.27k op/s 2024-09-15T22:00:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:34 smithi203 ceph-mon[35546]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 76 MiB/s wr, 10.27k op/s 2024-09-15T22:00:36.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:36 smithi174 ceph-mon[35244]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 56 MiB/s wr, 7.65k op/s 2024-09-15T22:00:36.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:36 smithi028 ceph-mon[30969]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 56 MiB/s wr, 7.65k op/s 2024-09-15T22:00:36.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:36 smithi203 ceph-mon[35546]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 56 MiB/s wr, 7.65k op/s 2024-09-15T22:00:38.005 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '1ebb5d00', quiesce took 59.1 seconds. Will release after: -39.55 2024-09-15T22:00:38.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:00:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:00:38.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:38 smithi028 ceph-mon[30969]: pgmap v101: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 49 MiB/s wr, 6.94k op/s 2024-09-15T22:00:38.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:38 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:00:38.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:38 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:00:38.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:38 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:00:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:38 smithi203 ceph-mon[35546]: pgmap v101: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 49 MiB/s wr, 6.94k op/s 2024-09-15T22:00:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:38 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:00:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:38 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:00:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:38 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:00:38.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:38 smithi174 ceph-mon[35244]: pgmap v101: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 49 MiB/s wr, 6.94k op/s 2024-09-15T22:00:38.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:38 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:00:38.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:38 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:00:38.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:38 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:00:40.369 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:40 smithi028 ceph-mon[30969]: pgmap v102: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 77 MiB/s wr, 10.92k op/s 2024-09-15T22:00:40.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:40 smithi203 ceph-mon[35546]: pgmap v102: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 77 MiB/s wr, 10.92k op/s 2024-09-15T22:00:40.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:40 smithi174 ceph-mon[35244]: pgmap v102: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 77 MiB/s wr, 10.92k op/s 2024-09-15T22:00:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:42 smithi028 ceph-mon[30969]: pgmap v103: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 73 MiB/s wr, 10.43k op/s 2024-09-15T22:00:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:42 smithi203 ceph-mon[35546]: pgmap v103: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 73 MiB/s wr, 10.43k op/s 2024-09-15T22:00:42.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:42 smithi174 ceph-mon[35244]: pgmap v103: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 73 MiB/s wr, 10.43k op/s 2024-09-15T22:00:44.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:44 smithi028 ceph-mon[30969]: pgmap v104: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 82 MiB/s wr, 11.96k op/s 2024-09-15T22:00:44.311 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:44 smithi174 ceph-mon[35244]: pgmap v104: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 82 MiB/s wr, 11.96k op/s 2024-09-15T22:00:44.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:44 smithi203 ceph-mon[35546]: pgmap v104: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 82 MiB/s wr, 11.96k op/s 2024-09-15T22:00:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 -- 172.21.15.28:0/1443481327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 msgr2=0x7feb60106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 --2- 172.21.15.28:0/1443481327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60106810 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7feb48009900 tx=0x7feb4802a410 comp rx=0 tx=0).stop 2024-09-15T22:00:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 -- 172.21.15.28:0/1443481327 shutdown_connections 2024-09-15T22:00:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 --2- 172.21.15.28:0/1443481327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb60106f40 0x7feb6010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 --2- 172.21.15.28:0/1443481327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60106810 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 --2- 172.21.15.28:0/1443481327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60101e10 0x7feb601021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 -- 172.21.15.28:0/1443481327 >> 172.21.15.28:0/1443481327 conn(0x7feb600fd8d0 msgr2=0x7feb600ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 -- 172.21.15.28:0/1443481327 shutdown_connections 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 -- 172.21.15.28:0/1443481327 wait complete. 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.289+0000 7feb66df3640 1 Processor -- start 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb66df3640 1 -- start start 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb66df3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60101e10 0x7feb600744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb66df3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb66df3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb60106f40 0x7feb60074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb66df3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb60111fa0 con 0x7feb601027c0 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb66df3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb60111e20 con 0x7feb60101e10 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb66df3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb60112120 con 0x7feb60106f40 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59688/0 (socket says 172.21.15.28:59688) 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 -- 172.21.15.28:0/1415489101 learned_addr learned my addr 172.21.15.28:0/1415489101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb64b8f640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60101e10 0x7feb600744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb65390640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb60106f40 0x7feb60074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 -- 172.21.15.28:0/1415489101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb60106f40 msgr2=0x7feb60074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb60106f40 0x7feb60074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 -- 172.21.15.28:0/1415489101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60101e10 msgr2=0x7feb600744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60101e10 0x7feb600744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb5ffff640 1 -- 172.21.15.28:0/1415489101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb601a6900 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb64b8f640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60101e10 0x7feb600744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.290+0000 7feb65390640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb60106f40 0x7feb60074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.291+0000 7feb5ffff640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60074a30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7feb48006010 tx=0x7feb48004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.291+0000 7feb5dffb640 1 -- 172.21.15.28:0/1415489101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4803f030 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.291+0000 7feb5dffb640 1 -- 172.21.15.28:0/1415489101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb48004440 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.291+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb601a6b90 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.291+0000 7feb5dffb640 1 -- 172.21.15.28:0/1415489101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48035070 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.291+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb601a70c0 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.292+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb601021f0 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.293+0000 7feb5dffb640 1 -- 172.21.15.28:0/1415489101 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7feb48042030 con 0x7feb601027c0 2024-09-15T22:00:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.294+0000 7feb5dffb640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feb4c0a3430 0x7feb4c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:00:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.294+0000 7feb5dffb640 1 -- 172.21.15.28:0/1415489101 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7feb480f0500 con 0x7feb601027c0 2024-09-15T22:00:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.294+0000 7feb64b8f640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feb4c0a3430 0x7feb4c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:00:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.294+0000 7feb64b8f640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feb4c0a3430 0x7feb4c0a58f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb50005fd0 tx=0x7feb50009500 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:00:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.298+0000 7feb5dffb640 1 -- 172.21.15.28:0/1415489101 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feb480bcb20 con 0x7feb601027c0 2024-09-15T22:00:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.658+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb6010a120 con 0x7feb601027c0 2024-09-15T22:00:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.659+0000 7feb5dffb640 1 -- 172.21.15.28:0/1415489101 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11989 (secure 0 0 0) 0x7feb480bc420 con 0x7feb601027c0 2024-09-15T22:00:45.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:00:45.008 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"btime":"2024-09-15T21:58:48:502775+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T21:58:48.285105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:00:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feb4c0a3430 msgr2=0x7feb4c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feb4c0a3430 0x7feb4c0a58f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb50005fd0 tx=0x7feb50009500 comp rx=0 tx=0).stop 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 msgr2=0x7feb60074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60074a30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7feb48006010 tx=0x7feb48004040 comp rx=0 tx=0).stop 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 shutdown_connections 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7feb4c0a3430 0x7feb4c0a58f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb60106f40 0x7feb60074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb601027c0 0x7feb60074a30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 --2- 172.21.15.28:0/1415489101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb60101e10 0x7feb600744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 >> 172.21.15.28:0/1415489101 conn(0x7feb600fd8d0 msgr2=0x7feb600fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 shutdown_connections 2024-09-15T22:00:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:44.661+0000 7feb66df3640 1 -- 172.21.15.28:0/1415489101 wait complete. 2024-09-15T22:00:45.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:45 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1415489101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:00:45.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:45 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1415489101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:00:45.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:45 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1415489101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:00:46.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:46 smithi028 ceph-mon[30969]: pgmap v105: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 56 MiB/s wr, 8.10k op/s 2024-09-15T22:00:46.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:46 smithi203 ceph-mon[35546]: pgmap v105: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 56 MiB/s wr, 8.10k op/s 2024-09-15T22:00:46.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:46 smithi174 ceph-mon[35244]: pgmap v105: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 56 MiB/s wr, 8.10k op/s 2024-09-15T22:00:47.983 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:00:48.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:48 smithi028 ceph-mon[30969]: pgmap v106: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 59 MiB/s wr, 8.53k op/s 2024-09-15T22:00:48.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:48 smithi203 ceph-mon[35546]: pgmap v106: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 59 MiB/s wr, 8.53k op/s 2024-09-15T22:00:48.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:48 smithi174 ceph-mon[35244]: pgmap v106: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 59 MiB/s wr, 8.53k op/s 2024-09-15T22:00:50.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:50 smithi028 ceph-mon[30969]: pgmap v107: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 71 MiB/s wr, 10.39k op/s 2024-09-15T22:00:50.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:50 smithi203 ceph-mon[35546]: pgmap v107: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 71 MiB/s wr, 10.39k op/s 2024-09-15T22:00:50.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:50 smithi174 ceph-mon[35244]: pgmap v107: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 71 MiB/s wr, 10.39k op/s 2024-09-15T22:00:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa73bfff640 1 -- 172.21.15.28:0/3095970168 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e8ef10d70 con 0x558e8f03c450 2024-09-15T22:00:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa73bfff640 10 client.25679 handle_command_reply: tid=0 2024-09-15T22:00:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 client.25679 shutdown 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 msgr2=0x558e8efc4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efc4540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa750017000 tx=0x7fa750007fc0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 shutdown_connections 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x558e8f03c450 0x558e8f03e870 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e8efc0cc0 0x558e8efc4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e8efa52d0 0x558e8efc4540 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 --2- 172.21.15.28:0/3095970168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e8efa49b0 0x558e8efc9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa73bfff640 0 client.25679 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 >> 172.21.15.28:0/3095970168 conn(0x558e8efa1060 msgr2=0x558e8efa01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 shutdown_connections 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 1 -- 172.21.15.28:0/3095970168 wait complete. 2024-09-15T22:00:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.333+0000 7fa765a25740 20 client.25679 trim_cache size 0 max 16384 2024-09-15T22:00:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:00:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:00:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:00:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7440a3690 msgr2=0x7fa7440a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7440a3690 0x7fa7440a5b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0049f0 tx=0x7fa74c02c8e0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 msgr2=0x7fa75c1a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 0x7fa75c1a46b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa75400ba50 tx=0x7fa75400bf10 comp rx=0 tx=0).stop 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 shutdown_connections 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7440a3690 0x7fa7440a5b50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa75c1069e0 0x7fa75c19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa75c101370 0x7fa75c1a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 --2- 172.21.15.28:0/299386442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa75c1009d0 0x7fa75c1a46b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:00:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 >> 172.21.15.28:0/299386442 conn(0x7fa75c0fc6b0 msgr2=0x7fa75c0fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:00:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 shutdown_connections 2024-09-15T22:00:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:00:51.334+0000 7fa762bb0640 1 -- 172.21.15.28:0/299386442 wait complete. 2024-09-15T22:00:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:52 smithi028 ceph-mon[30969]: pgmap v108: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 42 MiB/s wr, 6.27k op/s 2024-09-15T22:00:52.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:52 smithi203 ceph-mon[35546]: pgmap v108: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 42 MiB/s wr, 6.27k op/s 2024-09-15T22:00:52.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:52 smithi174 ceph-mon[35244]: pgmap v108: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 42 MiB/s wr, 6.27k op/s 2024-09-15T22:00:54.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:54 smithi028 ceph-mon[30969]: pgmap v109: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 48 MiB/s wr, 7.37k op/s 2024-09-15T22:00:54.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:54 smithi203 ceph-mon[35546]: pgmap v109: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 48 MiB/s wr, 7.37k op/s 2024-09-15T22:00:54.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:54 smithi174 ceph-mon[35244]: pgmap v109: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 48 MiB/s wr, 7.37k op/s 2024-09-15T22:00:54.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:00:55.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:55 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:00:55.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:55 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:00:55.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:55 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:00:56.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:56 smithi028 ceph-mon[30969]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 35 MiB/s wr, 5.35k op/s 2024-09-15T22:00:56.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:56 smithi203 ceph-mon[35546]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 35 MiB/s wr, 5.35k op/s 2024-09-15T22:00:56.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:56 smithi174 ceph-mon[35244]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 35 MiB/s wr, 5.35k op/s 2024-09-15T22:00:57.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:57 smithi028 ceph-mon[30969]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 30 MiB/s wr, 4.72k op/s 2024-09-15T22:00:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:57 smithi203 ceph-mon[35546]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 30 MiB/s wr, 4.72k op/s 2024-09-15T22:00:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:57 smithi174 ceph-mon[35244]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 30 MiB/s wr, 4.72k op/s 2024-09-15T22:00:59.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:00:59 smithi028 ceph-mon[30969]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 26 MiB/s wr, 4.01k op/s 2024-09-15T22:00:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:00:59 smithi203 ceph-mon[35546]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 26 MiB/s wr, 4.01k op/s 2024-09-15T22:00:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:00:59 smithi174 ceph-mon[35244]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 26 MiB/s wr, 4.01k op/s 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 -- 172.21.15.28:0/61084391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c100a80 msgr2=0x7f7b8c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 --2- 172.21.15.28:0/61084391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c100a80 0x7f7b8c100ee0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009930 tx=0x7f7b8002a420 comp rx=0 tx=0).stop 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 -- 172.21.15.28:0/61084391 shutdown_connections 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 --2- 172.21.15.28:0/61084391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b8c101420 0x7f7b8c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 --2- 172.21.15.28:0/61084391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c100a80 0x7f7b8c100ee0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 --2- 172.21.15.28:0/61084391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c106a80 0x7f7b8c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 -- 172.21.15.28:0/61084391 >> 172.21.15.28:0/61084391 conn(0x7f7b8c0fc7a0 msgr2=0x7f7b8c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 -- 172.21.15.28:0/61084391 shutdown_connections 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 -- 172.21.15.28:0/61084391 wait complete. 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.427+0000 7f7b91135640 1 Processor -- start 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b91135640 1 -- start start 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b91135640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b8c100a80 0x7f7b8c0fcd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b91135640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c101420 0x7f7b8c0fd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b91135640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c106a80 0x7f7b8c0fd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b91135640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c110e80 con 0x7f7b8c106a80 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b91135640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c110d00 con 0x7f7b8c101420 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b91135640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c111000 con 0x7f7b8c100a80 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b8b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c106a80 0x7f7b8c0fd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b8b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c106a80 0x7f7b8c0fd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43700/0 (socket says 172.21.15.28:43700) 2024-09-15T22:01:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b8b59e640 1 -- 172.21.15.28:0/878343276 learned_addr learned my addr 172.21.15.28:0/878343276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b8a59c640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c101420 0x7f7b8c0fd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.428+0000 7f7b8ad9d640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b8c100a80 0x7f7b8c0fcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b8b59e640 1 -- 172.21.15.28:0/878343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b8c100a80 msgr2=0x7f7b8c0fcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b8b59e640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b8c100a80 0x7f7b8c0fcd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b8b59e640 1 -- 172.21.15.28:0/878343276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c101420 msgr2=0x7f7b8c0fd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:00.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":31,"btime":"2024-09-15T22:00:54:046468+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:00:53.288080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24454,24458,24460,24481,24485]},"id":1}]} 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b8b59e640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c101420 0x7f7b8c0fd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b8b59e640 1 -- 172.21.15.28:0/878343276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c1a68f0 con 0x7f7b8c106a80 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b8b59e640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c106a80 0x7f7b8c0fd7e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400eaa0 tx=0x7f7b8400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b73fff640 1 -- 172.21.15.28:0/878343276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8400cdb0 con 0x7f7b8c106a80 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c1a6be0 con 0x7f7b8c106a80 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b73fff640 1 -- 172.21.15.28:0/878343276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b84005020 con 0x7f7b8c106a80 2024-09-15T22:01:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b73fff640 1 -- 172.21.15.28:0/878343276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b840116d0 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.429+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c1a70f0 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.430+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8c100ee0 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.431+0000 7f7b73fff640 1 -- 172.21.15.28:0/878343276 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f7b840054d0 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.432+0000 7f7b73fff640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7b740a33b0 0x7f7b740a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.432+0000 7f7b8ad9d640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7b740a33b0 0x7f7b740a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.432+0000 7f7b73fff640 1 -- 172.21.15.28:0/878343276 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f7b840d01f0 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.433+0000 7f7b8ad9d640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7b740a33b0 0x7f7b740a5870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00bb60 tx=0x7f7b7c00b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.437+0000 7f7b73fff640 1 -- 172.21.15.28:0/878343276 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7b84016030 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.648+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b8c10a020 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.649+0000 7f7b73fff640 1 -- 172.21.15.28:0/878343276 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11987 (secure 0 0 0) 0x7f7b8409c220 con 0x7f7b8c106a80 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 31 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7b740a33b0 msgr2=0x7f7b740a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7b740a33b0 0x7f7b740a5870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00bb60 tx=0x7f7b7c00b390 comp rx=0 tx=0).stop 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c106a80 msgr2=0x7f7b8c0fd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c106a80 0x7f7b8c0fd7e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f7b8400eaa0 tx=0x7f7b8400ef60 comp rx=0 tx=0).stop 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 shutdown_connections 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f7b740a33b0 0x7f7b740a5870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c106a80 0x7f7b8c0fd7e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c101420 0x7f7b8c0fd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 --2- 172.21.15.28:0/878343276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b8c100a80 0x7f7b8c0fcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 >> 172.21.15.28:0/878343276 conn(0x7f7b8c0fc7a0 msgr2=0x7f7b8c06aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 shutdown_connections 2024-09-15T22:01:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:00:59.651+0000 7f7b91135640 1 -- 172.21.15.28:0/878343276 wait complete. 2024-09-15T22:01:00.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:00 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/878343276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:01:00.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:00 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/878343276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:01:00.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:00 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/878343276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:01:01.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:01 smithi203 ceph-mon[35546]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 MiB/s wr, 1.94k op/s 2024-09-15T22:01:01.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:01 smithi028 ceph-mon[30969]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 MiB/s wr, 1.94k op/s 2024-09-15T22:01:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:01 smithi174 ceph-mon[35244]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 MiB/s wr, 1.94k op/s 2024-09-15T22:01:03.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:03 smithi028 ceph-mon[30969]: pgmap v114: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 MiB/s wr, 1.95k op/s 2024-09-15T22:01:03.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:03 smithi203 ceph-mon[35546]: pgmap v114: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 MiB/s wr, 1.95k op/s 2024-09-15T22:01:03.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:03 smithi174 ceph-mon[35244]: pgmap v114: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 12 MiB/s wr, 1.95k op/s 2024-09-15T22:01:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 -- 172.21.15.28:0/1097377231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988101ae0 msgr2=0x7f6988101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 --2- 172.21.15.28:0/1097377231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988101ae0 0x7f6988101f40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6970009930 tx=0x7f697002a420 comp rx=0 tx=0).stop 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 -- 172.21.15.28:0/1097377231 shutdown_connections 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 --2- 172.21.15.28:0/1097377231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6988102480 0x7f698810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 --2- 172.21.15.28:0/1097377231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988101ae0 0x7f6988101f40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 --2- 172.21.15.28:0/1097377231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6988107b80 0x7f6988107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 -- 172.21.15.28:0/1097377231 >> 172.21.15.28:0/1097377231 conn(0x7f69880fd7e0 msgr2=0x7f69880ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 -- 172.21.15.28:0/1097377231 shutdown_connections 2024-09-15T22:01:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.448+0000 7f698e45e640 1 -- 172.21.15.28:0/1097377231 wait complete. 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 Processor -- start 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 -- start start 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6988101ae0 0x7f6988198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988102480 0x7f69881988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6988107b80 0x7f698819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6988111ef0 con 0x7f6988102480 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6988111d70 con 0x7f6988101ae0 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.449+0000 7f698e45e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6988112070 con 0x7f6988107b80 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988102480 0x7f69881988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988102480 0x7f69881988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47488/0 (socket says 172.21.15.28:47488) 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 -- 172.21.15.28:0/4101717723 learned_addr learned my addr 172.21.15.28:0/4101717723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f6987fff640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6988101ae0 0x7f6988198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f698c9fb640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6988107b80 0x7f698819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 -- 172.21.15.28:0/4101717723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6988107b80 msgr2=0x7f698819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6988107b80 0x7f698819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 -- 172.21.15.28:0/4101717723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6988101ae0 msgr2=0x7f6988198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6988101ae0 0x7f6988198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 -- 172.21.15.28:0/4101717723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698819d370 con 0x7f6988102480 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f698c9fb640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6988107b80 0x7f698819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f697ffff640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988102480 0x7f69881988c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6970009230 tx=0x7f6970004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f6985ffb640 1 -- 172.21.15.28:0/4101717723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697003f030 con 0x7f6988102480 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f6985ffb640 1 -- 172.21.15.28:0/4101717723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6970004440 con 0x7f6988102480 2024-09-15T22:01:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.450+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698819d600 con 0x7f6988102480 2024-09-15T22:01:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.451+0000 7f6985ffb640 1 -- 172.21.15.28:0/4101717723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970035070 con 0x7f6988102480 2024-09-15T22:01:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.451+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69881a4ef0 con 0x7f6988102480 2024-09-15T22:01:05.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.453+0000 7f6985ffb640 1 -- 172.21.15.28:0/4101717723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6970042030 con 0x7f6988102480 2024-09-15T22:01:05.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.454+0000 7f6985ffb640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f69740a3460 0x7f69740a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.454+0000 7f6987fff640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f69740a3460 0x7f69740a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.454+0000 7f6985ffb640 1 -- 172.21.15.28:0/4101717723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f69700f0470 con 0x7f6988102480 2024-09-15T22:01:05.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.455+0000 7f6987fff640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f69740a3460 0x7f69740a5920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f69780095f0 tx=0x7f6978005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:05.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:05 smithi028 ceph-mon[30969]: pgmap v115: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 2.7 MiB/s wr, 479 op/s 2024-09-15T22:01:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 Processor -- start 2024-09-15T22:01:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 -- start start 2024-09-15T22:01:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e395ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e3a0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a95e2fa2f0 con 0x55a95e3a0cb0 2024-09-15T22:01:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a95e2a2270 con 0x55a95e395af0 2024-09-15T22:01:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f69911fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a95e2f62b0 con 0x55a95e396410 2024-09-15T22:01:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f698c9fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f698c9fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47502/0 (socket says 172.21.15.28:47502) 2024-09-15T22:01:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f6987fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e395ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f698c9fb640 1 -- 172.21.15.28:0/3919070849 learned_addr learned my addr 172.21.15.28:0/3919070849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.704+0000 7f697ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e3a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697ffff640 1 -- 172.21.15.28:0/3919070849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 msgr2=0x55a95e395ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697ffff640 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e395ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697ffff640 1 -- 172.21.15.28:0/3919070849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 msgr2=0x55a95e3a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697ffff640 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697ffff640 1 -- 172.21.15.28:0/3919070849 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a95e2a1340 con 0x55a95e396410 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f698c9fb640 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f6987fff640 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e395ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697ffff640 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e3a0770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f69700ed800 tx=0x7f69700bbdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d86a17dd58e4ad01 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697e7fc640 1 -- 172.21.15.28:0/3919070849 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970049050 con 0x55a95e396410 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697e7fc640 1 -- 172.21.15.28:0/3919070849 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69700c5000 con 0x55a95e396410 2024-09-15T22:01:05.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.705+0000 7f697e7fc640 1 -- 172.21.15.28:0/3919070849 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69700edbf0 con 0x55a95e396410 2024-09-15T22:01:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.706+0000 7f69911fb740 1 -- 172.21.15.28:0/3919070849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 msgr2=0x55a95e3a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.706+0000 7f69911fb740 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e3a0770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f69700ed800 tx=0x7f69700bbdb0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.706+0000 7f69911fb740 1 -- 172.21.15.28:0/3919070849 shutdown_connections 2024-09-15T22:01:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.706+0000 7f69911fb740 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.706+0000 7f69911fb740 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e3a0770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.706+0000 7f69911fb740 1 --2- 172.21.15.28:0/3919070849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e395ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.706+0000 7f69911fb740 1 -- 172.21.15.28:0/3919070849 >> 172.21.15.28:0/3919070849 conn(0x55a95e392060 msgr2=0x55a95e392440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.707+0000 7f69911fb740 1 -- 172.21.15.28:0/3919070849 shutdown_connections 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.707+0000 7f69911fb740 1 -- 172.21.15.28:0/3919070849 wait complete. 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.707+0000 7f69911fb740 1 -- start start 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.707+0000 7f69911fb740 1 Processor -- start 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f69911fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e39dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f69911fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e39e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f69911fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e39e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f69911fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a95e3a3640 con 0x55a95e3a0cb0 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f69911fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a95e283290 con 0x55a95e395af0 2024-09-15T22:01:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f69911fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a95e3a37c0 con 0x55a95e396410 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f69911fb740 10 client.0 _ll_register_callbacks cb 0x55a95e2b4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e39e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e39e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47518/0 (socket says 172.21.15.28:47518) 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 -- 172.21.15.28:0/589634930 learned_addr learned my addr 172.21.15.28:0/589634930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f6987fff640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e39dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f697ffff640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e39e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 -- 172.21.15.28:0/589634930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 msgr2=0x55a95e39e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e39e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 -- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 msgr2=0x55a95e39dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e39dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.708+0000 7f698c9fb640 1 -- 172.21.15.28:0/589634930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a95e00d210 con 0x55a95e3a0cb0 2024-09-15T22:01:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f697ffff640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e39e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f6987fff640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e39dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f698c9fb640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e39e710 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6980011ce0 tx=0x7f69800107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6abb64a0c0f22 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f69677fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f69677fe640 1 -- 172.21.15.28:0/589634930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6980004b20 con 0x55a95e3a0cb0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f69677fe640 1 -- 172.21.15.28:0/589634930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f6980004cc0 con 0x55a95e3a0cb0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69700c5000 con 0x55a95e3a0cb0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f69677fe640 1 -- 172.21.15.28:0/589634930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6980025050 con 0x55a95e3a0cb0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f69911fb740 10 client.16182 fetch_fsmap learned FSMap version 31 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.709+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a95e0d9620 con 0x55a95e3a0cb0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69677fe640 1 -- 172.21.15.28:0/589634930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69700c5000 con 0x55a95e3a0cb0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69677fe640 1 -- 172.21.15.28:0/589634930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f698001a590 con 0x55a95e3a0cb0 2024-09-15T22:01:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69911fb740 10 client.16182 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69911fb740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69911fb740 20 client.16182 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69911fb740 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e3b0a40 0x55a95e3b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69911fb740 4 client.16182 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a95e307a20 con 0x55a95e3b0a40 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.710+0000 7f6987fff640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e3b0a40 0x55a95e3b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.711+0000 7f6987fff640 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e3b0a40 0x55a95e3b2e30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a570f32e9f3437a server_cookie=3a2c149f5f4c5393 in_seq=0 out_seq=0 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.711+0000 7f69677fe640 10 client.16182 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69677fe640 1 -- 172.21.15.28:0/589634930 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a95e307a20 con 0x55a95e3b0a40 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69677fe640 10 client.16182 handle_command_reply: tid=0 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 client.16182 shutdown 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 msgr2=0x55a95e39e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e39e710 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6980011ce0 tx=0x7f69800107a0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 shutdown_connections 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e3b0a40 0x55a95e3b2e30 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e39e710 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e396410 0x55a95e39e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 --2- 172.21.15.28:0/589634930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e39dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69677fe640 0 client.16182 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 >> 172.21.15.28:0/589634930 conn(0x55a95e392060 msgr2=0x55a95e397d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 shutdown_connections 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 1 -- 172.21.15.28:0/589634930 wait complete. 2024-09-15T22:01:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.712+0000 7f69911fb740 20 client.16182 trim_cache size 0 max 16384 2024-09-15T22:01:05.724 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:05 smithi203 ceph-mon[35546]: pgmap v115: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 2.7 MiB/s wr, 479 op/s 2024-09-15T22:01:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 Processor -- start 2024-09-15T22:01:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 -- start start 2024-09-15T22:01:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e398560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e396410 0x55a95e39e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e3a0cb0 0x55a95e3b8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69881aee50 con 0x55a95e396410 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69881b09c0 con 0x55a95e395af0 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f69911fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69881af520 con 0x55a95e3a0cb0 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f697ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e396410 0x55a95e39e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f697ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e396410 0x55a95e39e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47532/0 (socket says 172.21.15.28:47532) 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f697ffff640 1 -- 172.21.15.28:0/3707553069 learned_addr learned my addr 172.21.15.28:0/3707553069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f698c9fb640 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e3a0cb0 0x55a95e3b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f6987fff640 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e398560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f697ffff640 1 -- 172.21.15.28:0/3707553069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e3a0cb0 msgr2=0x55a95e3b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.728+0000 7f697ffff640 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e3a0cb0 0x55a95e3b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f697ffff640 1 -- 172.21.15.28:0/3707553069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 msgr2=0x55a95e398560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f697ffff640 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e398560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f697ffff640 1 -- 172.21.15.28:0/3707553069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a95e3b3490 con 0x55a95e396410 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f6987fff640 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e398560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f697ffff640 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e396410 0x55a95e39e540 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f69700ed860 tx=0x7f6970009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efeb5606470ff99d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f697dffb640 1 -- 172.21.15.28:0/3707553069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69700f01b0 con 0x55a95e396410 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f697dffb640 1 -- 172.21.15.28:0/3707553069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6970003d20 con 0x55a95e396410 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.729+0000 7f697dffb640 1 -- 172.21.15.28:0/3707553069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970031e30 con 0x55a95e396410 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 -- 172.21.15.28:0/3707553069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e396410 msgr2=0x55a95e39e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e396410 0x55a95e39e540 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f69700ed860 tx=0x7f6970009260 comp rx=0 tx=0).stop 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 -- 172.21.15.28:0/3707553069 shutdown_connections 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e3a0cb0 0x55a95e3b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e396410 0x55a95e39e540 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 --2- 172.21.15.28:0/3707553069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e395af0 0x55a95e398560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 -- 172.21.15.28:0/3707553069 >> 172.21.15.28:0/3707553069 conn(0x55a95e392060 msgr2=0x55a95e39fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 -- 172.21.15.28:0/3707553069 shutdown_connections 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 -- 172.21.15.28:0/3707553069 wait complete. 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 -- start start 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.730+0000 7f69911fb740 1 Processor -- start 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f69911fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e395af0 0x55a95e3a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f69911fb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e396410 0x55a95e3a4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f69911fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f69911fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69881b02c0 con 0x55a95e3a0cb0 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f69911fb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69881afbf0 con 0x55a95e396410 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f69911fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69881ace30 con 0x55a95e395af0 2024-09-15T22:01:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f69911fb740 10 client.0 _ll_register_callbacks cb 0x7f6980009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47548/0 (socket says 172.21.15.28:47548) 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 -- 172.21.15.28:0/407303340 learned_addr learned my addr 172.21.15.28:0/407303340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f697ffff640 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e396410 0x55a95e3a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f6987fff640 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e395af0 0x55a95e3a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 -- 172.21.15.28:0/407303340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e395af0 msgr2=0x55a95e3a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e395af0 0x55a95e3a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 -- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e396410 msgr2=0x55a95e3a4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e396410 0x55a95e3a4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.731+0000 7f698c9fb640 1 -- 172.21.15.28:0/407303340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a95e392cf0 con 0x55a95e3a0cb0 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f698c9fb640 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a46b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f698001aa30 tx=0x7f698001a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0415d38659b9dd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69677fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69677fe640 1 -- 172.21.15.28:0/407303340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698001a780 con 0x55a95e3a0cb0 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69677fe640 1 -- 172.21.15.28:0/407303340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f6980034480 con 0x55a95e3a0cb0 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6970003d20 con 0x55a95e3a0cb0 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69677fe640 1 -- 172.21.15.28:0/407303340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f698003d050 con 0x55a95e3a0cb0 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69911fb740 10 client.16188 fetch_fsmap learned FSMap version 31 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a95e41b220 con 0x55a95e3a0cb0 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.732+0000 7f69677fe640 1 -- 172.21.15.28:0/407303340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6970003d20 con 0x55a95e3a0cb0 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f69677fe640 1 -- 172.21.15.28:0/407303340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f6980029700 con 0x55a95e3a0cb0 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f69911fb740 10 client.16188 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T22:01:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f69911fb740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f69911fb740 20 client.16188 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f69911fb740 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e41b940 0x55a95e41dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f69911fb740 4 client.16188 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:01:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a95e301d70 con 0x55a95e41b940 2024-09-15T22:01:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.733+0000 7f6987fff640 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e41b940 0x55a95e41dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.734+0000 7f69677fe640 10 client.16188 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.734+0000 7f6987fff640 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e41b940 0x55a95e41dd30 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3db32f6d07113f18 server_cookie=47e95b163311e6b8 in_seq=0 out_seq=0 2024-09-15T22:01:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:05 smithi174 ceph-mon[35244]: pgmap v115: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 2.7 MiB/s wr, 479 op/s 2024-09-15T22:01:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69677fe640 1 -- 172.21.15.28:0/407303340 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a95e301d70 con 0x55a95e41b940 2024-09-15T22:01:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69677fe640 10 client.16188 handle_command_reply: tid=0 2024-09-15T22:01:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 client.16188 shutdown 2024-09-15T22:01:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 msgr2=0x55a95e3a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a46b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f698001aa30 tx=0x7f698001a590 comp rx=0 tx=0).stop 2024-09-15T22:01:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 shutdown_connections 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a95e41b940 0x55a95e41dd30 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69677fe640 0 client.16188 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a95e3a0cb0 0x55a95e3a46b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a95e396410 0x55a95e3a4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 --2- 172.21.15.28:0/407303340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a95e395af0 0x55a95e3a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.775+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 >> 172.21.15.28:0/407303340 conn(0x55a95e392060 msgr2=0x55a95e3909a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.776+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 shutdown_connections 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.776+0000 7f69911fb740 1 -- 172.21.15.28:0/407303340 wait complete. 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.776+0000 7f69911fb740 20 client.16188 trim_cache size 0 max 16384 2024-09-15T22:01:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:01:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:01:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:01:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:01:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f69740a3460 msgr2=0x7f69740a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f69740a3460 0x7f69740a5920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f69780095f0 tx=0x7f6978005f90 comp rx=0 tx=0).stop 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988102480 msgr2=0x7f69881988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988102480 0x7f69881988c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6970009230 tx=0x7f6970004040 comp rx=0 tx=0).stop 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 shutdown_connections 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f69740a3460 0x7f69740a5920 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6988107b80 0x7f698819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6988102480 0x7f69881988c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 --2- 172.21.15.28:0/4101717723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6988101ae0 0x7f6988198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 >> 172.21.15.28:0/4101717723 conn(0x7f69880fd7e0 msgr2=0x7f69880fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 shutdown_connections 2024-09-15T22:01:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:05.777+0000 7f698e45e640 1 -- 172.21.15.28:0/4101717723 wait complete. 2024-09-15T22:01:07.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:07 smithi028 ceph-mon[30969]: pgmap v116: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 853 KiB/s wr, 171 op/s 2024-09-15T22:01:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:07 smithi203 ceph-mon[35546]: pgmap v116: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 853 KiB/s wr, 171 op/s 2024-09-15T22:01:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:07 smithi174 ceph-mon[35244]: pgmap v116: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 853 KiB/s wr, 171 op/s 2024-09-15T22:01:09.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:09 smithi028 ceph-mon[30969]: pgmap v117: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 46 op/s 2024-09-15T22:01:09.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:09 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:01:09.675 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id 1ebb5d00 --release --await' 2024-09-15T22:01:09.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id 1ebb5d00 --release --await 2024-09-15T22:01:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:09 smithi203 ceph-mon[35546]: pgmap v117: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 46 op/s 2024-09-15T22:01:09.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:09 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:01:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:09 smithi174 ceph-mon[35244]: pgmap v117: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 66 KiB/s wr, 46 op/s 2024-09-15T22:01:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:09 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:01:09.770 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:01:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:11 smithi028 ceph-mon[30969]: pgmap v118: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 42 op/s 2024-09-15T22:01:11.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:11 smithi203 ceph-mon[35546]: pgmap v118: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 42 op/s 2024-09-15T22:01:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:11 smithi174 ceph-mon[35244]: pgmap v118: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 42 op/s 2024-09-15T22:01:13.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:13 smithi028 ceph-mon[30969]: pgmap v119: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 49 op/s 2024-09-15T22:01:13.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:13 smithi203 ceph-mon[35546]: pgmap v119: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 49 op/s 2024-09-15T22:01:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:13 smithi174 ceph-mon[35244]: pgmap v119: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 58 KiB/s wr, 49 op/s 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 -- 172.21.15.28:0/1914187399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c1037a0 msgr2=0x7f2d4c10bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 --2- 172.21.15.28:0/1914187399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c1037a0 0x7f2d4c10bc00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009900 tx=0x7f2d4002a410 comp rx=0 tx=0).stop 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 -- 172.21.15.28:0/1914187399 shutdown_connections 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 --2- 172.21.15.28:0/1914187399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d4c10c2f0 0x7f2d4c10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 --2- 172.21.15.28:0/1914187399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c1037a0 0x7f2d4c10bc00 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 --2- 172.21.15.28:0/1914187399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d4c102df0 0x7f2d4c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 -- 172.21.15.28:0/1914187399 >> 172.21.15.28:0/1914187399 conn(0x7f2d4c0fc7d0 msgr2=0x7f2d4c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 -- 172.21.15.28:0/1914187399 shutdown_connections 2024-09-15T22:01:14.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.463+0000 7f2d52a45640 1 -- 172.21.15.28:0/1914187399 wait complete. 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 Processor -- start 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 -- start start 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d4c102df0 0x7f2d4c100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d4c1037a0 0x7f2d4c100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c2f0 0x7f2d4c1014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1120b0 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c111f30 con 0x7f2d4c1037a0 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.464+0000 7f2d52a45640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c112230 con 0x7f2d4c102df0 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c2f0 0x7f2d4c1014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c2f0 0x7f2d4c1014d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54706/0 (socket says 172.21.15.28:54706) 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 -- 172.21.15.28:0/158549519 learned_addr learned my addr 172.21.15.28:0/158549519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d4c1037a0 0x7f2d4c100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d4c102df0 0x7f2d4c100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 -- 172.21.15.28:0/158549519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d4c102df0 msgr2=0x7f2d4c100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d4c102df0 0x7f2d4c100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 -- 172.21.15.28:0/158549519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d4c1037a0 msgr2=0x7f2d4c100f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d4c1037a0 0x7f2d4c100f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 -- 172.21.15.28:0/158549519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c1a6a50 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d50fe2640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c2f0 0x7f2d4c1014d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400cce0 tx=0x7f2d4400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d497fa640 1 -- 172.21.15.28:0/158549519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4400f450 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c1a6d40 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d497fa640 1 -- 172.21.15.28:0/158549519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d4400fad0 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d497fa640 1 -- 172.21.15.28:0/158549519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44018c10 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.465+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c1a7250 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.468+0000 7f2d54e1e740 1 Processor -- start 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.468+0000 7f2d497fa640 1 -- 172.21.15.28:0/158549519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f2d4400f5f0 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d54e1e740 1 -- start start 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fd5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fe0370 0x561696fe2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561696f3b2f0 con 0x561696fd6b40 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561696ee2cf0 con 0x561696fd55d0 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561696f372b0 con 0x561696fe0370 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54710/0 (socket says 172.21.15.28:54710) 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/100341384 learned_addr learned my addr 172.21.15.28:0/100341384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:14.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fe0370 0x561696fe2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fd5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d497fa640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2d340a33b0 0x7f2d340a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/100341384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fe0370 msgr2=0x561696fe2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fe0370 0x561696fe2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/100341384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 msgr2=0x561696fd5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fd5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/100341384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561696ec73a0 con 0x561696fd6b40 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.469+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fd5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.470+0000 7f2d497fa640 1 -- 172.21.15.28:0/158549519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2d440d0650 con 0x7f2d4c10c2f0 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.470+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd5090 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2d400096c0 tx=0x7f2d40000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb248defcd701c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.470+0000 7f2d50fe2640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2d340a33b0 0x7f2d340a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.470+0000 7f2d327fc640 1 -- 172.21.15.28:0/100341384 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40044030 con 0x561696fd6b40 2024-09-15T22:01:14.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.470+0000 7f2d327fc640 1 -- 172.21.15.28:0/100341384 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d400317f0 con 0x561696fd6b40 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.470+0000 7f2d327fc640 1 -- 172.21.15.28:0/100341384 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40035040 con 0x561696fd6b40 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 -- 172.21.15.28:0/100341384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 msgr2=0x561696fd5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d50fe2640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2d340a33b0 0x7f2d340a5870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2d440a4d10 tx=0x7f2d4400af30 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd5090 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2d400096c0 tx=0x7f2d40000bc0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 -- 172.21.15.28:0/100341384 shutdown_connections 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fe0370 0x561696fe2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fd5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/100341384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd5090 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 -- 172.21.15.28:0/100341384 >> 172.21.15.28:0/100341384 conn(0x561696f9e310 msgr2=0x561696fd1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 -- 172.21.15.28:0/100341384 shutdown_connections 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 -- 172.21.15.28:0/100341384 wait complete. 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 -- start start 2024-09-15T22:01:14.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.471+0000 7f2d54e1e740 1 Processor -- start 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696fdfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696fdd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fe0370 0x561696fdd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561696fe3010 con 0x561696fe0370 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561696ef6060 con 0x561696fd6b40 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561696fe3190 con 0x561696fd55d0 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d54e1e740 10 client.0 _ll_register_callbacks cb 0x561696c498c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fe0370 0x561696fdd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fe0370 0x561696fdd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54726/0 (socket says 172.21.15.28:54726) 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/1903227062 learned_addr learned my addr 172.21.15.28:0/1903227062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696fdfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696fdd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 msgr2=0x561696fdfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696fdfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/1903227062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 msgr2=0x561696fdd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696fdd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/1903227062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561696cac3c0 con 0x561696fe0370 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.472+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696fdd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fe0370 0x561696fdd650 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009430 tx=0x7f2d400052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ea81a83e25801a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/1903227062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40044070 con 0x561696fe0370 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/1903227062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2d40004230 con 0x561696fe0370 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d400317f0 con 0x561696fe0370 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/1903227062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2d40035040 con 0x561696fe0370 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 10 client.16200 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:14.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561696feeb10 con 0x561696fe0370 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/1903227062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d400317f0 con 0x561696fe0370 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/1903227062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f2d40003920 con 0x561696fe0370 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 10 client.16200 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 10 client.16200 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 20 client.16200 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561696fefe60 0x561696ff2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 4 client.16200 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.473+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561696f48860 con 0x561696fefe60 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.474+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561696fefe60 0x561696ff2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.474+0000 7f2d2b7fe640 10 client.16200 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.474+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561696fefe60 0x561696ff2250 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a29c2fc76d9b5882 server_cookie=87c3931852d272a in_seq=0 out_seq=0 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/1903227062 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561696f48860 con 0x561696fefe60 2024-09-15T22:01:14.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d2b7fe640 10 client.16200 handle_command_reply: tid=0 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 client.16200 shutdown 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fe0370 msgr2=0x561696fdd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fe0370 0x561696fdd650 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009430 tx=0x7f2d400052d0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 shutdown_connections 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561696fefe60 0x561696ff2250 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fe0370 0x561696fdd650 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696fdd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1903227062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696fdfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 >> 172.21.15.28:0/1903227062 conn(0x561696f9e310 msgr2=0x561696fd7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d2b7fe640 0 client.16200 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 shutdown_connections 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1903227062 wait complete. 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.475+0000 7f2d54e1e740 20 client.16200 trim_cache size 0 max 16384 2024-09-15T22:01:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.488+0000 7f2d54e1e740 1 Processor -- start 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.488+0000 7f2d54e1e740 1 -- start start 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fec460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fdd710 0x561696fd8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b0fb0 con 0x561696fd6b40 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b2af0 con 0x561696fd55d0 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b1680 con 0x561696fdd710 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54738/0 (socket says 172.21.15.28:54738) 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 -- 172.21.15.28:0/1893234118 learned_addr learned my addr 172.21.15.28:0/1893234118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fdd710 0x561696fd8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 -- 172.21.15.28:0/1893234118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fdd710 msgr2=0x561696fd8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fdd710 0x561696fd8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 -- 172.21.15.28:0/1893234118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 msgr2=0x561696fec460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fec460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 -- 172.21.15.28:0/1893234118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561696fdb5b0 con 0x561696fd6b40 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.489+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd8120 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00b740 tx=0x7f2d3c00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8115144a997c4cfc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d31ffb640 1 -- 172.21.15.28:0/1893234118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c022070 con 0x561696fd6b40 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d31ffb640 1 -- 172.21.15.28:0/1893234118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d3c003e40 con 0x561696fd6b40 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d31ffb640 1 -- 172.21.15.28:0/1893234118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c015070 con 0x561696fd6b40 2024-09-15T22:01:14.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1893234118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 msgr2=0x561696fd8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd8120 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c00b740 tx=0x7f2d3c00be10 comp rx=0 tx=0).stop 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1893234118 shutdown_connections 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fdd710 0x561696fd8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fd6b40 0x561696fd8120 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/1893234118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd55d0 0x561696fec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1893234118 >> 172.21.15.28:0/1893234118 conn(0x561696f9e310 msgr2=0x561696fdd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1893234118 shutdown_connections 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.490+0000 7f2d54e1e740 1 -- 172.21.15.28:0/1893234118 wait complete. 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 -- start start 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 Processor -- start 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696ff3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696ff3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fdd710 0x561696ff3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b2420 con 0x561696fdd710 2024-09-15T22:01:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1b1d50 con 0x561696fd6b40 2024-09-15T22:01:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1aef90 con 0x561696fd55d0 2024-09-15T22:01:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d54e1e740 10 client.0 _ll_register_callbacks cb 0x7f2d40032f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:14.491 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:01:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fdd710 0x561696ff3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fdd710 0x561696ff3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54752/0 (socket says 172.21.15.28:54752) 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/3125025918 learned_addr learned my addr 172.21.15.28:0/3125025918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.491+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696ff3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696ff3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 msgr2=0x561696ff3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696ff3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/3125025918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 msgr2=0x561696ff3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696ff3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d4b7fe640 1 -- 172.21.15.28:0/3125025918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561697059a70 con 0x561696fdd710 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d4b7fe640 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fdd710 0x561696ff3dc0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2d40003e60 tx=0x7f2d40040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a60d9f02f867453 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/3125025918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40044070 con 0x561696fdd710 2024-09-15T22:01:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/3125025918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2d400035f0 con 0x561696fdd710 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d3c003e40 con 0x561696fdd710 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/3125025918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2d4005b040 con 0x561696fdd710 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d54e1e740 10 client.16206 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56169705ad70 con 0x561696fdd710 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.492+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/3125025918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d3c003e40 con 0x561696fdd710 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/3125025918 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f2d40061040 con 0x561696fdd710 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d54e1e740 10 client.16206 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d54e1e740 10 client.16206 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d54e1e740 20 client.16206 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x56169705b3d0 0x56169705d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d54e1e740 4 client.16206 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:01:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561696f42d70 con 0x56169705b3d0 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.493+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x56169705b3d0 0x56169705d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d2b7fe640 10 client.16206 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d4bfff640 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x56169705b3d0 0x56169705d7c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fdeb677f5c05fff3 server_cookie=50e16b543b165bd7 in_seq=0 out_seq=0 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d2b7fe640 1 -- 172.21.15.28:0/3125025918 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561696f42d70 con 0x56169705b3d0 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d2b7fe640 10 client.16206 handle_command_reply: tid=0 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 client.16206 shutdown 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fdd710 msgr2=0x561696ff3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fdd710 0x561696ff3dc0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2d40003e60 tx=0x7f2d40040040 comp rx=0 tx=0).stop 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 shutdown_connections 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x56169705b3d0 0x56169705d7c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561696fdd710 0x561696ff3dc0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d2b7fe640 0 client.16206 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:01:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561696fd6b40 0x561696ff3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 --2- 172.21.15.28:0/3125025918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561696fd55d0 0x561696ff3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 >> 172.21.15.28:0/3125025918 conn(0x561696f9e310 msgr2=0x561696fde8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 shutdown_connections 2024-09-15T22:01:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 1 -- 172.21.15.28:0/3125025918 wait complete. 2024-09-15T22:01:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.494+0000 7f2d54e1e740 20 client.16206 trim_cache size 0 max 16384 2024-09-15T22:01:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2d340a33b0 msgr2=0x7f2d340a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2d340a33b0 0x7f2d340a5870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2d440a4d10 tx=0x7f2d4400af30 comp rx=0 tx=0).stop 2024-09-15T22:01:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c2f0 msgr2=0x7f2d4c1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c2f0 0x7f2d4c1014d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400cce0 tx=0x7f2d4400a620 comp rx=0 tx=0).stop 2024-09-15T22:01:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 shutdown_connections 2024-09-15T22:01:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2d340a33b0 0x7f2d340a5870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d4c10c2f0 0x7f2d4c1014d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d4c1037a0 0x7f2d4c100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 --2- 172.21.15.28:0/158549519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d4c102df0 0x7f2d4c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 >> 172.21.15.28:0/158549519 conn(0x7f2d4c0fc7d0 msgr2=0x7f2d4c0fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 shutdown_connections 2024-09-15T22:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:13.495+0000 7f2d52a45640 1 -- 172.21.15.28:0/158549519 wait complete. 2024-09-15T22:01:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:15 smithi028 ceph-mon[30969]: pgmap v120: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 44 op/s 2024-09-15T22:01:15.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:15 smithi203 ceph-mon[35546]: pgmap v120: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 44 op/s 2024-09-15T22:01:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:15 smithi174 ceph-mon[35244]: pgmap v120: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 44 op/s 2024-09-15T22:01:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:01:17.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:17 smithi028 ceph-mon[30969]: pgmap v121: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 44 op/s 2024-09-15T22:01:17.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:17 smithi203 ceph-mon[35546]: pgmap v121: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 44 op/s 2024-09-15T22:01:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:17 smithi174 ceph-mon[35244]: pgmap v121: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 44 op/s 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 -- 172.21.15.28:0/1767018204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec0101350 msgr2=0x7f2ec010d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1767018204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec0101350 0x7f2ec010d880 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009930 tx=0x7f2eb802a420 comp rx=0 tx=0).stop 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 -- 172.21.15.28:0/1767018204 shutdown_connections 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1767018204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec0101350 0x7f2ec010d880 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1767018204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec01009b0 0x7f2ec0100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1767018204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ec01069c0 0x7f2ec0106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 -- 172.21.15.28:0/1767018204 >> 172.21.15.28:0/1767018204 conn(0x7f2ec0077990 msgr2=0x7f2ec00fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.882+0000 7f2ec7627640 1 -- 172.21.15.28:0/1767018204 shutdown_connections 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 -- 172.21.15.28:0/1767018204 wait complete. 2024-09-15T22:01:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 Processor -- start 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 -- start start 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ec01009b0 0x7f2ec01982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec0101350 0x7f2ec0198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec01069c0 0x7f2ec019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0110e00 con 0x7f2ec01069c0 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0110c80 con 0x7f2ec01009b0 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.883+0000 7f2ec7627640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec0110f80 con 0x7f2ec0101350 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec01069c0 0x7f2ec019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec01069c0 0x7f2ec019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54780/0 (socket says 172.21.15.28:54780) 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 -- 172.21.15.28:0/1818270773 learned_addr learned my addr 172.21.15.28:0/1818270773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ec01009b0 0x7f2ec01982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec0101350 0x7f2ec0198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 -- 172.21.15.28:0/1818270773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec0101350 msgr2=0x7f2ec0198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec0101350 0x7f2ec0198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 -- 172.21.15.28:0/1818270773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ec01009b0 msgr2=0x7f2ec01982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ec01009b0 0x7f2ec01982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 -- 172.21.15.28:0/1818270773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec019d2b0 con 0x7f2ec01069c0 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec5bc4640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec01069c0 0x7f2ec019cb80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8005ab0 tx=0x7f2eb80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ebe7fc640 1 -- 172.21.15.28:0/1818270773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb803f030 con 0x7f2ec01069c0 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ebe7fc640 1 -- 172.21.15.28:0/1818270773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb8003ce0 con 0x7f2ec01069c0 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec019d540 con 0x7f2ec01069c0 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.884+0000 7f2ebe7fc640 1 -- 172.21.15.28:0/1818270773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8035070 con 0x7f2ec01069c0 2024-09-15T22:01:18.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.885+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec01a4e30 con 0x7f2ec01069c0 2024-09-15T22:01:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2ebe7fc640 1 -- 172.21.15.28:0/1818270773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f2eb8042030 con 0x7f2ec01069c0 2024-09-15T22:01:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 Processor -- start 2024-09-15T22:01:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 -- start start 2024-09-15T22:01:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 0x557168285db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168294b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682950c0 0x5571682974b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5571681ea2f0 con 0x5571682950c0 2024-09-15T22:01:18.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557168192270 con 0x5571682862f0 2024-09-15T22:01:18.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5571681e62b0 con 0x5571682859d0 2024-09-15T22:01:18.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2ec4bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682950c0 0x5571682974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2ec4bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682950c0 0x5571682974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54782/0 (socket says 172.21.15.28:54782) 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.888+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/1766362775 learned_addr learned my addr 172.21.15.28:0/1766362775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168294b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 0x557168285db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ebe7fc640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ea80a3460 0x7f2ea80a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec5bc4640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ea80a3460 0x7f2ea80a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/1766362775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 msgr2=0x557168285db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 0x557168285db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/1766362775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 msgr2=0x557168294b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168294b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/1766362775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557168191340 con 0x5571682950c0 2024-09-15T22:01:18.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ebe7fc640 1 -- 172.21.15.28:0/1818270773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2eb80f0550 con 0x7f2ec01069c0 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682950c0 0x5571682974b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4011760 tx=0x7f2eb400a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72e7f7f9eb1d208 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ea77fe640 1 -- 172.21.15.28:0/1766362775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4004020 con 0x5571682950c0 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ea77fe640 1 -- 172.21.15.28:0/1766362775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb4005500 con 0x5571682950c0 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.889+0000 7f2ea77fe640 1 -- 172.21.15.28:0/1766362775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4004c60 con 0x5571682950c0 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1766362775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682950c0 msgr2=0x5571682974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2ec5bc4640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ea80a3460 0x7f2ea80a5920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80ed9f0 tx=0x7f2eb8087410 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682950c0 0x5571682974b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4011760 tx=0x7f2eb400a6a0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1766362775 shutdown_connections 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682950c0 0x5571682974b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168294b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1766362775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 0x557168285db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1766362775 >> 172.21.15.28:0/1766362775 conn(0x557168282060 msgr2=0x557168282440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1766362775 shutdown_connections 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1766362775 wait complete. 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.890+0000 7f2eca3c3740 1 -- start start 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 1 Processor -- start 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682859d0 0x557168292030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168292570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 0x557168292ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557168297cf0 con 0x5571682859d0 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557168173290 con 0x5571682862f0 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557168297e70 con 0x5571682950c0 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2eca3c3740 10 client.0 _ll_register_callbacks cb 0x5571681a4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2ec4bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682859d0 0x557168292030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2ec4bc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682859d0 0x557168292030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54790/0 (socket says 172.21.15.28:54790) 2024-09-15T22:01:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/919017198 learned_addr learned my addr 172.21.15.28:0/919017198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.891+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168292570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/919017198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 msgr2=0x557168292ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 0x557168292ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 msgr2=0x557168292570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168292570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ec4bc2640 1 -- 172.21.15.28:0/919017198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557167efd210 con 0x5571682859d0 2024-09-15T22:01:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168292570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:01:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682859d0 0x557168292030 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4004bf0 tx=0x7f2eb4004c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7899b98c06640a6b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ea4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/919017198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb400f990 con 0x5571682859d0 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/919017198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2eb400fb30 con 0x5571682859d0 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2eb4005500 con 0x5571682859d0 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/919017198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2eb4030050 con 0x5571682859d0 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2eca3c3740 10 client.16218 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557167fc9620 con 0x5571682859d0 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/919017198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb4005500 con 0x5571682859d0 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.892+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/919017198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f2eb4025020 con 0x5571682859d0 2024-09-15T22:01:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2eca3c3740 10 client.16218 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2eca3c3740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2eca3c3740 20 client.16218 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571682a4fa0 0x5571682a7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2eca3c3740 4 client.16218 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571681f7a20 con 0x5571682a4fa0 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571682a4fa0 0x5571682a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571682a4fa0 0x5571682a7390 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77818861543d1383 server_cookie=12808c65a100da78 in_seq=0 out_seq=0 2024-09-15T22:01:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.893+0000 7f2ea4ff9640 10 client.16218 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/919017198 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5571681f7a20 con 0x5571682a4fa0 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2ea4ff9640 10 client.16218 handle_command_reply: tid=0 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 client.16218 shutdown 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682859d0 msgr2=0x557168292030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682859d0 0x557168292030 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4004bf0 tx=0x7f2eb4004c20 comp rx=0 tx=0).stop 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 shutdown_connections 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571682a4fa0 0x5571682a7390 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 0x557168292ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682862f0 0x557168292570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2ea4ff9640 0 client.16218 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/919017198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682859d0 0x557168292030 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 >> 172.21.15.28:0/919017198 conn(0x557168282060 msgr2=0x557168287dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 shutdown_connections 2024-09-15T22:01:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 1 -- 172.21.15.28:0/919017198 wait complete. 2024-09-15T22:01:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.894+0000 7f2eca3c3740 20 client.16218 trim_cache size 0 max 16384 2024-09-15T22:01:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 Processor -- start 2024-09-15T22:01:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 -- start start 2024-09-15T22:01:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 0x557168287cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x5571682acad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682950c0 0x5571682ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec01acb70 con 0x5571682862f0 2024-09-15T22:01:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec01ae6b0 con 0x5571682950c0 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.909+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec01ad240 con 0x5571682859d0 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x5571682acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x5571682acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54796/0 (socket says 172.21.15.28:54796) 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 -- 172.21.15.28:0/948142867 learned_addr learned my addr 172.21.15.28:0/948142867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682950c0 0x5571682ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 -- 172.21.15.28:0/948142867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 msgr2=0x557168287cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 0x557168287cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 -- 172.21.15.28:0/948142867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682950c0 msgr2=0x5571682ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682950c0 0x5571682ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 -- 172.21.15.28:0/948142867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557168297850 con 0x5571682862f0 2024-09-15T22:01:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682950c0 0x5571682ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:01:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x5571682acad0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x5571682886c0 tx=0x7f2eb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7da7244cda1d542 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ea6ffd640 1 -- 172.21.15.28:0/948142867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0023030 con 0x5571682862f0 2024-09-15T22:01:18.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ea6ffd640 1 -- 172.21.15.28:0/948142867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb0003e50 con 0x5571682862f0 2024-09-15T22:01:18.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.910+0000 7f2ea6ffd640 1 -- 172.21.15.28:0/948142867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb001f440 con 0x5571682862f0 2024-09-15T22:01:18.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 -- 172.21.15.28:0/948142867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 msgr2=0x5571682acad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x5571682acad0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x5571682886c0 tx=0x7f2eb0004a40 comp rx=0 tx=0).stop 2024-09-15T22:01:18.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 -- 172.21.15.28:0/948142867 shutdown_connections 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682950c0 0x5571682ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x5571682acad0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/948142867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682859d0 0x557168287cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 -- 172.21.15.28:0/948142867 >> 172.21.15.28:0/948142867 conn(0x557168282060 msgr2=0x5571682875a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 -- 172.21.15.28:0/948142867 shutdown_connections 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 -- 172.21.15.28:0/948142867 wait complete. 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 -- start start 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.911+0000 7f2eca3c3740 1 Processor -- start 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682859d0 0x557168299040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x557168299580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2eca3c3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 0x557168299ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec01adfe0 con 0x5571682862f0 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec01ad910 con 0x5571682859d0 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2eca3c3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec01aab50 con 0x5571682950c0 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2eca3c3740 10 client.0 _ll_register_callbacks cb 0x7f2eb000a250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x557168299580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec4bc2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 0x557168299ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x557168299580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54802/0 (socket says 172.21.15.28:54802) 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 -- 172.21.15.28:0/1516144166 learned_addr learned my addr 172.21.15.28:0/1516144166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682859d0 0x557168299040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 -- 172.21.15.28:0/1516144166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 msgr2=0x557168299ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 0x557168299ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 -- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682859d0 msgr2=0x557168299040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682859d0 0x557168299040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 -- 172.21.15.28:0/1516144166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55716830eca0 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ec53c3640 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x557168299580 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000a400 tx=0x7f2eb0004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07c81511a2d8cd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.912+0000 7f2ea4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/1516144166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0035060 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/1516144166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f2eb0003970 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2eb0003e50 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/1516144166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2eb003c050 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 10 client.16224 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55716830fec0 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/1516144166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eb0003e50 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/1516144166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f2eb002f030 con 0x5571682862f0 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 10 client.16224 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 10 client.16224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:18.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 20 client.16224 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571683105e0 0x557168312a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 4 client.16224 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:01:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.913+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5571681f1d70 con 0x5571683105e0 2024-09-15T22:01:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.914+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571683105e0 0x557168312a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.914+0000 7f2ec4bc2640 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571683105e0 0x557168312a00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d118fdbcbb1434f5 server_cookie=383080860fca75ab in_seq=0 out_seq=0 2024-09-15T22:01:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.914+0000 7f2ea4ff9640 10 client.16224 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2ea4ff9640 1 -- 172.21.15.28:0/1516144166 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5571681f1d70 con 0x5571683105e0 2024-09-15T22:01:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2ea4ff9640 10 client.16224 handle_command_reply: tid=0 2024-09-15T22:01:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 client.16224 shutdown 2024-09-15T22:01:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 msgr2=0x557168299580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x557168299580 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000a400 tx=0x7f2eb0004330 comp rx=0 tx=0).stop 2024-09-15T22:01:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 shutdown_connections 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5571683105e0 0x557168312a00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571682950c0 0x557168299ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571682862f0 0x557168299580 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 --2- 172.21.15.28:0/1516144166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571682859d0 0x557168299040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2ea4ff9640 0 client.16224 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.927+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 >> 172.21.15.28:0/1516144166 conn(0x557168282060 msgr2=0x557168287da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 shutdown_connections 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2eca3c3740 1 -- 172.21.15.28:0/1516144166 wait complete. 2024-09-15T22:01:18.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2eca3c3740 20 client.16224 trim_cache size 0 max 16384 2024-09-15T22:01:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:01:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:01:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:01:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:01:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ea80a3460 msgr2=0x7f2ea80a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ea80a3460 0x7f2ea80a5920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80ed9f0 tx=0x7f2eb8087410 comp rx=0 tx=0).stop 2024-09-15T22:01:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec01069c0 msgr2=0x7f2ec019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec01069c0 0x7f2ec019cb80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8005ab0 tx=0x7f2eb80042f0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 shutdown_connections 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2ea80a3460 0x7f2ea80a5920 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec01069c0 0x7f2ec019cb80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ec0101350 0x7f2ec0198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 --2- 172.21.15.28:0/1818270773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ec01009b0 0x7f2ec01982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 >> 172.21.15.28:0/1818270773 conn(0x7f2ec0077990 msgr2=0x7f2ec00ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 shutdown_connections 2024-09-15T22:01:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:18.928+0000 7f2ec7627640 1 -- 172.21.15.28:0/1818270773 wait complete. 2024-09-15T22:01:19.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:19 smithi028 ceph-mon[30969]: pgmap v122: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 35 op/s 2024-09-15T22:01:19.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:19 smithi203 ceph-mon[35546]: pgmap v122: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 35 op/s 2024-09-15T22:01:19.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:19 smithi174 ceph-mon[35244]: pgmap v122: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 35 op/s 2024-09-15T22:01:20.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:01:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 -- 172.21.15.28:0/609140331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 msgr2=0x7febec1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 --2- 172.21.15.28:0/609140331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1006d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7febdc009930 tx=0x7febdc02a420 comp rx=0 tx=0).stop 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 -- 172.21.15.28:0/609140331 shutdown_connections 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 --2- 172.21.15.28:0/609140331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febec106f50 0x7febec10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 --2- 172.21.15.28:0/609140331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec100ca0 0x7febec106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 --2- 172.21.15.28:0/609140331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1006d0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 -- 172.21.15.28:0/609140331 >> 172.21.15.28:0/609140331 conn(0x7febec077210 msgr2=0x7febec0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 -- 172.21.15.28:0/609140331 shutdown_connections 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.798+0000 7febf0872640 1 -- 172.21.15.28:0/609140331 wait complete. 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.799+0000 7febf0872640 1 Processor -- start 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.799+0000 7febf0872640 1 -- start start 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.799+0000 7febf0872640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.799+0000 7febf0872640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febec100ca0 0x7febec198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febf0872640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec106f50 0x7febec19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febf0872640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febec111fa0 con 0x7febec1002f0 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febf0872640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7febec111e20 con 0x7febec100ca0 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febf0872640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7febec112120 con 0x7febec106f50 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60026/0 (socket says 172.21.15.28:60026) 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 -- 172.21.15.28:0/621090021 learned_addr learned my addr 172.21.15.28:0/621090021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febe9d9b640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febec100ca0 0x7febec198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febead9d640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec106f50 0x7febec19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 -- 172.21.15.28:0/621090021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec106f50 msgr2=0x7febec19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec106f50 0x7febec19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 -- 172.21.15.28:0/621090021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febec100ca0 msgr2=0x7febec198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febec100ca0 0x7febec198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 -- 172.21.15.28:0/621090021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febec19d470 con 0x7febec1002f0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febea59c640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1984c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7febdc009230 tx=0x7febdc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febdb7fe640 1 -- 172.21.15.28:0/621090021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc03f030 con 0x7febec1002f0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febec19d700 con 0x7febec1002f0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febdb7fe640 1 -- 172.21.15.28:0/621090021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febdc003ad0 con 0x7febec1002f0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.800+0000 7febdb7fe640 1 -- 172.21.15.28:0/621090021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc035070 con 0x7febec1002f0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.801+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febec1a5040 con 0x7febec1002f0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.803+0000 7febdb7fe640 1 -- 172.21.15.28:0/621090021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7febdc042030 con 0x7febec1002f0 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 Processor -- start 2024-09-15T22:01:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 -- start start 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77eb6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77eb9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b77e11620 con 0x564b77eb7240 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b77db8e50 con 0x564b77eac980 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febf2c4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564b77e0d5e0 con 0x564b77eac060 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febe9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77eb9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febe9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77eb9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60032/0 (socket says 172.21.15.28:60032) 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febe9d9b640 1 -- 172.21.15.28:0/868585390 learned_addr learned my addr 172.21.15.28:0/868585390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febead9d640 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.804+0000 7febea59c640 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77eb6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febdb7fe640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7febd40a33b0 0x7febd40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febe9d9b640 1 -- 172.21.15.28:0/868585390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 msgr2=0x564b77eac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febe9d9b640 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febe9d9b640 1 -- 172.21.15.28:0/868585390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 msgr2=0x564b77eb6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febe9d9b640 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77eb6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febe9d9b640 1 -- 172.21.15.28:0/868585390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b77da9d40 con 0x564b77eb7240 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febead9d640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7febd40a33b0 0x7febd40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febdb7fe640 1 -- 172.21.15.28:0/621090021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7febdc0f0420 con 0x7febec1002f0 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febe9d9b640 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77eb9630 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7febe000ac70 tx=0x7febe000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bba4d12194b7f6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febd8ff9640 1 -- 172.21.15.28:0/868585390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe001d070 con 0x564b77eb7240 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febead9d640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7febd40a33b0 0x7febd40a5870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7febe4005fd0 tx=0x7febe4004a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febd8ff9640 1 -- 172.21.15.28:0/868585390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe0018b20 con 0x564b77eb7240 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.805+0000 7febd8ff9640 1 -- 172.21.15.28:0/868585390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0010bf0 con 0x564b77eb7240 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 -- 172.21.15.28:0/868585390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 msgr2=0x564b77eb9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77eb9630 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7febe000ac70 tx=0x7febe000f790 comp rx=0 tx=0).stop 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 -- 172.21.15.28:0/868585390 shutdown_connections 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77eb9630 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77eb6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 --2- 172.21.15.28:0/868585390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 -- 172.21.15.28:0/868585390 >> 172.21.15.28:0/868585390 conn(0x564b77ea85f0 msgr2=0x564b77ea89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 -- 172.21.15.28:0/868585390 shutdown_connections 2024-09-15T22:01:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 -- 172.21.15.28:0/868585390 wait complete. 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.806+0000 7febf2c4b740 1 -- start start 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 1 Processor -- start 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b77eb9e10 con 0x564b77eac980 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b77d99f10 con 0x564b77eb7240 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564b77eb9f90 con 0x564b77eac060 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febf2c4b740 10 client.0 _ll_register_callbacks cb 0x564b77dbddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60034/0 (socket says 172.21.15.28:60034) 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 -- 172.21.15.28:0/1523088325 learned_addr learned my addr 172.21.15.28:0/1523088325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febea59c640 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febead9d640 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 -- 172.21.15.28:0/1523088325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 msgr2=0x564b77eb4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 -- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 msgr2=0x564b77eb4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.807+0000 7febe9d9b640 1 -- 172.21.15.28:0/1523088325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b77b20580 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febe9d9b640 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb4640 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7febe0010770 tx=0x7febe0010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb7c5e83766425a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febd1ffb640 1 -- 172.21.15.28:0/1523088325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe001d050 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febd1ffb640 1 -- 172.21.15.28:0/1523088325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7febe0010d60 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febe0018b20 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febd1ffb640 1 -- 172.21.15.28:0/1523088325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7febe0032050 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 10 client.16236 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564b77bea960 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febd1ffb640 1 -- 172.21.15.28:0/1523088325 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe0018b20 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febd1ffb640 1 -- 172.21.15.28:0/1523088325 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7febe00046c0 con 0x564b77eac980 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 10 client.16236 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 10 client.16236 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 20 client.16236 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77ec6d20 0x564b77ec9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 4 client.16236 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.808+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b77e1eb90 con 0x564b77ec6d20 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.809+0000 7febea59c640 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77ec6d20 0x564b77ec9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.809+0000 7febd1ffb640 10 client.16236 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.809+0000 7febea59c640 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77ec6d20 0x564b77ec9110 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e390e090eae62e8 server_cookie=226da80b90b11c70 in_seq=0 out_seq=0 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febd1ffb640 1 -- 172.21.15.28:0/1523088325 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b77e1eb90 con 0x564b77ec6d20 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febd1ffb640 10 client.16236 handle_command_reply: tid=0 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 client.16236 shutdown 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 msgr2=0x564b77eb4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb4640 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7febe0010770 tx=0x7febe0010430 comp rx=0 tx=0).stop 2024-09-15T22:01:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 shutdown_connections 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77ec6d20 0x564b77ec9110 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb4640 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1523088325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 >> 172.21.15.28:0/1523088325 conn(0x564b77ea85f0 msgr2=0x564b77eae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febd1ffb640 0 client.16236 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 shutdown_connections 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 1 -- 172.21.15.28:0/1523088325 wait complete. 2024-09-15T22:01:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.810+0000 7febf2c4b740 20 client.16236 trim_cache size 0 max 16384 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.827+0000 7febf2c4b740 1 Processor -- start 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.827+0000 7febf2c4b740 1 -- start start 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77ece770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77ececb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febf2c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febec1aefb0 con 0x564b77eb7240 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febf2c4b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7febec1b0af0 con 0x564b77eac980 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febf2c4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7febec1af680 con 0x564b77eac060 2024-09-15T22:01:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febead9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77ececb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febead9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77ececb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60036/0 (socket says 172.21.15.28:60036) 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febead9d640 1 -- 172.21.15.28:0/68521567 learned_addr learned my addr 172.21.15.28:0/68521567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febe9d9b640 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77ece770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febea59c640 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febe9d9b640 1 -- 172.21.15.28:0/68521567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 msgr2=0x564b77eb5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febe9d9b640 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febe9d9b640 1 -- 172.21.15.28:0/68521567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 msgr2=0x564b77ececb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febe9d9b640 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77ececb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febe9d9b640 1 -- 172.21.15.28:0/68521567 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b77eb9f40 con 0x564b77eac980 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.828+0000 7febead9d640 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77ececb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.829+0000 7febe9d9b640 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77ece770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7febe0004b10 tx=0x7febe000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eb257e9c28a9931 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.829+0000 7febd3fff640 1 -- 172.21.15.28:0/68521567 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe001d040 con 0x564b77eac980 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.829+0000 7febd3fff640 1 -- 172.21.15.28:0/68521567 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe0029070 con 0x564b77eac980 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.829+0000 7febd3fff640 1 -- 172.21.15.28:0/68521567 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0010b60 con 0x564b77eac980 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 -- 172.21.15.28:0/68521567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 msgr2=0x564b77ece770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77ece770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7febe0004b10 tx=0x7febe000f790 comp rx=0 tx=0).stop 2024-09-15T22:01:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 -- 172.21.15.28:0/68521567 shutdown_connections 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eb7240 0x564b77ececb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eac980 0x564b77ece770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 --2- 172.21.15.28:0/68521567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 -- 172.21.15.28:0/68521567 >> 172.21.15.28:0/68521567 conn(0x564b77ea85f0 msgr2=0x564b77eae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 -- 172.21.15.28:0/68521567 shutdown_connections 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 -- 172.21.15.28:0/68521567 wait complete. 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 -- start start 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.830+0000 7febf2c4b740 1 Processor -- start 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febf2c4b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febf2c4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febec1b0420 con 0x564b77eac980 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febf2c4b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7febec1afd50 con 0x564b77eb7240 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febf2c4b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7febec1acf90 con 0x564b77eac060 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60048/0 (socket says 172.21.15.28:60048) 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 -- 172.21.15.28:0/1953986045 learned_addr learned my addr 172.21.15.28:0/1953986045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febea59c640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febead9d640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febf2c4b740 10 client.0 _ll_register_callbacks cb 0x7febe4009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 -- 172.21.15.28:0/1953986045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 msgr2=0x564b77eb1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 -- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 msgr2=0x564b77eb1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:20.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 -- 172.21.15.28:0/1953986045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b77f30740 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febead9d640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febe9d9b640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb16a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7febe0005940 tx=0x7febe000a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9eae395305eb9bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.831+0000 7febd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febd1ffb640 1 -- 172.21.15.28:0/1953986045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe001d040 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febd1ffb640 1 -- 172.21.15.28:0/1953986045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7febe0010d60 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febe0029070 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febd1ffb640 1 -- 172.21.15.28:0/1953986045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7febe004d050 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 10 client.16242 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564b77f35e10 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febd1ffb640 1 -- 172.21.15.28:0/1953986045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe0029070 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febd1ffb640 1 -- 172.21.15.28:0/1953986045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7febe0024690 con 0x564b77eac980 2024-09-15T22:01:20.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 10 client.16242 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 10 client.16242 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 20 client.16242 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77f364c0 0x564b77f388e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 4 client.16242 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "1ebb5d00", "release": true, "await": true}] 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.832+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "set_id": "1ebb5d00", "release": true, "await": true}) -- 0x564b77ea3950 con 0x564b77f364c0 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.833+0000 7febea59c640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77f364c0 0x564b77f388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.833+0000 7febea59c640 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77f364c0 0x564b77f388e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7126ae6c00716b87 server_cookie=6cefde5e3c21d79 in_seq=0 out_seq=0 2024-09-15T22:01:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.833+0000 7febd1ffb640 10 client.16242 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:01:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.862+0000 7febd1ffb640 1 -- 172.21.15.28:0/1953986045 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x564b77ea3950 con 0x564b77f364c0 2024-09-15T22:01:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.862+0000 7febd1ffb640 10 client.16242 handle_command_reply: tid=0 2024-09-15T22:01:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.862+0000 7febf2c4b740 1 client.16242 shutdown 2024-09-15T22:01:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.862+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 msgr2=0x564b77eb16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.862+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb16a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7febe0005940 tx=0x7febe000a540 comp rx=0 tx=0).stop 2024-09-15T22:01:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 shutdown_connections 2024-09-15T22:01:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x564b77f364c0 0x564b77f388e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b77eb7240 0x564b77eb1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b77eac980 0x564b77eb16a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febd1ffb640 0 client.16242 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 --2- 172.21.15.28:0/1953986045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b77eac060 0x564b77eb1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 >> 172.21.15.28:0/1953986045 conn(0x564b77ea85f0 msgr2=0x564b77ea6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 shutdown_connections 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 1 -- 172.21.15.28:0/1953986045 wait complete. 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.864+0000 7febf2c4b740 20 client.16242 trim_cache size 0 max 16384 2024-09-15T22:01:20.866 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 32, 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 8, 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "1ebb5d00": { 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 8, 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:01:20.867 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 90.0, 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 139.6, 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:01:20.868 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:01:20.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.866+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7febd40a33b0 msgr2=0x7febd40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.866+0000 7febf0872640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7febd40a33b0 0x7febd40a5870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7febe4005fd0 tx=0x7febe4004a40 comp rx=0 tx=0).stop 2024-09-15T22:01:20.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.866+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 msgr2=0x7febec1984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:20.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.866+0000 7febf0872640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1984c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7febdc009230 tx=0x7febdc004270 comp rx=0 tx=0).stop 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 shutdown_connections 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7febd40a33b0 0x7febd40a5870 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec106f50 0x7febec19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febec100ca0 0x7febec198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 --2- 172.21.15.28:0/621090021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febec1002f0 0x7febec1984c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 >> 172.21.15.28:0/621090021 conn(0x7febec077210 msgr2=0x7febec0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 shutdown_connections 2024-09-15T22:01:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:20.868+0000 7febf0872640 1 -- 172.21.15.28:0/621090021 wait complete. 2024-09-15T22:01:21.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:21 smithi028 ceph-mon[30969]: pgmap v123: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-09-15T22:01:21.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:21 smithi203 ceph-mon[35546]: pgmap v123: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-09-15T22:01:21.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:21 smithi174 ceph-mon[35244]: pgmap v123: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 11 op/s 2024-09-15T22:01:23.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:23 smithi028 ceph-mon[30969]: pgmap v124: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 934 KiB/s wr, 174 op/s 2024-09-15T22:01:23.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:23 smithi203 ceph-mon[35546]: pgmap v124: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 934 KiB/s wr, 174 op/s 2024-09-15T22:01:23.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:23 smithi174 ceph-mon[35244]: pgmap v124: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 934 KiB/s wr, 174 op/s 2024-09-15T22:01:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:25 smithi028 ceph-mon[30969]: pgmap v125: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.7 MiB/s wr, 302 op/s 2024-09-15T22:01:25.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:25 smithi203 ceph-mon[35546]: pgmap v125: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.7 MiB/s wr, 302 op/s 2024-09-15T22:01:25.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:25 smithi174 ceph-mon[35244]: pgmap v125: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.7 MiB/s wr, 302 op/s 2024-09-15T22:01:27.088 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '1ebb5d00', total seconds elapsed: 108.2 2024-09-15T22:01:27.088 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-15T22:01:27.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:01:27.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:27 smithi028 ceph-mon[30969]: pgmap v126: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 MiB/s wr, 640 op/s 2024-09-15T22:01:27.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:27 smithi203 ceph-mon[35546]: pgmap v126: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 MiB/s wr, 640 op/s 2024-09-15T22:01:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:27 smithi174 ceph-mon[35244]: pgmap v126: 129 pgs: 129 active+clean; 14 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 MiB/s wr, 640 op/s 2024-09-15T22:01:29.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:29 smithi028 ceph-mon[30969]: pgmap v127: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 27 MiB/s wr, 4.27k op/s 2024-09-15T22:01:29.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:29 smithi203 ceph-mon[35546]: pgmap v127: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 27 MiB/s wr, 4.27k op/s 2024-09-15T22:01:29.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:29 smithi174 ceph-mon[35244]: pgmap v127: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 27 MiB/s wr, 4.27k op/s 2024-09-15T22:01:31.166 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:01:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 -- 172.21.15.28:0/2623694566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 msgr2=0x7fab6c1021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 --2- 172.21.15.28:0/2623694566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c1021f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009ca0 tx=0x7fab5c02a8c0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 -- 172.21.15.28:0/2623694566 shutdown_connections 2024-09-15T22:01:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 --2- 172.21.15.28:0/2623694566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab6c106f40 0x7fab6c10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 --2- 172.21.15.28:0/2623694566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab6c1027c0 0x7fab6c106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 --2- 172.21.15.28:0/2623694566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c1021f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 -- 172.21.15.28:0/2623694566 >> 172.21.15.28:0/2623694566 conn(0x7fab6c0fd8d0 msgr2=0x7fab6c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 -- 172.21.15.28:0/2623694566 shutdown_connections 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 -- 172.21.15.28:0/2623694566 wait complete. 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.824+0000 7fab747da640 1 Processor -- start 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab747da640 1 -- start start 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab747da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab747da640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab6c1027c0 0x7fab6c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab747da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab6c106f40 0x7fab6c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab747da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c111fa0 con 0x7fab6c101e10 2024-09-15T22:01:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab747da640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c111e20 con 0x7fab6c106f40 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60064/0 (socket says 172.21.15.28:60064) 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab747da640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c112120 con 0x7fab6c1027c0 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 -- 172.21.15.28:0/2947587159 learned_addr learned my addr 172.21.15.28:0/2947587159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72d77640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab6c106f40 0x7fab6c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab71d75640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab6c1027c0 0x7fab6c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 -- 172.21.15.28:0/2947587159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab6c1027c0 msgr2=0x7fab6c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab6c1027c0 0x7fab6c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 -- 172.21.15.28:0/2947587159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab6c106f40 msgr2=0x7fab6c19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab6c106f40 0x7fab6c19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72576640 1 -- 172.21.15.28:0/2947587159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab6c19d410 con 0x7fab6c101e10 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab72d77640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab6c106f40 0x7fab6c19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:01:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.825+0000 7fab71d75640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab6c1027c0 0x7fab6c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.826+0000 7fab72576640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c198460 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fab5c02ade0 tx=0x7fab5c0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.826+0000 7fab6b7fe640 1 -- 172.21.15.28:0/2947587159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c02fa20 con 0x7fab6c101e10 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.826+0000 7fab6b7fe640 1 -- 172.21.15.28:0/2947587159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab5c02fbc0 con 0x7fab6c101e10 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.826+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab6c19d6a0 con 0x7fab6c101e10 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.826+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab6c1a4fe0 con 0x7fab6c101e10 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.826+0000 7fab6b7fe640 1 -- 172.21.15.28:0/2947587159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0466e0 con 0x7fab6c101e10 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.828+0000 7fab6b7fe640 1 -- 172.21.15.28:0/2947587159 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fab5c0305c0 con 0x7fab6c101e10 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 Processor -- start 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 -- start start 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94c7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94d8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624e942d2f0 con 0x5624e94d6780 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5624e93d4cf0 con 0x5624e94c75d0 2024-09-15T22:01:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab76bb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5624e94292b0 con 0x5624e94c8b40 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab6b7fe640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab540a33b0 0x7fab540a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab71d75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94d8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab71d75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94d8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60078/0 (socket says 172.21.15.28:60078) 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab71d75640 1 -- 172.21.15.28:0/4263894870 learned_addr learned my addr 172.21.15.28:0/4263894870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab72576640 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.829+0000 7fab72d77640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab540a33b0 0x7fab540a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab72d77640 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab6b7fe640 1 -- 172.21.15.28:0/2947587159 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fab5c0f13c0 con 0x7fab6c101e10 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab71d75640 1 -- 172.21.15.28:0/4263894870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 msgr2=0x5624e94c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab71d75640 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94c7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab71d75640 1 -- 172.21.15.28:0/4263894870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 msgr2=0x5624e94c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab71d75640 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab71d75640 1 -- 172.21.15.28:0/4263894870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624e93b93a0 con 0x5624e94d6780 2024-09-15T22:01:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab71d75640 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94d8b70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fab60009c00 tx=0x7fab60010370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965ac098e6cc5cff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab72d77640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab540a33b0 0x7fab540a5870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fab640128a0 tx=0x7fab64009860 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab68ff9640 1 -- 172.21.15.28:0/4263894870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60010da0 con 0x5624e94d6780 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab68ff9640 1 -- 172.21.15.28:0/4263894870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab600035f0 con 0x5624e94d6780 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.830+0000 7fab68ff9640 1 -- 172.21.15.28:0/4263894870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab600126b0 con 0x5624e94d6780 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 -- 172.21.15.28:0/4263894870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 msgr2=0x5624e94d8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94d8b70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fab60009c00 tx=0x7fab60010370 comp rx=0 tx=0).stop 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 -- 172.21.15.28:0/4263894870 shutdown_connections 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94d8b70 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 --2- 172.21.15.28:0/4263894870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 -- 172.21.15.28:0/4263894870 >> 172.21.15.28:0/4263894870 conn(0x5624e9490310 msgr2=0x5624e94c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 -- 172.21.15.28:0/4263894870 shutdown_connections 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 -- 172.21.15.28:0/4263894870 wait complete. 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 -- start start 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.831+0000 7fab76bb3740 1 Processor -- start 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94c75d0 0x5624e94d6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94d3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94d6780 0x5624e94d38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab76bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624e94d9480 con 0x5624e94c75d0 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab76bb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5624e93e8060 con 0x5624e94c8b40 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab76bb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5624e94d9600 con 0x5624e94d6780 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab76bb3740 10 client.0 _ll_register_callbacks cb 0x5624e913b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94c75d0 0x5624e94d6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94c75d0 0x5624e94d6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60080/0 (socket says 172.21.15.28:60080) 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 -- 172.21.15.28:0/1686383154 learned_addr learned my addr 172.21.15.28:0/1686383154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab71d75640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94d3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72d77640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94d6780 0x5624e94d38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 -- 172.21.15.28:0/1686383154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94d6780 msgr2=0x5624e94d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94d6780 0x5624e94d38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 -- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 msgr2=0x5624e94d3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94d3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 -- 172.21.15.28:0/1686383154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624e919e3c0 con 0x5624e94c75d0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab71d75640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94d3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab72576640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94c75d0 0x5624e94d6080 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fab5c0bd150 tx=0x7fab5c08d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa546d2510454e9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.832+0000 7fab51ffb640 1 -- 172.21.15.28:0/1686383154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0bdb50 con 0x5624e94c75d0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab51ffb640 1 -- 172.21.15.28:0/1686383154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fab5c0308b0 con 0x5624e94c75d0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab600035f0 con 0x5624e94c75d0 2024-09-15T22:01:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab51ffb640 1 -- 172.21.15.28:0/1686383154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fab5c0882a0 con 0x5624e94c75d0 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 10 client.16254 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5624e94d9480 con 0x5624e94c75d0 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab51ffb640 1 -- 172.21.15.28:0/1686383154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab600035f0 con 0x5624e94c75d0 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab51ffb640 1 -- 172.21.15.28:0/1686383154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fab5c04e030 con 0x5624e94c75d0 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 10 client.16254 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 10 client.16254 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 20 client.16254 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e94e6340 0x5624e94e87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 4 client.16254 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624e943a860 con 0x5624e94e6340 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.833+0000 7fab71d75640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e94e6340 0x5624e94e87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.834+0000 7fab71d75640 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e94e6340 0x5624e94e87b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=79d4cb951bb21eee server_cookie=40a9dae71c1ec4b3 in_seq=0 out_seq=0 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.834+0000 7fab51ffb640 10 client.16254 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:01:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab51ffb640 1 -- 172.21.15.28:0/1686383154 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5624e943a860 con 0x5624e94e6340 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab51ffb640 10 client.16254 handle_command_reply: tid=0 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 client.16254 shutdown 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94c75d0 msgr2=0x5624e94d6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94c75d0 0x5624e94d6080 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fab5c0bd150 tx=0x7fab5c08d070 comp rx=0 tx=0).stop 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 shutdown_connections 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94d6780 0x5624e94d38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94d3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e94e6340 0x5624e94e87b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1686383154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94c75d0 0x5624e94d6080 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab51ffb640 0 client.16254 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 >> 172.21.15.28:0/1686383154 conn(0x5624e9490310 msgr2=0x5624e94c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 shutdown_connections 2024-09-15T22:01:31.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 1 -- 172.21.15.28:0/1686383154 wait complete. 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.835+0000 7fab76bb3740 20 client.16254 trim_cache size 0 max 16384 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 Processor -- start 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 -- start start 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94e2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94ee060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1acc80 con 0x5624e94d6780 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1ae7c0 con 0x5624e94c75d0 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab76bb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1ad350 con 0x5624e94c8b40 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab72d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab72d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94ee5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60094/0 (socket says 172.21.15.28:60094) 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab72d77640 1 -- 172.21.15.28:0/1379185925 learned_addr learned my addr 172.21.15.28:0/1379185925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab72576640 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94e2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.848+0000 7fab71d75640 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94ee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab72d77640 1 -- 172.21.15.28:0/1379185925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 msgr2=0x5624e94ee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab72d77640 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94ee060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab72d77640 1 -- 172.21.15.28:0/1379185925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 msgr2=0x5624e94e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab72d77640 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94e2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab72d77640 1 -- 172.21.15.28:0/1379185925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624e94d18f0 con 0x5624e94d6780 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab72576640 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab72d77640 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94ee5a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fab640097a0 tx=0x7fab64005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43388167588e6852 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab53fff640 1 -- 172.21.15.28:0/1379185925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6402e070 con 0x5624e94d6780 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab53fff640 1 -- 172.21.15.28:0/1379185925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab64012e20 con 0x5624e94d6780 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.849+0000 7fab53fff640 1 -- 172.21.15.28:0/1379185925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6401f430 con 0x5624e94d6780 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- 172.21.15.28:0/1379185925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 msgr2=0x5624e94ee5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94ee5a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fab640097a0 tx=0x7fab64005f90 comp rx=0 tx=0).stop 2024-09-15T22:01:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- 172.21.15.28:0/1379185925 shutdown_connections 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94ee5a0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c8b40 0x5624e94ee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1379185925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c75d0 0x5624e94e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- 172.21.15.28:0/1379185925 >> 172.21.15.28:0/1379185925 conn(0x5624e9490310 msgr2=0x5624e94ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- 172.21.15.28:0/1379185925 shutdown_connections 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- 172.21.15.28:0/1379185925 wait complete. 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- start start 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 Processor -- start 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c75d0 0x5624e94de890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94dedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94db860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1ae0f0 con 0x5624e94d6780 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1ada20 con 0x5624e94c8b40 2024-09-15T22:01:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab6c1aac60 con 0x5624e94c75d0 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab76bb3740 10 client.0 _ll_register_callbacks cb 0x7fab5c0b90b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab72d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94db860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab72d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94db860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60108/0 (socket says 172.21.15.28:60108) 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab72d77640 1 -- 172.21.15.28:0/1487879753 learned_addr learned my addr 172.21.15.28:0/1487879753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.850+0000 7fab72576640 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c75d0 0x5624e94de890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab71d75640 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94dedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab72d77640 1 -- 172.21.15.28:0/1487879753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c75d0 msgr2=0x5624e94de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab72d77640 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c75d0 0x5624e94de890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab72d77640 1 -- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 msgr2=0x5624e94dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab72d77640 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94dedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab72d77640 1 -- 172.21.15.28:0/1487879753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624e94dbda0 con 0x5624e94d6780 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab72d77640 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94db860 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fab64011c50 tx=0x7fab640099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758679dec93e1842 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab51ffb640 1 -- 172.21.15.28:0/1487879753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6402e030 con 0x5624e94d6780 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab51ffb640 1 -- 172.21.15.28:0/1487879753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fab64026040 con 0x5624e94d6780 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab64012e20 con 0x5624e94d6780 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab51ffb640 1 -- 172.21.15.28:0/1487879753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fab64011e50 con 0x5624e94d6780 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab76bb3740 10 client.16260 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5624e9551860 con 0x5624e94d6780 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.851+0000 7fab51ffb640 1 -- 172.21.15.28:0/1487879753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab64012e20 con 0x5624e94d6780 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab51ffb640 1 -- 172.21.15.28:0/1487879753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fab64004ca0 con 0x5624e94d6780 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 10 client.16260 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 10 client.16260 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 20 client.16260 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e9551f50 0x5624e9554370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 4 client.16260 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:01:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5624e9434d70 con 0x5624e9551f50 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab72576640 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e9551f50 0x5624e9554370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab72576640 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e9551f50 0x5624e9554370 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d48731a81d80bd0d server_cookie=fc7a49d10f23602d in_seq=0 out_seq=0 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab51ffb640 10 client.16260 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab51ffb640 1 -- 172.21.15.28:0/1487879753 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5624e9434d70 con 0x5624e9551f50 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab51ffb640 10 client.16260 handle_command_reply: tid=0 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 1 client.16260 shutdown 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 msgr2=0x5624e94db860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.852+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94db860 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fab64011c50 tx=0x7fab640099f0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 shutdown_connections 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x5624e9551f50 0x5624e9554370 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab51ffb640 0 client.16260 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624e94d6780 0x5624e94db860 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5624e94c8b40 0x5624e94dedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 --2- 172.21.15.28:0/1487879753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5624e94c75d0 0x5624e94de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 >> 172.21.15.28:0/1487879753 conn(0x5624e9490310 msgr2=0x5624e94c9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 shutdown_connections 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 1 -- 172.21.15.28:0/1487879753 wait complete. 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab76bb3740 20 client.16260 trim_cache size 0 max 16384 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab540a33b0 msgr2=0x7fab540a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab540a33b0 0x7fab540a5870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fab640128a0 tx=0x7fab64009860 comp rx=0 tx=0).stop 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 msgr2=0x7fab6c198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c198460 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fab5c02ade0 tx=0x7fab5c0319f0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 shutdown_connections 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fab540a33b0 0x7fab540a5870 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab6c106f40 0x7fab6c19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab6c1027c0 0x7fab6c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 --2- 172.21.15.28:0/2947587159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab6c101e10 0x7fab6c198460 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 >> 172.21.15.28:0/2947587159 conn(0x7fab6c0fd8d0 msgr2=0x7fab6c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.853+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 shutdown_connections 2024-09-15T22:01:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:29.854+0000 7fab747da640 1 -- 172.21.15.28:0/2947587159 wait complete. 2024-09-15T22:01:31.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:31 smithi028 ceph-mon[30969]: pgmap v128: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 29 MiB/s wr, 4.62k op/s 2024-09-15T22:01:31.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:31 smithi203 ceph-mon[35546]: pgmap v128: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 29 MiB/s wr, 4.62k op/s 2024-09-15T22:01:31.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:31 smithi174 ceph-mon[35244]: pgmap v128: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 29 MiB/s wr, 4.62k op/s 2024-09-15T22:01:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.109+0000 7f9905bed640 1 -- 172.21.15.28:0/3767639498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 msgr2=0x7f9900114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.109+0000 7f9905bed640 1 --2- 172.21.15.28:0/3767639498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 0x7f9900114d80 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009860 tx=0x7f98f402a360 comp rx=0 tx=0).stop 2024-09-15T22:01:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.111+0000 7f9905bed640 1 -- 172.21.15.28:0/3767639498 shutdown_connections 2024-09-15T22:01:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.111+0000 7f9905bed640 1 --2- 172.21.15.28:0/3767639498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 0x7f9900114d80 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.111+0000 7f9905bed640 1 --2- 172.21.15.28:0/3767639498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9900077040 0x7f9900112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.111+0000 7f9905bed640 1 --2- 172.21.15.28:0/3767639498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900076720 0x7f9900076b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.111+0000 7f9905bed640 1 -- 172.21.15.28:0/3767639498 >> 172.21.15.28:0/3767639498 conn(0x7f99000ff9b0 msgr2=0x7f9900101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:32.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.117+0000 7f9905bed640 1 -- 172.21.15.28:0/3767639498 shutdown_connections 2024-09-15T22:01:32.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.121+0000 7f9905bed640 1 -- 172.21.15.28:0/3767639498 wait complete. 2024-09-15T22:01:32.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.122+0000 7f9905bed640 1 Processor -- start 2024-09-15T22:01:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.125+0000 7f9905bed640 1 -- start start 2024-09-15T22:01:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.125+0000 7f9905bed640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900076720 0x7f99001a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.125+0000 7f9905bed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9900077040 0x7f99001a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.126+0000 7f9905bed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 0x7f990019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.126+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900076720 0x7f99001a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.126+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900076720 0x7f99001a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:42044/0 (socket says 172.21.15.28:42044) 2024-09-15T22:01:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.126+0000 7f98ff7fe640 1 -- 172.21.15.28:0/1147201402 learned_addr learned my addr 172.21.15.28:0/1147201402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.126+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9900118520 con 0x7f9900112990 2024-09-15T22:01:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.126+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99001183a0 con 0x7f9900077040 2024-09-15T22:01:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.126+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f99001186a0 con 0x7f9900076720 2024-09-15T22:01:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.127+0000 7f98fffff640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 0x7f990019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.128+0000 7f98fffff640 1 -- 172.21.15.28:0/1147201402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900076720 msgr2=0x7f99001a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.128+0000 7f98fffff640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900076720 0x7f99001a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.128+0000 7f98fffff640 1 -- 172.21.15.28:0/1147201402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9900077040 msgr2=0x7f99001a4bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T22:01:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.128+0000 7f98fffff640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9900077040 0x7f99001a4bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.128+0000 7f98fffff640 1 -- 172.21.15.28:0/1147201402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990019eff0 con 0x7f9900112990 2024-09-15T22:01:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.130+0000 7f98fffff640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 0x7f990019e820 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009780 tx=0x7f98f4032650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.132+0000 7f98fcff9640 1 -- 172.21.15.28:0/1147201402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4037040 con 0x7f9900112990 2024-09-15T22:01:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.132+0000 7f98fcff9640 1 -- 172.21.15.28:0/1147201402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f4032740 con 0x7f9900112990 2024-09-15T22:01:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.133+0000 7f98fcff9640 1 -- 172.21.15.28:0/1147201402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4003a80 con 0x7f9900112990 2024-09-15T22:01:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.133+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990019f280 con 0x7f9900112990 2024-09-15T22:01:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.133+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990019f630 con 0x7f9900112990 2024-09-15T22:01:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.137+0000 7f9908a5d740 1 Processor -- start 2024-09-15T22:01:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.139+0000 7f9908a5d740 1 -- start start 2024-09-15T22:01:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61faf0 0x56199c61fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c620410 0x56199c62a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c62d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f9908a5d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56199c5842f0 con 0x56199c62acb0 2024-09-15T22:01:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f9908a5d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56199c52c270 con 0x56199c620410 2024-09-15T22:01:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f9908a5d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56199c5802b0 con 0x56199c61faf0 2024-09-15T22:01:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c620410 0x56199c62a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c620410 0x56199c62a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56526/0 (socket says 172.21.15.28:56526) 2024-09-15T22:01:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.140+0000 7f98ff7fe640 1 -- 172.21.15.28:0/500197629 learned_addr learned my addr 172.21.15.28:0/500197629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.141+0000 7f98ff7fe640 1 -- 172.21.15.28:0/500197629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61faf0 msgr2=0x56199c61fed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T22:01:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.141+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/500197629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61faf0 0x56199c61fed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.141+0000 7f98ff7fe640 1 -- 172.21.15.28:0/500197629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 msgr2=0x56199c62d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T22:01:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.141+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/500197629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c62d0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.141+0000 7f98ff7fe640 1 -- 172.21.15.28:0/500197629 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56199c52b340 con 0x56199c620410 2024-09-15T22:01:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.141+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/500197629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c620410 0x56199c62a770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009dc0 tx=0x7f98f0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b3b5ae67eb8846 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.141+0000 7f98e5ffb640 1 -- 172.21.15.28:0/500197629 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0005220 con 0x56199c620410 2024-09-15T22:01:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.142+0000 7f98e5ffb640 1 -- 172.21.15.28:0/500197629 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f0004c80 con 0x56199c620410 2024-09-15T22:01:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.143+0000 7f9908a5d740 1 -- 172.21.15.28:0/500197629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c620410 msgr2=0x56199c62a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.143+0000 7f9908a5d740 1 --2- 172.21.15.28:0/500197629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c620410 0x56199c62a770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009dc0 tx=0x7f98f0011810 comp rx=0 tx=0).stop 2024-09-15T22:01:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.143+0000 7f9908a5d740 1 -- 172.21.15.28:0/500197629 shutdown_connections 2024-09-15T22:01:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.143+0000 7f9908a5d740 1 --2- 172.21.15.28:0/500197629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c62d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.143+0000 7f9908a5d740 1 --2- 172.21.15.28:0/500197629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c620410 0x56199c62a770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.143+0000 7f9908a5d740 1 --2- 172.21.15.28:0/500197629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61faf0 0x56199c61fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.143+0000 7f9908a5d740 1 -- 172.21.15.28:0/500197629 >> 172.21.15.28:0/500197629 conn(0x56199c61c060 msgr2=0x56199c61c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.145+0000 7f9908a5d740 1 -- 172.21.15.28:0/500197629 shutdown_connections 2024-09-15T22:01:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.146+0000 7f9908a5d740 1 -- 172.21.15.28:0/500197629 wait complete. 2024-09-15T22:01:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.146+0000 7f9908a5d740 1 -- start start 2024-09-15T22:01:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 1 Processor -- start 2024-09-15T22:01:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c62a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c627940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c628310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56199c63e8e0 con 0x56199c62acb0 2024-09-15T22:01:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56199c50d290 con 0x56199c61faf0 2024-09-15T22:01:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56199c63ea60 con 0x56199c627ed0 2024-09-15T22:01:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.147+0000 7f9908a5d740 10 client.0 _ll_register_callbacks cb 0x56199c53e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:32.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.148+0000 7f98fcff9640 1 -- 172.21.15.28:0/1147201402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f98f4042070 con 0x7f9900112990 2024-09-15T22:01:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.149+0000 7f98fcff9640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98e80a32a0 0x7f98e80a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.149+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c627940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.149+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c627940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35214/0 (socket says 172.21.15.28:35214) 2024-09-15T22:01:32.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.149+0000 7f98ff7fe640 1 -- 172.21.15.28:0/665887762 learned_addr learned my addr 172.21.15.28:0/665887762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.150+0000 7f98fffff640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98e80a32a0 0x7f98e80a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.150+0000 7f98feffd640 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c62a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.150+0000 7f98fcff9640 1 -- 172.21.15.28:0/1147201402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f98f40eef90 con 0x7f9900112990 2024-09-15T22:01:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.150+0000 7f98feffd640 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c628310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.151+0000 7f98feffd640 1 -- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 msgr2=0x56199c62a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.151+0000 7f98feffd640 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c62a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.151+0000 7f98feffd640 1 -- 172.21.15.28:0/665887762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 msgr2=0x56199c627940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.151+0000 7f98feffd640 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c627940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.151+0000 7f98feffd640 1 -- 172.21.15.28:0/665887762 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56199c297210 con 0x56199c627ed0 2024-09-15T22:01:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.151+0000 7f98fffff640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98e80a32a0 0x7f98e80a5760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f98f40c3780 tx=0x7f98f40bac40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.152+0000 7f98feffd640 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c628310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x56199c622690 tx=0x7f98f8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39a573a8d403a848 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.152+0000 7f98deffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:01:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.152+0000 7f98deffd640 1 -- 172.21.15.28:0/665887762 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800ddf0 con 0x56199c627ed0 2024-09-15T22:01:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.152+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56199c591a20 con 0x56199c627ed0 2024-09-15T22:01:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.153+0000 7f98deffd640 1 -- 172.21.15.28:0/665887762 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f98f8009e30 con 0x56199c627ed0 2024-09-15T22:01:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.157+0000 7f98deffd640 1 -- 172.21.15.28:0/665887762 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f98f8022050 con 0x56199c627ed0 2024-09-15T22:01:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.162+0000 7f9908a5d740 10 client.25777 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.162+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56199c363620 con 0x56199c627ed0 2024-09-15T22:01:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.163+0000 7f98deffd640 1 -- 172.21.15.28:0/665887762 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56199c591a20 con 0x56199c627ed0 2024-09-15T22:01:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.164+0000 7f98deffd640 1 -- 172.21.15.28:0/665887762 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f98f800b040 con 0x56199c627ed0 2024-09-15T22:01:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.167+0000 7f9908a5d740 10 client.25777 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.167+0000 7f9908a5d740 10 client.25777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.167+0000 7f9908a5d740 20 client.25777 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.167+0000 7f9908a5d740 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c64bf60 0x56199c64e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.167+0000 7f9908a5d740 4 client.25777 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:01:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.167+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56199c58bd70 con 0x56199c64bf60 2024-09-15T22:01:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.171+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c64bf60 0x56199c64e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.174+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c64bf60 0x56199c64e350 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28ec64c54be69d38 server_cookie=dc9ac37c76c8aa43 in_seq=0 out_seq=0 2024-09-15T22:01:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.174+0000 7f98deffd640 10 client.25777 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.177+0000 7f98deffd640 1 -- 172.21.15.28:0/665887762 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56199c58bd70 con 0x56199c64bf60 2024-09-15T22:01:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.177+0000 7f98deffd640 10 client.25777 handle_command_reply: tid=0 2024-09-15T22:01:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.177+0000 7f9908a5d740 1 client.25777 shutdown 2024-09-15T22:01:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.180+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 msgr2=0x56199c628310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.180+0000 7f9908a5d740 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c628310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x56199c622690 tx=0x7f98f8004a40 comp rx=0 tx=0).stop 2024-09-15T22:01:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.184+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 shutdown_connections 2024-09-15T22:01:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.184+0000 7f9908a5d740 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c64bf60 0x56199c64e350 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.184+0000 7f9908a5d740 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c628310 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.184+0000 7f9908a5d740 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c62acb0 0x56199c627940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.184+0000 7f9908a5d740 1 --2- 172.21.15.28:0/665887762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c62a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.184+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 >> 172.21.15.28:0/665887762 conn(0x56199c61c060 msgr2=0x56199c621550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.185+0000 7f98deffd640 0 client.25777 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:32.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.189+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 shutdown_connections 2024-09-15T22:01:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.193+0000 7f9908a5d740 1 -- 172.21.15.28:0/665887762 wait complete. 2024-09-15T22:01:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.193+0000 7f9908a5d740 20 client.25777 trim_cache size 0 max 16384 2024-09-15T22:01:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.226+0000 7f9908a5d740 1 Processor -- start 2024-09-15T22:01:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.227+0000 7f9908a5d740 1 -- start start 2024-09-15T22:01:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.228+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c61c060 0x56199c61b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.228+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c621550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.228+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c621a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.228+0000 7f9908a5d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99001ac7d0 con 0x56199c61c060 2024-09-15T22:01:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.228+0000 7f9908a5d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99001ae310 con 0x56199c61faf0 2024-09-15T22:01:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.228+0000 7f9908a5d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f99001acea0 con 0x56199c627ed0 2024-09-15T22:01:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.229+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c621550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.229+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c621550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:56546/0 (socket says 172.21.15.28:56546) 2024-09-15T22:01:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.229+0000 7f98ff7fe640 1 -- 172.21.15.28:0/3093512392 learned_addr learned my addr 172.21.15.28:0/3093512392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.229+0000 7f98feffd640 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c61c060 0x56199c61b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.230+0000 7f98feffd640 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c621a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.231+0000 7f98feffd640 1 -- 172.21.15.28:0/3093512392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 msgr2=0x56199c621a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.231+0000 7f98feffd640 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c621a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.231+0000 7f98feffd640 1 -- 172.21.15.28:0/3093512392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 msgr2=0x56199c621550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:32.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.231+0000 7f98feffd640 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c621550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.231+0000 7f98feffd640 1 -- 172.21.15.28:0/3093512392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56199c64ab50 con 0x56199c61c060 2024-09-15T22:01:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.238+0000 7f98feffd640 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c61c060 0x56199c61b9b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005c40 tx=0x7f98f8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1205fa8248395 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:32.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.238+0000 7f98e57fa640 1 -- 172.21.15.28:0/3093512392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800ddf0 con 0x56199c61c060 2024-09-15T22:01:32.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.238+0000 7f98e57fa640 1 -- 172.21.15.28:0/3093512392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f8010110 con 0x56199c61c060 2024-09-15T22:01:32.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.239+0000 7f98e57fa640 1 -- 172.21.15.28:0/3093512392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8027410 con 0x56199c61c060 2024-09-15T22:01:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.240+0000 7f9908a5d740 1 -- 172.21.15.28:0/3093512392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c61c060 msgr2=0x56199c61b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.240+0000 7f9908a5d740 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c61c060 0x56199c61b9b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005c40 tx=0x7f98f8005bd0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.244+0000 7f9908a5d740 1 -- 172.21.15.28:0/3093512392 shutdown_connections 2024-09-15T22:01:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.244+0000 7f9908a5d740 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c627ed0 0x56199c621a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.244+0000 7f9908a5d740 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c621550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.244+0000 7f9908a5d740 1 --2- 172.21.15.28:0/3093512392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c61c060 0x56199c61b9b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.244+0000 7f9908a5d740 1 -- 172.21.15.28:0/3093512392 >> 172.21.15.28:0/3093512392 conn(0x56199c62a800 msgr2=0x56199c622b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.244+0000 7f9908a5d740 1 -- 172.21.15.28:0/3093512392 shutdown_connections 2024-09-15T22:01:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.244+0000 7f9908a5d740 1 -- 172.21.15.28:0/3093512392 wait complete. 2024-09-15T22:01:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.245+0000 7f9908a5d740 1 -- start start 2024-09-15T22:01:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.245+0000 7f9908a5d740 1 Processor -- start 2024-09-15T22:01:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.246+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61c060 0x56199c644690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.246+0000 7f9908a5d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c6536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.246+0000 7f98feffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61c060 0x56199c644690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.246+0000 7f98feffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61c060 0x56199c644690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:42074/0 (socket says 172.21.15.28:42074) 2024-09-15T22:01:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.247+0000 7f98feffd640 1 -- 172.21.15.28:0/247894155 learned_addr learned my addr 172.21.15.28:0/247894155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.247+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c6536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.247+0000 7f9908a5d740 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c627ed0 0x56199c653e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.247+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99001adc40 con 0x56199c627ed0 2024-09-15T22:01:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.247+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99001ad570 con 0x56199c61faf0 2024-09-15T22:01:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.247+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f99000736b0 con 0x56199c61c060 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.247+0000 7f9908a5d740 10 client.0 _ll_register_callbacks cb 0x7f98f8009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f98feffd640 1 -- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 msgr2=0x56199c6536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f98feffd640 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c6536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f98feffd640 1 -- 172.21.15.28:0/247894155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c627ed0 msgr2=0x56199c653e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f98feffd640 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c627ed0 0x56199c653e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f98feffd640 1 -- 172.21.15.28:0/247894155 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56199c628910 con 0x56199c61c060 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98f8010110 con 0x56199c61c060 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f98feffd640 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61c060 0x56199c644690 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f98f80046c0 tx=0x7f98f8009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f5eb2aba6ce1bf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.248+0000 7f98deffd640 10 client.25783 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:01:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.250+0000 7f98deffd640 1 -- 172.21.15.28:0/247894155 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f802b050 con 0x56199c61c060 2024-09-15T22:01:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.251+0000 7f98deffd640 1 -- 172.21.15.28:0/247894155 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f98f8030040 con 0x56199c61c060 2024-09-15T22:01:32.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.251+0000 7f98deffd640 1 -- 172.21.15.28:0/247894155 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f98f801f040 con 0x56199c61c060 2024-09-15T22:01:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.252+0000 7f9908a5d740 10 client.25783 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.252+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=32}) -- 0x56199c6b6bf0 con 0x56199c61c060 2024-09-15T22:01:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.255+0000 7f98deffd640 1 -- 172.21.15.28:0/247894155 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f98f800b070 con 0x56199c61c060 2024-09-15T22:01:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.256+0000 7f9908a5d740 10 client.25783 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.256+0000 7f9908a5d740 10 client.25783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.256+0000 7f9908a5d740 20 client.25783 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.256+0000 7f9908a5d740 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c62acb0 0x56199c6b9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.256+0000 7f9908a5d740 4 client.25783 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:01:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.256+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f98f8010110 con 0x56199c62acb0 2024-09-15T22:01:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.256+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c62acb0 0x56199c6b9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.265+0000 7f98ff7fe640 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c62acb0 0x56199c6b9170 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa4c8d64067eddc9 server_cookie=b2cddc8c07602911 in_seq=0 out_seq=0 2024-09-15T22:01:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:32.265+0000 7f98deffd640 10 client.25783 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:33 smithi028 ceph-mon[30969]: pgmap v129: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 49 MiB/s wr, 7.49k op/s 2024-09-15T22:01:33.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:33 smithi203 ceph-mon[35546]: pgmap v129: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 49 MiB/s wr, 7.49k op/s 2024-09-15T22:01:33.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:33 smithi174 ceph-mon[35244]: pgmap v129: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 49 MiB/s wr, 7.49k op/s 2024-09-15T22:01:35.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:35 smithi028 ceph-mon[30969]: pgmap v130: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 54 MiB/s wr, 8.23k op/s 2024-09-15T22:01:35.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:35 smithi203 ceph-mon[35546]: pgmap v130: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 54 MiB/s wr, 8.23k op/s 2024-09-15T22:01:35.762 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:35 smithi174 ceph-mon[35244]: pgmap v130: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 54 MiB/s wr, 8.23k op/s 2024-09-15T22:01:37.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:37 smithi203 ceph-mon[35546]: pgmap v131: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 55 MiB/s wr, 8.32k op/s 2024-09-15T22:01:37.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:37 smithi174 ceph-mon[35244]: pgmap v131: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 55 MiB/s wr, 8.32k op/s 2024-09-15T22:01:37.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:37 smithi028 ceph-mon[30969]: pgmap v131: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 55 MiB/s wr, 8.32k op/s 2024-09-15T22:01:38.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:38 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:01:38.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:38 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:01:38.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:38 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:01:39.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:39 smithi203 ceph-mon[35546]: pgmap v132: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 71 MiB/s wr, 11.11k op/s 2024-09-15T22:01:39.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:39 smithi174 ceph-mon[35244]: pgmap v132: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 71 MiB/s wr, 11.11k op/s 2024-09-15T22:01:39.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:39 smithi028 ceph-mon[30969]: pgmap v132: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 71 MiB/s wr, 11.11k op/s 2024-09-15T22:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f98deffd640 1 -- 172.21.15.28:0/247894155 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f98f8010110 con 0x56199c62acb0 2024-09-15T22:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f98deffd640 10 client.25783 handle_command_reply: tid=0 2024-09-15T22:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 client.25783 shutdown 2024-09-15T22:01:40.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61c060 msgr2=0x56199c644690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61c060 0x56199c644690 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f98f80046c0 tx=0x7f98f8009b90 comp rx=0 tx=0).stop 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 shutdown_connections 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56199c62acb0 0x56199c6b9170 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56199c627ed0 0x56199c653e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f98deffd640 0 client.25783 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56199c61faf0 0x56199c6536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 --2- 172.21.15.28:0/247894155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56199c61c060 0x56199c644690 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 >> 172.21.15.28:0/247894155 conn(0x56199c62a800 msgr2=0x56199c621550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 shutdown_connections 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.412+0000 7f9908a5d740 1 -- 172.21.15.28:0/247894155 wait complete. 2024-09-15T22:01:40.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9908a5d740 20 client.25783 trim_cache size 0 max 16384 2024-09-15T22:01:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:01:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:01:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:01:40.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98e80a32a0 msgr2=0x7f98e80a5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98e80a32a0 0x7f98e80a5760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f98f40c3780 tx=0x7f98f40bac40 comp rx=0 tx=0).stop 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 msgr2=0x7f990019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 0x7f990019e820 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009780 tx=0x7f98f4032650 comp rx=0 tx=0).stop 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 shutdown_connections 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98e80a32a0 0x7f98e80a5760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9900112990 0x7f990019e820 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9900077040 0x7f99001a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 --2- 172.21.15.28:0/1147201402 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9900076720 0x7f99001a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 >> 172.21.15.28:0/1147201402 conn(0x7f99000ff9b0 msgr2=0x7f9900101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 shutdown_connections 2024-09-15T22:01:40.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:40.413+0000 7f9905bed640 1 -- 172.21.15.28:0/1147201402 wait complete. 2024-09-15T22:01:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:41 smithi203 ceph-mon[35546]: pgmap v133: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 52 MiB/s wr, 8.15k op/s 2024-09-15T22:01:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:41 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:01:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:41 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:01:41.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:41 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:01:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:41 smithi174 ceph-mon[35244]: pgmap v133: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 52 MiB/s wr, 8.15k op/s 2024-09-15T22:01:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:41 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:01:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:41 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:01:41.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:41 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:01:41.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:41 smithi028 ceph-mon[30969]: pgmap v133: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 52 MiB/s wr, 8.15k op/s 2024-09-15T22:01:41.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:41 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:01:41.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:41 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:01:41.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:41 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:01:43.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:01:43.121 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:01:43.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:43 smithi203 ceph-mon[35546]: pgmap v134: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 62 MiB/s wr, 9.91k op/s 2024-09-15T22:01:43.759 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:43 smithi174 ceph-mon[35244]: pgmap v134: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 62 MiB/s wr, 9.91k op/s 2024-09-15T22:01:43.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:43 smithi028 ceph-mon[30969]: pgmap v134: 129 pgs: 129 active+clean; 15 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 62 MiB/s wr, 9.91k op/s 2024-09-15T22:01:45.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:45 smithi203 ceph-mon[35546]: pgmap v135: 129 pgs: 129 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 46 MiB/s wr, 7.62k op/s 2024-09-15T22:01:45.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:45 smithi174 ceph-mon[35244]: pgmap v135: 129 pgs: 129 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 46 MiB/s wr, 7.62k op/s 2024-09-15T22:01:45.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:45 smithi028 ceph-mon[30969]: pgmap v135: 129 pgs: 129 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 46 MiB/s wr, 7.62k op/s 2024-09-15T22:01:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:47 smithi203 ceph-mon[35546]: pgmap v136: 129 pgs: 129 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 40 MiB/s wr, 6.72k op/s 2024-09-15T22:01:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:47 smithi174 ceph-mon[35244]: pgmap v136: 129 pgs: 129 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 40 MiB/s wr, 6.72k op/s 2024-09-15T22:01:47.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:47 smithi028 ceph-mon[30969]: pgmap v136: 129 pgs: 129 active+clean; 15 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 40 MiB/s wr, 6.72k op/s 2024-09-15T22:01:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:49 smithi203 ceph-mon[35546]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 51 MiB/s wr, 8.33k op/s 2024-09-15T22:01:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:49 smithi174 ceph-mon[35244]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 51 MiB/s wr, 8.33k op/s 2024-09-15T22:01:49.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:49 smithi028 ceph-mon[30969]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 51 MiB/s wr, 8.33k op/s 2024-09-15T22:01:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:01:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:51 smithi203 ceph-mon[35546]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 36 MiB/s wr, 5.73k op/s 2024-09-15T22:01:51.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:51 smithi174 ceph-mon[35244]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 36 MiB/s wr, 5.73k op/s 2024-09-15T22:01:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:51 smithi028 ceph-mon[30969]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 36 MiB/s wr, 5.73k op/s 2024-09-15T22:01:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:53 smithi203 ceph-mon[35546]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 46 MiB/s wr, 7.11k op/s 2024-09-15T22:01:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:53 smithi174 ceph-mon[35244]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 46 MiB/s wr, 7.11k op/s 2024-09-15T22:01:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.812+0000 7f2602b18640 1 -- 172.21.15.28:0/2012077313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc101350 msgr2=0x7f25fc10d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.812+0000 7f2602b18640 1 --2- 172.21.15.28:0/2012077313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc101350 0x7f25fc10d880 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009cd0 tx=0x7f25f402a8e0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.812+0000 7f2602b18640 1 -- 172.21.15.28:0/2012077313 shutdown_connections 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.812+0000 7f2602b18640 1 --2- 172.21.15.28:0/2012077313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc101350 0x7f25fc10d880 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.812+0000 7f2602b18640 1 --2- 172.21.15.28:0/2012077313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25fc1009b0 0x7f25fc100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.812+0000 7f2602b18640 1 --2- 172.21.15.28:0/2012077313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc1069c0 0x7f25fc106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.812+0000 7f2602b18640 1 -- 172.21.15.28:0/2012077313 >> 172.21.15.28:0/2012077313 conn(0x7f25fc077990 msgr2=0x7f25fc0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.813+0000 7f2602b18640 1 -- 172.21.15.28:0/2012077313 shutdown_connections 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.813+0000 7f2602b18640 1 -- 172.21.15.28:0/2012077313 wait complete. 2024-09-15T22:01:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.813+0000 7f2602b18640 1 Processor -- start 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.813+0000 7f2602b18640 1 -- start start 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f2602b18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1009b0 0x7f25fc197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f2602b18640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25fc101350 0x7f25fc197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f2602b18640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc1069c0 0x7f25fc19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f2602b18640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc110e00 con 0x7f25fc1009b0 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f2602b18640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc110c80 con 0x7f25fc1069c0 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f2602b18640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc110f80 con 0x7f25fc101350 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1009b0 0x7f25fc197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1009b0 0x7f25fc197210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50706/0 (socket says 172.21.15.28:50706) 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 -- 172.21.15.28:0/223050003 learned_addr learned my addr 172.21.15.28:0/223050003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f25fbfff640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25fc101350 0x7f25fc197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26010b5640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc1069c0 0x7f25fc19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 -- 172.21.15.28:0/223050003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25fc101350 msgr2=0x7f25fc197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25fc101350 0x7f25fc197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 -- 172.21.15.28:0/223050003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc1069c0 msgr2=0x7f25fc19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc1069c0 0x7f25fc19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 -- 172.21.15.28:0/223050003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc19c200 con 0x7f25fc1009b0 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26010b5640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc1069c0 0x7f25fc19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f25fbfff640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25fc101350 0x7f25fc197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:01:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f26008b4640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1009b0 0x7f25fc197210 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f25f000eb00 tx=0x7f25f0009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f25f9ffb640 1 -- 172.21.15.28:0/223050003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f001c070 con 0x7f25fc1009b0 2024-09-15T22:01:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25fc19c4f0 con 0x7f25fc1009b0 2024-09-15T22:01:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.814+0000 7f25f9ffb640 1 -- 172.21.15.28:0/223050003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f000bc30 con 0x7f25fc1009b0 2024-09-15T22:01:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.815+0000 7f25f9ffb640 1 -- 172.21.15.28:0/223050003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f00166a0 con 0x7f25fc1009b0 2024-09-15T22:01:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.815+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25fc0698a0 con 0x7f25fc1009b0 2024-09-15T22:01:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.817+0000 7f25f9ffb640 1 -- 172.21.15.28:0/223050003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f25f001f070 con 0x7f25fc1009b0 2024-09-15T22:01:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 Processor -- start 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 -- start start 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f25f9ffb640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25ec0a3460 0x7f25ec0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0cefd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0cfeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0d014b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559af0c542f0 con 0x559af0cef990 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559af0bfc270 con 0x559af0cff0c0 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.818+0000 7f26058b1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559af0c502b0 con 0x559af0cf02b0 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0cefd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26010b5640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25ec0a3460 0x7f25ec0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0cefd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50708/0 (socket says 172.21.15.28:50708) 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 -- 172.21.15.28:0/1129353049 learned_addr learned my addr 172.21.15.28:0/1129353049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f25fbfff640 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0cfeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f25f9ffb640 1 -- 172.21.15.28:0/223050003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f25f00d1680 con 0x7f25fc1009b0 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 -- 172.21.15.28:0/1129353049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 msgr2=0x559af0cfeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0cfeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 -- 172.21.15.28:0/1129353049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 msgr2=0x559af0d014b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0d014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 -- 172.21.15.28:0/1129353049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af0bfb340 con 0x559af0cef990 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f26008b4640 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0cefd70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f25f0099fd0 tx=0x7f25f009d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1c625d1a8d69ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f25e6ffd640 1 -- 172.21.15.28:0/1129353049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0099040 con 0x559af0cef990 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f25e6ffd640 1 -- 172.21.15.28:0/1129353049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f000bde0 con 0x559af0cef990 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.819+0000 7f25e6ffd640 1 -- 172.21.15.28:0/1129353049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f00a19b0 con 0x559af0cef990 2024-09-15T22:01:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 -- 172.21.15.28:0/1129353049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 msgr2=0x559af0cefd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26010b5640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25ec0a3460 0x7f25ec0a5920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f25f4006010 tx=0x7f25f4009800 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0cefd70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f25f0099fd0 tx=0x7f25f009d090 comp rx=0 tx=0).stop 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 -- 172.21.15.28:0/1129353049 shutdown_connections 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0d014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0cfeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 --2- 172.21.15.28:0/1129353049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0cefd70 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 -- 172.21.15.28:0/1129353049 >> 172.21.15.28:0/1129353049 conn(0x559af0cec060 msgr2=0x559af0cec440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 -- 172.21.15.28:0/1129353049 shutdown_connections 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 -- 172.21.15.28:0/1129353049 wait complete. 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 -- start start 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.820+0000 7f26058b1740 1 Processor -- start 2024-09-15T22:01:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26058b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cef990 0x559af0cfc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26058b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0cfc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26058b1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cff0c0 0x559af0cfcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26058b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559af0d01c90 con 0x559af0cf02b0 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26058b1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559af0bdd290 con 0x559af0cef990 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26058b1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559af0d01e10 con 0x559af0cff0c0 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26058b1740 10 client.0 _ll_register_callbacks cb 0x559af0c0e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0cfc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0cfc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50718/0 (socket says 172.21.15.28:50718) 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f25fbfff640 1 -- 172.21.15.28:0/4043955744 learned_addr learned my addr 172.21.15.28:0/4043955744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26010b5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cff0c0 0x559af0cfcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.821+0000 7f26008b4640 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cef990 0x559af0cfc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25fbfff640 1 -- 172.21.15.28:0/4043955744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cff0c0 msgr2=0x559af0cfcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25fbfff640 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cff0c0 0x559af0cfcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25fbfff640 1 -- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cef990 msgr2=0x559af0cfc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25fbfff640 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cef990 0x559af0cfc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25fbfff640 1 -- 172.21.15.28:0/4043955744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559af0967210 con 0x559af0cf02b0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f26008b4640 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cef990 0x559af0cfc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25fbfff640 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0cfc590 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f25e8013780 tx=0x7f25e8013c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91336be0491370e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25dbfff640 1 -- 172.21.15.28:0/4043955744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e8011dd0 con 0x559af0cf02b0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25dbfff640 1 -- 172.21.15.28:0/4043955744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f25e8005020 con 0x559af0cf02b0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25f000bde0 con 0x559af0cf02b0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25dbfff640 1 -- 172.21.15.28:0/4043955744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f25e8026050 con 0x559af0cf02b0 2024-09-15T22:01:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f26058b1740 10 client.16284 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559af0a33620 con 0x559af0cf02b0 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.822+0000 7f25dbfff640 1 -- 172.21.15.28:0/4043955744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f000bde0 con 0x559af0cf02b0 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f25dbfff640 1 -- 172.21.15.28:0/4043955744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f25e801a030 con 0x559af0cf02b0 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26058b1740 10 client.16284 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26058b1740 10 client.16284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26058b1740 20 client.16284 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26058b1740 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d11570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26058b1740 4 client.16284 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559af0c61a20 con 0x559af0d0f100 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26008b4640 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d11570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f26008b4640 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d11570 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7f449bd2e5d19c1 server_cookie=d2bbaf3b74ef8259 in_seq=0 out_seq=0 2024-09-15T22:01:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.823+0000 7f25dbfff640 10 client.16284 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.824+0000 7f25dbfff640 1 -- 172.21.15.28:0/4043955744 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559af0c61a20 con 0x559af0d0f100 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.824+0000 7f25dbfff640 10 client.16284 handle_command_reply: tid=0 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.824+0000 7f26058b1740 1 client.16284 shutdown 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 msgr2=0x559af0cfc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0cfc590 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f25e8013780 tx=0x7f25e8013c40 comp rx=0 tx=0).stop 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 shutdown_connections 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d11570 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cff0c0 0x559af0cfcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f25dbfff640 0 client.16284 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0cfc590 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 --2- 172.21.15.28:0/4043955744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cef990 0x559af0cfc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 >> 172.21.15.28:0/4043955744 conn(0x559af0cec060 msgr2=0x559af0cf1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 shutdown_connections 2024-09-15T22:01:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 1 -- 172.21.15.28:0/4043955744 wait complete. 2024-09-15T22:01:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.825+0000 7f26058b1740 20 client.16284 trim_cache size 0 max 16384 2024-09-15T22:01:53.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:53 smithi028 ceph-mon[30969]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 46 MiB/s wr, 7.11k op/s 2024-09-15T22:01:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.841+0000 7f26058b1740 1 Processor -- start 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26058b1740 1 -- start start 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26058b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0d16e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26058b1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0d173c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26058b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0cf9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26058b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1accc0 con 0x559af0cef990 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26058b1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ae800 con 0x559af0cff0c0 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26058b1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ad390 con 0x559af0cf02b0 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f26010b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0cf9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0d173c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0d173c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:34574/0 (socket says 172.21.15.28:34574) 2024-09-15T22:01:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.842+0000 7f25fbfff640 1 -- 172.21.15.28:0/3480554184 learned_addr learned my addr 172.21.15.28:0/3480554184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25fbfff640 1 -- 172.21.15.28:0/3480554184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 msgr2=0x559af0cf9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25fbfff640 1 --2- 172.21.15.28:0/3480554184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0cf9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25fbfff640 1 -- 172.21.15.28:0/3480554184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 msgr2=0x559af0d16e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25fbfff640 1 --2- 172.21.15.28:0/3480554184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0d16e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25fbfff640 1 -- 172.21.15.28:0/3480554184 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559af0cfafe0 con 0x559af0cf02b0 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25fbfff640 1 --2- 172.21.15.28:0/3480554184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0d173c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25e801b470 tx=0x7f25e8009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=112f7850c24a8db2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25e67fc640 1 -- 172.21.15.28:0/3480554184 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e801b6e0 con 0x559af0cf02b0 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25e67fc640 1 -- 172.21.15.28:0/3480554184 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25e801b880 con 0x559af0cf02b0 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.843+0000 7f25e67fc640 1 -- 172.21.15.28:0/3480554184 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e80126b0 con 0x559af0cf02b0 2024-09-15T22:01:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 -- 172.21.15.28:0/3480554184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 msgr2=0x559af0d173c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 --2- 172.21.15.28:0/3480554184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0d173c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25e801b470 tx=0x7f25e8009dd0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 -- 172.21.15.28:0/3480554184 shutdown_connections 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 --2- 172.21.15.28:0/3480554184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0cf9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 --2- 172.21.15.28:0/3480554184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cf02b0 0x559af0d173c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 --2- 172.21.15.28:0/3480554184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cef990 0x559af0d16e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 -- 172.21.15.28:0/3480554184 >> 172.21.15.28:0/3480554184 conn(0x559af0cec060 msgr2=0x559af0cfe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 -- 172.21.15.28:0/3480554184 shutdown_connections 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 -- 172.21.15.28:0/3480554184 wait complete. 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 -- start start 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.844+0000 7f26058b1740 1 Processor -- start 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26058b1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cef990 0x559af0cf6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26058b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0d07060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26058b1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0d075a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26058b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ae130 con 0x559af0cf02b0 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26058b1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1ada60 con 0x559af0cff0c0 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26058b1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc1aaca0 con 0x559af0cef990 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26058b1740 10 client.0 _ll_register_callbacks cb 0x7f25e8009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0d07060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0d07060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50730/0 (socket says 172.21.15.28:50730) 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26008b4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cef990 0x559af0cf6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 -- 172.21.15.28:0/1313885182 learned_addr learned my addr 172.21.15.28:0/1313885182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26010b5640 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0d075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 -- 172.21.15.28:0/1313885182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cef990 msgr2=0x559af0cf6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cef990 0x559af0cf6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 -- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 msgr2=0x559af0d075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:01:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0d075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f25fbfff640 1 -- 172.21.15.28:0/1313885182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559af0d07be0 con 0x559af0cf02b0 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.845+0000 7f26008b4640 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cef990 0x559af0cf6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f25fbfff640 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0d07060 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25e801b3f0 tx=0x7f25e8013160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9937b4924b83928e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f25dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f25dbfff640 1 -- 172.21.15.28:0/1313885182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e801ba30 con 0x559af0cf02b0 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25e801b880 con 0x559af0cf02b0 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f25dbfff640 1 -- 172.21.15.28:0/1313885182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f25e801bbd0 con 0x559af0cf02b0 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f25dbfff640 1 -- 172.21.15.28:0/1313885182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f25e801bdd0 con 0x559af0cf02b0 2024-09-15T22:01:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f26058b1740 10 client.16287 fetch_fsmap learned FSMap version 32 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559af0d8f650 con 0x559af0cf02b0 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.846+0000 7f25dbfff640 1 -- 172.21.15.28:0/1313885182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25e803a020 con 0x559af0cf02b0 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f25dbfff640 1 -- 172.21.15.28:0/1313885182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f25e801a040 con 0x559af0cf02b0 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f26058b1740 10 client.16287 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f26058b1740 10 client.16287 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f26058b1740 20 client.16287 populate_metadata read hostname 'smithi028' 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f26058b1740 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d91cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f26058b1740 4 client.16287 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559af0c5bd70 con 0x559af0d0f100 2024-09-15T22:01:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.847+0000 7f26008b4640 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d91cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:53.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.848+0000 7f25dbfff640 10 client.16287 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:01:53.854 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:01:53.848+0000 7f26008b4640 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d91cf0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbda7157153a11 server_cookie=112d8c0345d096c9 in_seq=0 out_seq=0 2024-09-15T22:01:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 -- 172.21.15.28:0/3042169236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a81027c0 msgr2=0x7f52a8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/3042169236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a81027c0 0x7f52a8106810 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f529c009900 tx=0x7f529c02a410 comp rx=0 tx=0).stop 2024-09-15T22:01:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 -- 172.21.15.28:0/3042169236 shutdown_connections 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/3042169236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52a8106f40 0x7f52a810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/3042169236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a81027c0 0x7f52a8106810 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/3042169236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a8101e10 0x7f52a81021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 -- 172.21.15.28:0/3042169236 >> 172.21.15.28:0/3042169236 conn(0x7f52a80fd8d0 msgr2=0x7f52a80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 -- 172.21.15.28:0/3042169236 shutdown_connections 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 -- 172.21.15.28:0/3042169236 wait complete. 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.782+0000 7f52ae1c2640 1 Processor -- start 2024-09-15T22:01:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.783+0000 7f52ae1c2640 1 -- start start 2024-09-15T22:01:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.784+0000 7f52ae1c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52a8101e10 0x7f52a8198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.784+0000 7f52ae1c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a81027c0 0x7f52a8198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.784+0000 7f52ae1c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a8106f40 0x7f52a819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.784+0000 7f52ae1c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8111fa0 con 0x7f52a8106f40 2024-09-15T22:01:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.784+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a81027c0 0x7f52a8198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.784+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a81027c0 0x7f52a8198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:34886/0 (socket says 172.21.15.28:34886) 2024-09-15T22:01:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.784+0000 7f52a6ffd640 1 -- 172.21.15.28:0/4025756163 learned_addr learned my addr 172.21.15.28:0/4025756163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.785+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8111e20 con 0x7f52a81027c0 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.785+0000 7f52a7fff640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a8106f40 0x7f52a819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.785+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52a8112120 con 0x7f52a8101e10 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.786+0000 7f52a77fe640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52a8101e10 0x7f52a8198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.786+0000 7f52a77fe640 1 -- 172.21.15.28:0/4025756163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a81027c0 msgr2=0x7f52a8198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.786+0000 7f52a77fe640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a81027c0 0x7f52a8198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.786+0000 7f52a77fe640 1 -- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a8106f40 msgr2=0x7f52a819ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.786+0000 7f52a77fe640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a8106f40 0x7f52a819ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.786+0000 7f52a77fe640 1 -- 172.21.15.28:0/4025756163 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a819d3b0 con 0x7f52a8101e10 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.786+0000 7f52a6ffd640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a81027c0 0x7f52a8198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.787+0000 7f52a77fe640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52a8101e10 0x7f52a8198400 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f529800e9f0 tx=0x7f529800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.787+0000 7f52a4ff9640 1 -- 172.21.15.28:0/4025756163 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529800ccc0 con 0x7f52a8101e10 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.787+0000 7f52a4ff9640 1 -- 172.21.15.28:0/4025756163 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5298010910 con 0x7f52a8101e10 2024-09-15T22:01:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.787+0000 7f52a4ff9640 1 -- 172.21.15.28:0/4025756163 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298018960 con 0x7f52a8101e10 2024-09-15T22:01:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.787+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a819d6a0 con 0x7f52a8101e10 2024-09-15T22:01:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.787+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a8074420 con 0x7f52a8101e10 2024-09-15T22:01:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.788+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a81021f0 con 0x7f52a8101e10 2024-09-15T22:01:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.789+0000 7f52a4ff9640 1 -- 172.21.15.28:0/4025756163 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f5298010ac0 con 0x7f52a8101e10 2024-09-15T22:01:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.790+0000 7f52a4ff9640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52900a77c0 0x7f52900a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:01:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.790+0000 7f52a6ffd640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52900a77c0 0x7f52900a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:01:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.790+0000 7f52a4ff9640 1 -- 172.21.15.28:0/4025756163 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f52980d0500 con 0x7f52a8101e10 2024-09-15T22:01:54.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.791+0000 7f52a6ffd640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52900a77c0 0x7f52900a9c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f529c031550 tx=0x7f529c005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:01:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:54.793+0000 7f52a4ff9640 1 -- 172.21.15.28:0/4025756163 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f529809c530 con 0x7f52a8101e10 2024-09-15T22:01:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.087+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52a810a180 con 0x7f52a8101e10 2024-09-15T22:01:55.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.088+0000 7f52a4ff9640 1 -- 172.21.15.28:0/4025756163 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11986 (secure 0 0 0) 0x7f52980a0830 con 0x7f52a8101e10 2024-09-15T22:01:55.091 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:01:55.091 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"btime":"2024-09-15T22:01:08:357214+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:01:08.288094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:01:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-09-15T22:01:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52900a77c0 msgr2=0x7f52900a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52900a77c0 0x7f52900a9c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f529c031550 tx=0x7f529c005a20 comp rx=0 tx=0).stop 2024-09-15T22:01:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52a8101e10 msgr2=0x7f52a8198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:01:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52a8101e10 0x7f52a8198400 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f529800e9f0 tx=0x7f529800eeb0 comp rx=0 tx=0).stop 2024-09-15T22:01:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 shutdown_connections 2024-09-15T22:01:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52900a77c0 0x7f52900a9c80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a8106f40 0x7f52a819ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52a81027c0 0x7f52a8198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 --2- 172.21.15.28:0/4025756163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52a8101e10 0x7f52a8198400 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:01:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.090+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 >> 172.21.15.28:0/4025756163 conn(0x7f52a80fd8d0 msgr2=0x7f52a80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:01:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.091+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 shutdown_connections 2024-09-15T22:01:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:01:55.091+0000 7f52ae1c2640 1 -- 172.21.15.28:0/4025756163 wait complete. 2024-09-15T22:01:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:55 smithi203 ceph-mon[35546]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 35 MiB/s wr, 5.21k op/s 2024-09-15T22:01:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:55 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4025756163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:01:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:55 smithi174 ceph-mon[35244]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 35 MiB/s wr, 5.21k op/s 2024-09-15T22:01:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:55 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4025756163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:01:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:55 smithi028 ceph-mon[30969]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 35 MiB/s wr, 5.21k op/s 2024-09-15T22:01:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:55 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4025756163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:01:57.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:57 smithi174 ceph-mon[35244]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 36 MiB/s wr, 5.24k op/s 2024-09-15T22:01:57.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:57 smithi028 ceph-mon[30969]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 36 MiB/s wr, 5.24k op/s 2024-09-15T22:01:57.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:57 smithi203 ceph-mon[35546]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 36 MiB/s wr, 5.24k op/s 2024-09-15T22:01:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:01:59 smithi174 ceph-mon[35244]: pgmap v142: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 47 MiB/s wr, 6.81k op/s 2024-09-15T22:01:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:01:59 smithi028 ceph-mon[30969]: pgmap v142: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 47 MiB/s wr, 6.81k op/s 2024-09-15T22:01:59.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:01:59 smithi203 ceph-mon[35546]: pgmap v142: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 47 MiB/s wr, 6.81k op/s 2024-09-15T22:02:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:01 smithi174 ceph-mon[35244]: pgmap v143: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 37 MiB/s wr, 5.49k op/s 2024-09-15T22:02:01.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:01 smithi028 ceph-mon[30969]: pgmap v143: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 37 MiB/s wr, 5.49k op/s 2024-09-15T22:02:01.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:01 smithi203 ceph-mon[35546]: pgmap v143: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 37 MiB/s wr, 5.49k op/s 2024-09-15T22:02:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 -- 172.21.15.28:0/4039429351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0106a80 msgr2=0x7fb8a0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 --2- 172.21.15.28:0/4039429351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0106a80 0x7fb8a0106e60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009930 tx=0x7fb89c02a420 comp rx=0 tx=0).stop 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 -- 172.21.15.28:0/4039429351 shutdown_connections 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 --2- 172.21.15.28:0/4039429351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8a0101420 0x7fb8a010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 --2- 172.21.15.28:0/4039429351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8a0100a80 0x7fb8a0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 --2- 172.21.15.28:0/4039429351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0106a80 0x7fb8a0106e60 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 -- 172.21.15.28:0/4039429351 >> 172.21.15.28:0/4039429351 conn(0x7fb8a00fc7c0 msgr2=0x7fb8a00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 -- 172.21.15.28:0/4039429351 shutdown_connections 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.009+0000 7fb8a7824640 1 -- 172.21.15.28:0/4039429351 wait complete. 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 Processor -- start 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 -- start start 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8a0100a80 0x7fb8a0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0101420 0x7fb8a0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8a0106a80 0x7fb8a019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0110ed0 con 0x7fb8a0101420 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0110d50 con 0x7fb8a0100a80 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a7824640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0111050 con 0x7fb8a0106a80 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a4dbf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0101420 0x7fb8a0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a55c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8a0100a80 0x7fb8a0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a4dbf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0101420 0x7fb8a0198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44076/0 (socket says 172.21.15.28:44076) 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a4dbf640 1 -- 172.21.15.28:0/1959254855 learned_addr learned my addr 172.21.15.28:0/1959254855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.010+0000 7fb8a5dc1640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8a0106a80 0x7fb8a019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a4dbf640 1 -- 172.21.15.28:0/1959254855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8a0106a80 msgr2=0x7fb8a019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8a0106a80 0x7fb8a019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a4dbf640 1 -- 172.21.15.28:0/1959254855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8a0100a80 msgr2=0x7fb8a0198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8a0100a80 0x7fb8a0198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a4dbf640 1 -- 172.21.15.28:0/1959254855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a019d3b0 con 0x7fb8a0101420 2024-09-15T22:02:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0101420 0x7fb8a0198940 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e9f0 tx=0x7fb89000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8967fc640 1 -- 172.21.15.28:0/1959254855 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89000cd80 con 0x7fb8a0101420 2024-09-15T22:02:03.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8967fc640 1 -- 172.21.15.28:0/1959254855 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb890010910 con 0x7fb8a0101420 2024-09-15T22:02:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8967fc640 1 -- 172.21.15.28:0/1959254855 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890018990 con 0x7fb8a0101420 2024-09-15T22:02:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a019d6a0 con 0x7fb8a0101420 2024-09-15T22:02:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.011+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a01a4f80 con 0x7fb8a0101420 2024-09-15T22:02:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.014+0000 7fb8967fc640 1 -- 172.21.15.28:0/1959254855 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fb890010ac0 con 0x7fb8a0101420 2024-09-15T22:02:03.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.014+0000 7fb8967fc640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb88c0a3430 0x7fb88c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.014+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb88c0a3430 0x7fb88c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.014+0000 7fb8967fc640 1 -- 172.21.15.28:0/1959254855 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb8900d02f0 con 0x7fb8a0101420 2024-09-15T22:02:03.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.015+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb88c0a3430 0x7fb88c0a58f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb89c006010 tx=0x7fb89c000bc0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 Processor -- start 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 -- start start 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3d4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55af5f33a2f0 con 0x55af5f3d5b40 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55af5f2e1cf0 con 0x55af5f3e3780 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af5f3362b0 con 0x55af5f3d45d0 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a55c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3d4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a55c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3d4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44084/0 (socket says 172.21.15.28:44084) 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.164+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1741717254 learned_addr learned my addr 172.21.15.28:0/1741717254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a5dc1640 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1741717254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 msgr2=0x55af5f3d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1741717254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 msgr2=0x55af5f3e5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1741717254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af5f2c63a0 con 0x55af5f3d5b40 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3d4090 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009930 tx=0x7fb89c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3dc8be0797089b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb88b7fe640 1 -- 172.21.15.28:0/1741717254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c035040 con 0x55af5f3d5b40 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb88b7fe640 1 -- 172.21.15.28:0/1741717254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb89c04e070 con 0x55af5f3d5b40 2024-09-15T22:02:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.165+0000 7fb88b7fe640 1 -- 172.21.15.28:0/1741717254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c032cd0 con 0x55af5f3d5b40 2024-09-15T22:02:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1741717254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 msgr2=0x55af5f3d4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3d4090 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009930 tx=0x7fb89c03d000 comp rx=0 tx=0).stop 2024-09-15T22:02:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1741717254 shutdown_connections 2024-09-15T22:02:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1741717254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3d4090 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1741717254 >> 172.21.15.28:0/1741717254 conn(0x55af5f39d310 msgr2=0x55af5f3d0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1741717254 shutdown_connections 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1741717254 wait complete. 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 -- start start 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.166+0000 7fb8a9bfd740 1 Processor -- start 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d45d0 0x55af5f3e30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d5b40 0x55af5f3e03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55af5f3e6420 con 0x55af5f3d45d0 2024-09-15T22:02:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55af5f2f5060 con 0x55af5f3e3780 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af5f3e65a0 con 0x55af5f3d5b40 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a9bfd740 10 client.0 _ll_register_callbacks cb 0x55af5f0488c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d45d0 0x55af5f3e30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d45d0 0x55af5f3e30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44098/0 (socket says 172.21.15.28:44098) 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1271484858 learned_addr learned my addr 172.21.15.28:0/1271484858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a5dc1640 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d5b40 0x55af5f3e03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d5b40 msgr2=0x55af5f3e03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d5b40 0x55af5f3e03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1271484858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 msgr2=0x55af5f3e0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.167+0000 7fb8a55c0640 1 -- 172.21.15.28:0/1271484858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af5f0ab3c0 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d45d0 0x55af5f3e30f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb89c005dd0 tx=0x7fb89c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aefee8c706e2d94 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb888ff9640 1 -- 172.21.15.28:0/1271484858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c035070 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb888ff9640 1 -- 172.21.15.28:0/1271484858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb89c0046c0 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb89c04e070 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb888ff9640 1 -- 172.21.15.28:0/1271484858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb89c064050 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb8a9bfd740 10 client.16305 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55af5f3f1e30 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.168+0000 7fb888ff9640 1 -- 172.21.15.28:0/1271484858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb89c04e070 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb888ff9640 1 -- 172.21.15.28:0/1271484858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fb89c0038d0 con 0x55af5f3d45d0 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a9bfd740 10 client.16305 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a9bfd740 10 client.16305 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:02:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a9bfd740 20 client.16305 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f3f3180 0x55af5f3f5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a9bfd740 4 client.16305 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:02:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af5f347860 con 0x55af5f3f3180 2024-09-15T22:02:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f3f3180 0x55af5f3f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb8a4dbf640 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f3f3180 0x55af5f3f5570 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=973b7a2b26b7f9bf server_cookie=ad755a90b146de62 in_seq=0 out_seq=0 2024-09-15T22:02:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.169+0000 7fb888ff9640 10 client.16305 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:02:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb888ff9640 1 -- 172.21.15.28:0/1271484858 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55af5f347860 con 0x55af5f3f3180 2024-09-15T22:02:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb888ff9640 10 client.16305 handle_command_reply: tid=0 2024-09-15T22:02:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 client.16305 shutdown 2024-09-15T22:02:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d45d0 msgr2=0x55af5f3e30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d45d0 0x55af5f3e30f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb89c005dd0 tx=0x7fb89c005e00 comp rx=0 tx=0).stop 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 shutdown_connections 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f3f3180 0x55af5f3f5570 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d5b40 0x55af5f3e03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb888ff9640 0 client.16305 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/1271484858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d45d0 0x55af5f3e30f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.170+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 >> 172.21.15.28:0/1271484858 conn(0x55af5f39d310 msgr2=0x55af5f3d6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.171+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 shutdown_connections 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.171+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/1271484858 wait complete. 2024-09-15T22:02:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.171+0000 7fb8a9bfd740 20 client.16305 trim_cache size 0 max 16384 2024-09-15T22:02:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 Processor -- start 2024-09-15T22:02:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 -- start start 2024-09-15T22:02:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3dfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3e0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a01acc60 con 0x55af5f3d5b40 2024-09-15T22:02:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a01ae7a0 con 0x55af5f3e3780 2024-09-15T22:02:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.194+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a01ad330 con 0x55af5f3d45d0 2024-09-15T22:02:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3dfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3dfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:41306/0 (socket says 172.21.15.28:41306) 2024-09-15T22:02:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 -- 172.21.15.28:0/465406013 learned_addr learned my addr 172.21.15.28:0/465406013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 -- 172.21.15.28:0/465406013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 msgr2=0x55af5f3f8590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:02:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/465406013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 -- 172.21.15.28:0/465406013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 msgr2=0x55af5f3e0090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T22:02:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/465406013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3e0090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.197+0000 7fb8a55c0640 1 -- 172.21.15.28:0/465406013 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af5f3d7120 con 0x55af5f3d45d0 2024-09-15T22:02:03.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.198+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/465406013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3dfb50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb89c03f040 tx=0x7fb89c009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40c97f43565f5b6b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.200+0000 7fb88affd640 1 -- 172.21.15.28:0/465406013 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c035070 con 0x55af5f3d45d0 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.200+0000 7fb88affd640 1 -- 172.21.15.28:0/465406013 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb89c04e070 con 0x55af5f3d45d0 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.201+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/465406013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 msgr2=0x55af5f3dfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.201+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/465406013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3dfb50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb89c03f040 tx=0x7fb89c009680 comp rx=0 tx=0).stop 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.201+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/465406013 shutdown_connections 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.201+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/465406013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.201+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/465406013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3e0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.201+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/465406013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3d45d0 0x55af5f3dfb50 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb89c03f040 tx=0x7fb89c009680 comp rx=0 tx=0).stop 2024-09-15T22:02:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.201+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/465406013 >> 172.21.15.28:0/465406013 conn(0x55af5f39d310 msgr2=0x55af5f3e2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.207+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/465406013 shutdown_connections 2024-09-15T22:02:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.207+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/465406013 wait complete. 2024-09-15T22:02:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.208+0000 7fb8a9bfd740 1 -- start start 2024-09-15T22:02:03.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.208+0000 7fb8a9bfd740 1 Processor -- start 2024-09-15T22:02:03.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3dad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a9bfd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3f8ad0 0x55af5f456720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a01ae0d0 con 0x55af5f3d5b40 2024-09-15T22:02:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a01ada00 con 0x55af5f3f8ad0 2024-09-15T22:02:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a9bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a01aac40 con 0x55af5f3e3780 2024-09-15T22:02:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a9bfd740 10 client.0 _ll_register_callbacks cb 0x7fb89c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a4dbf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a5dc1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3f8ad0 0x55af5f456720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a4dbf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:41314/0 (socket says 172.21.15.28:41314) 2024-09-15T22:02:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.209+0000 7fb8a4dbf640 1 -- 172.21.15.28:0/2111473457 learned_addr learned my addr 172.21.15.28:0/2111473457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.210+0000 7fb8a5dc1640 1 -- 172.21.15.28:0/2111473457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3e3780 msgr2=0x55af5f3f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.210+0000 7fb8a5dc1640 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.210+0000 7fb8a5dc1640 1 -- 172.21.15.28:0/2111473457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 msgr2=0x55af5f3dad70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T22:02:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.210+0000 7fb8a5dc1640 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3dad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.210+0000 7fb8a5dc1640 1 -- 172.21.15.28:0/2111473457 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af5f3f8f50 con 0x55af5f3f8ad0 2024-09-15T22:02:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.210+0000 7fb8a5dc1640 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3f8ad0 0x55af5f456720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55af5f3d5070 tx=0x7fb898012b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74475ce7212ccf4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.210+0000 7fb888ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:02:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.213+0000 7fb888ff9640 1 -- 172.21.15.28:0/2111473457 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898003b80 con 0x55af5f3f8ad0 2024-09-15T22:02:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.213+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb89c04e070 con 0x55af5f3f8ad0 2024-09-15T22:02:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.213+0000 7fb888ff9640 1 -- 172.21.15.28:0/2111473457 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fb898003d20 con 0x55af5f3f8ad0 2024-09-15T22:02:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.213+0000 7fb888ff9640 1 -- 172.21.15.28:0/2111473457 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb898004030 con 0x55af5f3f8ad0 2024-09-15T22:02:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.214+0000 7fb8a9bfd740 10 client.25862 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.214+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55af5f473570 con 0x55af5f3f8ad0 2024-09-15T22:02:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.214+0000 7fb888ff9640 1 -- 172.21.15.28:0/2111473457 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb89c04e070 con 0x55af5f3f8ad0 2024-09-15T22:02:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.214+0000 7fb888ff9640 1 -- 172.21.15.28:0/2111473457 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fb89800f040 con 0x55af5f3f8ad0 2024-09-15T22:02:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.215+0000 7fb8a9bfd740 10 client.25862 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.215+0000 7fb8a9bfd740 10 client.25862 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:02:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.215+0000 7fb8a9bfd740 20 client.25862 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.215+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f473cf0 0x55af5f476110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:03.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.216+0000 7fb8a9bfd740 4 client.25862 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:02:03.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.216+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55af5f341d70 con 0x55af5f473cf0 2024-09-15T22:02:03.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.216+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f473cf0 0x55af5f476110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:03.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.216+0000 7fb8a55c0640 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f473cf0 0x55af5f476110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9796547cbeb2ca2b server_cookie=8521de0bd06c65ed in_seq=0 out_seq=0 2024-09-15T22:02:03.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.217+0000 7fb888ff9640 10 client.25862 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:02:03.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.217+0000 7fb888ff9640 1 -- 172.21.15.28:0/2111473457 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55af5f341d70 con 0x55af5f473cf0 2024-09-15T22:02:03.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.217+0000 7fb888ff9640 10 client.25862 handle_command_reply: tid=0 2024-09-15T22:02:03.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.217+0000 7fb8a9bfd740 1 client.25862 shutdown 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3f8ad0 msgr2=0x55af5f456720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3f8ad0 0x55af5f456720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55af5f3d5070 tx=0x7fb898012b00 comp rx=0 tx=0).stop 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 shutdown_connections 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55af5f473cf0 0x55af5f476110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55af5f3f8ad0 0x55af5f456720 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af5f3e3780 0x55af5f3f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 --2- 172.21.15.28:0/2111473457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55af5f3d5b40 0x55af5f3dad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 >> 172.21.15.28:0/2111473457 conn(0x55af5f39d310 msgr2=0x55af5f3d6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb888ff9640 0 client.25862 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:02:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 shutdown_connections 2024-09-15T22:02:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.221+0000 7fb8a9bfd740 1 -- 172.21.15.28:0/2111473457 wait complete. 2024-09-15T22:02:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.222+0000 7fb8a9bfd740 20 client.25862 trim_cache size 0 max 16384 2024-09-15T22:02:03.224 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb88c0a3430 msgr2=0x7fb88c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb88c0a3430 0x7fb88c0a58f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb89c006010 tx=0x7fb89c000bc0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0101420 msgr2=0x7fb8a0198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0101420 0x7fb8a0198940 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e9f0 tx=0x7fb89000eeb0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 shutdown_connections 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb88c0a3430 0x7fb88c0a58f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8a0106a80 0x7fb8a019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8a0101420 0x7fb8a0198940 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 --2- 172.21.15.28:0/1959254855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8a0100a80 0x7fb8a0198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 >> 172.21.15.28:0/1959254855 conn(0x7fb8a00fc7c0 msgr2=0x7fb8a0104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 shutdown_connections 2024-09-15T22:02:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:03.223+0000 7fb8a7824640 1 -- 172.21.15.28:0/1959254855 wait complete. 2024-09-15T22:02:03.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:03 smithi174 ceph-mon[35244]: pgmap v144: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 42 MiB/s wr, 6.34k op/s 2024-09-15T22:02:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:03 smithi028 ceph-mon[30969]: pgmap v144: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 42 MiB/s wr, 6.34k op/s 2024-09-15T22:02:03.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:03 smithi203 ceph-mon[35546]: pgmap v144: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 42 MiB/s wr, 6.34k op/s 2024-09-15T22:02:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:05 smithi174 ceph-mon[35244]: pgmap v145: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 29 MiB/s wr, 4.45k op/s 2024-09-15T22:02:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:05 smithi028 ceph-mon[30969]: pgmap v145: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 29 MiB/s wr, 4.45k op/s 2024-09-15T22:02:05.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:05 smithi203 ceph-mon[35546]: pgmap v145: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 29 MiB/s wr, 4.45k op/s 2024-09-15T22:02:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:07 smithi174 ceph-mon[35244]: pgmap v146: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 29 MiB/s wr, 4.49k op/s 2024-09-15T22:02:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:07 smithi028 ceph-mon[30969]: pgmap v146: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 29 MiB/s wr, 4.49k op/s 2024-09-15T22:02:07.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:07 smithi203 ceph-mon[35546]: pgmap v146: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 29 MiB/s wr, 4.49k op/s 2024-09-15T22:02:09.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:09 smithi174 ceph-mon[35244]: pgmap v147: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 34 MiB/s wr, 5.46k op/s 2024-09-15T22:02:09.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:09 smithi028 ceph-mon[30969]: pgmap v147: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 34 MiB/s wr, 5.46k op/s 2024-09-15T22:02:09.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:09 smithi203 ceph-mon[35546]: pgmap v147: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 34 MiB/s wr, 5.46k op/s 2024-09-15T22:02:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f25dbfff640 1 -- 172.21.15.28:0/1313885182 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559af0c5bd70 con 0x559af0d0f100 2024-09-15T22:02:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f25dbfff640 10 client.16287 handle_command_reply: tid=0 2024-09-15T22:02:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 client.16287 shutdown 2024-09-15T22:02:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 msgr2=0x559af0d07060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0d07060 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25e801b3f0 tx=0x7f25e8013160 comp rx=0 tx=0).stop 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 shutdown_connections 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559af0d0f100 0x559af0d91cf0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559af0cff0c0 0x559af0d075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559af0cf02b0 0x559af0d07060 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 --2- 172.21.15.28:0/1313885182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559af0cef990 0x559af0cf6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 >> 172.21.15.28:0/1313885182 conn(0x559af0cec060 msgr2=0x559af0cea9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f25dbfff640 0 client.16287 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 shutdown_connections 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.393+0000 7f26058b1740 1 -- 172.21.15.28:0/1313885182 wait complete. 2024-09-15T22:02:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f26058b1740 20 client.16287 trim_cache size 0 max 16384 2024-09-15T22:02:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:02:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:02:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:02:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:02:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25ec0a3460 msgr2=0x7f25ec0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25ec0a3460 0x7f25ec0a5920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f25f4006010 tx=0x7f25f4009800 comp rx=0 tx=0).stop 2024-09-15T22:02:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1009b0 msgr2=0x7f25fc197210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1009b0 0x7f25fc197210 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f25f000eb00 tx=0x7f25f0009f20 comp rx=0 tx=0).stop 2024-09-15T22:02:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 shutdown_connections 2024-09-15T22:02:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25ec0a3460 0x7f25ec0a5920 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25fc1069c0 0x7f25fc19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25fc101350 0x7f25fc197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 --2- 172.21.15.28:0/223050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25fc1009b0 0x7f25fc197210 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.394+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 >> 172.21.15.28:0/223050003 conn(0x7f25fc077990 msgr2=0x7f25fc103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.395+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 shutdown_connections 2024-09-15T22:02:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:10.395+0000 7f2602b18640 1 -- 172.21.15.28:0/223050003 wait complete. 2024-09-15T22:02:11.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:11 smithi174 ceph-mon[35244]: pgmap v148: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 26 MiB/s wr, 4.36k op/s 2024-09-15T22:02:11.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:11 smithi028 ceph-mon[30969]: pgmap v148: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 26 MiB/s wr, 4.36k op/s 2024-09-15T22:02:11.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:11 smithi203 ceph-mon[35546]: pgmap v148: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 26 MiB/s wr, 4.36k op/s 2024-09-15T22:02:13.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:13 smithi174 ceph-mon[35244]: pgmap v149: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 27 MiB/s wr, 4.68k op/s 2024-09-15T22:02:13.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:13 smithi028 ceph-mon[30969]: pgmap v149: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 27 MiB/s wr, 4.68k op/s 2024-09-15T22:02:13.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:13 smithi203 ceph-mon[35546]: pgmap v149: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 27 MiB/s wr, 4.68k op/s 2024-09-15T22:02:15.082 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-15T22:02:15.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-15T22:02:15.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:15 smithi174 ceph-mon[35244]: pgmap v150: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 22 MiB/s wr, 3.96k op/s 2024-09-15T22:02:15.768 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:15 smithi203 ceph-mon[35546]: pgmap v150: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 22 MiB/s wr, 3.96k op/s 2024-09-15T22:02:15.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:15 smithi028 ceph-mon[30969]: pgmap v150: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 22 MiB/s wr, 3.96k op/s 2024-09-15T22:02:17.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:17 smithi174 ceph-mon[35244]: pgmap v151: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 23 MiB/s wr, 4.14k op/s 2024-09-15T22:02:17.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:17 smithi028 ceph-mon[30969]: pgmap v151: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 23 MiB/s wr, 4.14k op/s 2024-09-15T22:02:17.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:17 smithi203 ceph-mon[35546]: pgmap v151: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 23 MiB/s wr, 4.14k op/s 2024-09-15T22:02:19.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:19 smithi028 ceph-mon[30969]: pgmap v152: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 33 MiB/s wr, 5.79k op/s 2024-09-15T22:02:19.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:19 smithi203 ceph-mon[35546]: pgmap v152: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 33 MiB/s wr, 5.79k op/s 2024-09-15T22:02:20.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:19 smithi174 ceph-mon[35244]: pgmap v152: 129 pgs: 129 active+clean; 16 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 33 MiB/s wr, 5.79k op/s 2024-09-15T22:02:21.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:21 smithi028 ceph-mon[30969]: pgmap v153: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 28 MiB/s wr, 4.84k op/s 2024-09-15T22:02:21.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:21 smithi203 ceph-mon[35546]: pgmap v153: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 28 MiB/s wr, 4.84k op/s 2024-09-15T22:02:22.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:21 smithi174 ceph-mon[35244]: pgmap v153: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 28 MiB/s wr, 4.84k op/s 2024-09-15T22:02:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 -- 172.21.15.28:0/1520517598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 msgr2=0x7f9a24106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 --2- 172.21.15.28:0/1520517598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a24106da0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009930 tx=0x7f9a1402a420 comp rx=0 tx=0).stop 2024-09-15T22:02:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 -- 172.21.15.28:0/1520517598 shutdown_connections 2024-09-15T22:02:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 --2- 172.21.15.28:0/1520517598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a24101350 0x7f9a2410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 --2- 172.21.15.28:0/1520517598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a241009b0 0x7f9a24100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 --2- 172.21.15.28:0/1520517598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a24106da0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 -- 172.21.15.28:0/1520517598 >> 172.21.15.28:0/1520517598 conn(0x7f9a24077990 msgr2=0x7f9a240fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 -- 172.21.15.28:0/1520517598 shutdown_connections 2024-09-15T22:02:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 -- 172.21.15.28:0/1520517598 wait complete. 2024-09-15T22:02:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.526+0000 7f9a2933b640 1 Processor -- start 2024-09-15T22:02:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a2933b640 1 -- start start 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a2933b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a241009b0 0x7f9a240fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a2933b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a24101350 0x7f9a240fd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a2933b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a240fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a2933b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a24110e00 con 0x7f9a241069c0 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a2933b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a24110c80 con 0x7f9a24101350 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a2933b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a24110f80 con 0x7f9a241009b0 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a240fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a240fd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53482/0 (socket says 172.21.15.28:53482) 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 -- 172.21.15.28:0/4260510472 learned_addr learned my addr 172.21.15.28:0/4260510472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a227fc640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a24101350 0x7f9a240fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a241009b0 0x7f9a240fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 -- 172.21.15.28:0/4260510472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a241009b0 msgr2=0x7f9a240fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a241009b0 0x7f9a240fcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 -- 172.21.15.28:0/4260510472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a24101350 msgr2=0x7f9a240fd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a24101350 0x7f9a240fd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 -- 172.21.15.28:0/4260510472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a241a68d0 con 0x7f9a241069c0 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a241009b0 0x7f9a240fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.527+0000 7f9a237fe640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a240fd860 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00eb10 tx=0x7f9a1c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.528+0000 7f9a0bfff640 1 -- 172.21.15.28:0/4260510472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c00ce20 con 0x7f9a241069c0 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.528+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a241a6bc0 con 0x7f9a241069c0 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.528+0000 7f9a0bfff640 1 -- 172.21.15.28:0/4260510472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1c005020 con 0x7f9a241069c0 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.528+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a241a70c0 con 0x7f9a241069c0 2024-09-15T22:02:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.528+0000 7f9a0bfff640 1 -- 172.21.15.28:0/4260510472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c010690 con 0x7f9a241069c0 2024-09-15T22:02:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.530+0000 7f9a0bfff640 1 -- 172.21.15.28:0/4260510472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f9a1c005540 con 0x7f9a241069c0 2024-09-15T22:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.530+0000 7f9a0bfff640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9a0c0a3460 0x7f9a0c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.531+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9a0c0a3460 0x7f9a0c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.531+0000 7f9a0bfff640 1 -- 172.21.15.28:0/4260510472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9a1c0d06b0 con 0x7f9a241069c0 2024-09-15T22:02:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.531+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9a0c0a3460 0x7f9a0c0a5920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000c00 tx=0x7f9a14005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.549+0000 7f9a2c0d5740 1 Processor -- start 2024-09-15T22:02:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.549+0000 7f9a2c0d5740 1 -- start start 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e758ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e763770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e763cb0 0x55ae7e7660a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae7e6bd2f0 con 0x55ae7e758af0 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ae7e665270 con 0x55ae7e763cb0 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ae7e6b92b0 con 0x55ae7e759410 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e758ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e758ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53488/0 (socket says 172.21.15.28:53488) 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3088501258 learned_addr learned my addr 172.21.15.28:0/3088501258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a227fc640 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e763770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a237fe640 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e763cb0 0x55ae7e7660a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3088501258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 msgr2=0x55ae7e763770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e763770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3088501258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e763cb0 msgr2=0x55ae7e7660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e763cb0 0x55ae7e7660a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3088501258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae7e664340 con 0x55ae7e758af0 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.550+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e758ed0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009930 tx=0x7f9a1403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949f575affc5354c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a097fa640 1 -- 172.21.15.28:0/3088501258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14035050 con 0x55ae7e758af0 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a097fa640 1 -- 172.21.15.28:0/3088501258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1404c070 con 0x55ae7e758af0 2024-09-15T22:02:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a097fa640 1 -- 172.21.15.28:0/3088501258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14004b40 con 0x55ae7e758af0 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3088501258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 msgr2=0x55ae7e758ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e758ed0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009930 tx=0x7f9a1403c000 comp rx=0 tx=0).stop 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3088501258 shutdown_connections 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e763cb0 0x55ae7e7660a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e763770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3088501258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e758ed0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3088501258 >> 172.21.15.28:0/3088501258 conn(0x55ae7e755060 msgr2=0x55ae7e755440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3088501258 shutdown_connections 2024-09-15T22:02:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.551+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3088501258 wait complete. 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 -- start start 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 Processor -- start 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e760ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e759410 0x55ae7e7610e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e761620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae7e766880 con 0x55ae7e759410 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ae7e646290 con 0x55ae7e758af0 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ae7e766a00 con 0x55ae7e763cb0 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a2c0d5740 10 client.0 _ll_register_callbacks cb 0x55ae7e677140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e759410 0x55ae7e7610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e759410 0x55ae7e7610e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53490/0 (socket says 172.21.15.28:53490) 2024-09-15T22:02:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a227fc640 1 -- 172.21.15.28:0/3376662348 learned_addr learned my addr 172.21.15.28:0/3376662348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a237fe640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e761620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.552+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e760ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a227fc640 1 -- 172.21.15.28:0/3376662348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 msgr2=0x55ae7e761620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a227fc640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e761620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a227fc640 1 -- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 msgr2=0x55ae7e760ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a227fc640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e760ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a227fc640 1 -- 172.21.15.28:0/3376662348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae7e3d0210 con 0x55ae7e759410 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a237fe640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e761620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e760ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a227fc640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e759410 0x55ae7e7610e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9a18011570 tx=0x7f9a18009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24838d1a5f3bc044 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:02:23.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3376662348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a18013670 con 0x55ae7e759410 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3376662348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9a18013810 con 0x55ae7e759410 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a1404c070 con 0x55ae7e759410 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3376662348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9a18026050 con 0x55ae7e759410 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a2c0d5740 10 client.16317 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ae7e49c620 con 0x55ae7e759410 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.553+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3376662348 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1404c070 con 0x55ae7e759410 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3376662348 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9a18014440 con 0x55ae7e759410 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a2c0d5740 10 client.16317 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a2c0d5740 10 client.16317 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a2c0d5740 20 client.16317 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7760c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a2c0d5740 4 client.16317 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae7e6caa20 con 0x55ae7e773cb0 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7760c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e9ed3221ff41089 server_cookie=d852eb70f67756b in_seq=0 out_seq=0 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.554+0000 7f9a06ff5640 10 client.16317 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3376662348 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ae7e6caa20 con 0x55ae7e773cb0 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a06ff5640 10 client.16317 handle_command_reply: tid=0 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 client.16317 shutdown 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e759410 msgr2=0x55ae7e7610e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e759410 0x55ae7e7610e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9a18011570 tx=0x7f9a18009f90 comp rx=0 tx=0).stop 2024-09-15T22:02:23.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 shutdown_connections 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7760c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a06ff5640 0 client.16317 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e761620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e759410 0x55ae7e7610e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3376662348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e760ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 >> 172.21.15.28:0/3376662348 conn(0x55ae7e755060 msgr2=0x55ae7e7541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 shutdown_connections 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.555+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3376662348 wait complete. 2024-09-15T22:02:23.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.556+0000 7f9a2c0d5740 20 client.16317 trim_cache size 0 max 16384 2024-09-15T22:02:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 Processor -- start 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 -- start start 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e770240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e755440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e763cb0 0x55ae7e761450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a241b1020 con 0x55ae7e763cb0 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a241b2b60 con 0x55ae7e758af0 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.571+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a241b16f0 con 0x55ae7e759410 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e763cb0 0x55ae7e761450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e763cb0 0x55ae7e761450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53498/0 (socket says 172.21.15.28:53498) 2024-09-15T22:02:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 -- 172.21.15.28:0/2919578838 learned_addr learned my addr 172.21.15.28:0/2919578838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a227fc640 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e755440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e770240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 -- 172.21.15.28:0/2919578838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 msgr2=0x55ae7e755440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e755440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 -- 172.21.15.28:0/2919578838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 msgr2=0x55ae7e770240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e770240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 -- 172.21.15.28:0/2919578838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae7e75fb30 con 0x55ae7e763cb0 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e770240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a237fe640 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e763cb0 0x55ae7e761450 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0ce0a0 tx=0x7f9a1c005290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d595559129ab4d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a08ff9640 1 -- 172.21.15.28:0/2919578838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c0a09e0 con 0x55ae7e763cb0 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a08ff9640 1 -- 172.21.15.28:0/2919578838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1c0ba070 con 0x55ae7e763cb0 2024-09-15T22:02:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.572+0000 7f9a08ff9640 1 -- 172.21.15.28:0/2919578838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c09caf0 con 0x55ae7e763cb0 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/2919578838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e763cb0 msgr2=0x55ae7e761450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e763cb0 0x55ae7e761450 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0ce0a0 tx=0x7f9a1c005290 comp rx=0 tx=0).stop 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/2919578838 shutdown_connections 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e763cb0 0x55ae7e761450 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e759410 0x55ae7e755440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/2919578838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e758af0 0x55ae7e770240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/2919578838 >> 172.21.15.28:0/2919578838 conn(0x55ae7e755060 msgr2=0x55ae7e762c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/2919578838 shutdown_connections 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/2919578838 wait complete. 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 -- start start 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.573+0000 7f9a2c0d5740 1 Processor -- start 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e76c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e759410 0x55ae7e767610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e767b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a241b2490 con 0x55ae7e758af0 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a241b1dc0 con 0x55ae7e759410 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a241af000 con 0x55ae7e763cb0 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 10 client.0 _ll_register_callbacks cb 0x7f9a18009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e76c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e76c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53504/0 (socket says 172.21.15.28:53504) 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a227fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e759410 0x55ae7e767610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3159175155 learned_addr learned my addr 172.21.15.28:0/3159175155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a237fe640 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e767b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3159175155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 msgr2=0x55ae7e767b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e767b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e759410 msgr2=0x55ae7e767610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e759410 0x55ae7e767610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 -- 172.21.15.28:0/3159175155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae7e768380 con 0x55ae7e758af0 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a227fc640 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e759410 0x55ae7e767610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a22ffd640 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e76c2f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9a14004730 tx=0x7f9a14004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b967234a5ee21e67 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3159175155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14035070 con 0x55ae7e758af0 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3159175155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9a14003a00 con 0x55ae7e758af0 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.574+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a1c0ba070 con 0x55ae7e758af0 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3159175155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9a14003c00 con 0x55ae7e758af0 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 10 client.16323 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ae7e7de920 con 0x55ae7e758af0 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3159175155 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1c0ba070 con 0x55ae7e758af0 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3159175155 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9a14047470 con 0x55ae7e758af0 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 10 client.16323 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 10 client.16323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 20 client.16323 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 4 client.16323 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae7e6c4d70 con 0x55ae7e773cb0 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.575+0000 7f9a227fc640 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7e0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.576+0000 7f9a06ff5640 10 client.16323 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:23.576+0000 7f9a227fc640 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7e0c70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3369e2b20bc6e09 server_cookie=f0abad386b47c4d6 in_seq=0 out_seq=0 2024-09-15T22:02:23.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:23 smithi028 ceph-mon[30969]: pgmap v154: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 33 MiB/s wr, 5.68k op/s 2024-09-15T22:02:23.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:23 smithi203 ceph-mon[35546]: pgmap v154: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 33 MiB/s wr, 5.68k op/s 2024-09-15T22:02:24.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:23 smithi174 ceph-mon[35244]: pgmap v154: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 33 MiB/s wr, 5.68k op/s 2024-09-15T22:02:25.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:02:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:02:25.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:25 smithi028 ceph-mon[30969]: pgmap v155: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 35 MiB/s wr, 5.76k op/s 2024-09-15T22:02:25.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:25 smithi203 ceph-mon[35546]: pgmap v155: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 35 MiB/s wr, 5.76k op/s 2024-09-15T22:02:26.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:25 smithi174 ceph-mon[35244]: pgmap v155: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 35 MiB/s wr, 5.76k op/s 2024-09-15T22:02:27.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:27 smithi028 ceph-mon[30969]: pgmap v156: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 34 MiB/s wr, 5.61k op/s 2024-09-15T22:02:27.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:27 smithi203 ceph-mon[35546]: pgmap v156: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 34 MiB/s wr, 5.61k op/s 2024-09-15T22:02:28.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:27 smithi174 ceph-mon[35244]: pgmap v156: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 34 MiB/s wr, 5.61k op/s 2024-09-15T22:02:29.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:29 smithi028 ceph-mon[30969]: pgmap v157: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 37 MiB/s wr, 6.18k op/s 2024-09-15T22:02:29.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:29 smithi203 ceph-mon[35546]: pgmap v157: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 37 MiB/s wr, 6.18k op/s 2024-09-15T22:02:30.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:29 smithi174 ceph-mon[35244]: pgmap v157: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 37 MiB/s wr, 6.18k op/s 2024-09-15T22:02:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a06ff5640 1 -- 172.21.15.28:0/3159175155 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ae7e6c4d70 con 0x55ae7e773cb0 2024-09-15T22:02:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a06ff5640 10 client.16323 handle_command_reply: tid=0 2024-09-15T22:02:31.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 client.16323 shutdown 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 msgr2=0x55ae7e76c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e76c2f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9a14004730 tx=0x7f9a14004800 comp rx=0 tx=0).stop 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 shutdown_connections 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ae7e773cb0 0x55ae7e7e0c70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ae7e763cb0 0x55ae7e767b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae7e759410 0x55ae7e767610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 --2- 172.21.15.28:0/3159175155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae7e758af0 0x55ae7e76c2f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 >> 172.21.15.28:0/3159175155 conn(0x55ae7e755060 msgr2=0x55ae7e7539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a06ff5640 0 client.16323 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.896+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 shutdown_connections 2024-09-15T22:02:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.897+0000 7f9a2c0d5740 1 -- 172.21.15.28:0/3159175155 wait complete. 2024-09-15T22:02:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.897+0000 7f9a2c0d5740 20 client.16323 trim_cache size 0 max 16384 2024-09-15T22:02:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:02:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:02:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:02:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9a0c0a3460 msgr2=0x7f9a0c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9a0c0a3460 0x7f9a0c0a5920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000c00 tx=0x7f9a14005a20 comp rx=0 tx=0).stop 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 msgr2=0x7f9a240fd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a240fd860 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00eb10 tx=0x7f9a1c00ef10 comp rx=0 tx=0).stop 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 shutdown_connections 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9a0c0a3460 0x7f9a0c0a5920 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a241069c0 0x7f9a240fd860 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a24101350 0x7f9a240fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 --2- 172.21.15.28:0/4260510472 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a241009b0 0x7f9a240fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 >> 172.21.15.28:0/4260510472 conn(0x7f9a24077990 msgr2=0x7f9a240699e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:31.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 shutdown_connections 2024-09-15T22:02:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:31.898+0000 7f9a2933b640 1 -- 172.21.15.28:0/4260510472 wait complete. 2024-09-15T22:02:31.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:31 smithi028 ceph-mon[30969]: pgmap v158: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 MiB/s wr, 4.59k op/s 2024-09-15T22:02:31.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:31 smithi203 ceph-mon[35546]: pgmap v158: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 MiB/s wr, 4.59k op/s 2024-09-15T22:02:32.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:31 smithi174 ceph-mon[35244]: pgmap v158: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 27 MiB/s wr, 4.59k op/s 2024-09-15T22:02:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.690+0000 7f92919fd640 1 -- 172.21.15.28:0/3035906377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f928c112ba0 msgr2=0x7f928c114f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.690+0000 7f92919fd640 1 --2- 172.21.15.28:0/3035906377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f928c112ba0 0x7f928c114f90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f927c009900 tx=0x7f927c02a4b0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.691+0000 7f92919fd640 1 -- 172.21.15.28:0/3035906377 shutdown_connections 2024-09-15T22:02:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.691+0000 7f92919fd640 1 --2- 172.21.15.28:0/3035906377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f928c112ba0 0x7f928c114f90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.691+0000 7f92919fd640 1 --2- 172.21.15.28:0/3035906377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f928c074eb0 0x7f928c112660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.691+0000 7f92919fd640 1 --2- 172.21.15.28:0/3035906377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f928c074590 0x7f928c074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.691+0000 7f92919fd640 1 -- 172.21.15.28:0/3035906377 >> 172.21.15.28:0/3035906377 conn(0x7f928c0fd8f0 msgr2=0x7f928c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.691+0000 7f92919fd640 1 -- 172.21.15.28:0/3035906377 shutdown_connections 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.691+0000 7f92919fd640 1 -- 172.21.15.28:0/3035906377 wait complete. 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.692+0000 7f92919fd640 1 Processor -- start 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.692+0000 7f92919fd640 1 -- start start 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.692+0000 7f92919fd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f928c074590 0x7f928c0739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f92919fd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f928c074eb0 0x7f928c06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f92919fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f928c112ba0 0x7f928c06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f92919fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f928c1185d0 con 0x7f928c112ba0 2024-09-15T22:02:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f92919fd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f928c118450 con 0x7f928c074eb0 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f92919fd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f928c118750 con 0x7f928c074590 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f928a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f928c074eb0 0x7f928c06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f928affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f928c074590 0x7f928c0739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f928affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f928c074590 0x7f928c0739c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:57878/0 (socket says 172.21.15.28:57878) 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f928a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f928c074eb0 0x7f928c06ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:43398/0 (socket says 172.21.15.28:43398) 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.693+0000 7f928affd640 1 -- 172.21.15.28:0/1890268898 learned_addr learned my addr 172.21.15.28:0/1890268898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.694+0000 7f928a7fc640 1 -- 172.21.15.28:0/1890268898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f928c074590 msgr2=0x7f928c0739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.694+0000 7f928a7fc640 1 --2- 172.21.15.28:0/1890268898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f928c074590 0x7f928c0739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.694+0000 7f928a7fc640 1 -- 172.21.15.28:0/1890268898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f928c112ba0 msgr2=0x7f928c06cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.694+0000 7f928a7fc640 1 --2- 172.21.15.28:0/1890268898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f928c112ba0 0x7f928c06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.694+0000 7f928a7fc640 1 -- 172.21.15.28:0/1890268898 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928c06d550 con 0x7f928c074eb0 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.694+0000 7f928affd640 1 --2- 172.21.15.28:0/1890268898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f928c074590 0x7f928c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.694+0000 7f928a7fc640 1 --2- 172.21.15.28:0/1890268898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f928c074eb0 0x7f928c06ca50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f928400eb00 tx=0x7f928400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.696+0000 7f92909fb640 1 -- 172.21.15.28:0/1890268898 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284010430 con 0x7f928c074eb0 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.696+0000 7f92919fd640 1 -- 172.21.15.28:0/1890268898 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f928c06d840 con 0x7f928c074eb0 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.696+0000 7f92919fd640 1 -- 172.21.15.28:0/1890268898 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f928c071810 con 0x7f928c074eb0 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.696+0000 7f92909fb640 1 -- 172.21.15.28:0/1890268898 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9284010ab0 con 0x7f928c074eb0 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.696+0000 7f92909fb640 1 -- 172.21.15.28:0/1890268898 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284017b70 con 0x7f928c074eb0 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 Processor -- start 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 -- start start 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb52850 0x55bf6bb52c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb53170 0x55bf6bb5d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f92909fb640 1 -- 172.21.15.28:0/1890268898 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f9284017d10 con 0x7f928c074eb0 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf6bab7160 con 0x55bf6bb52850 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bf6ba5e720 con 0x55bf6bb53170 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.701+0000 7f9293dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bf6bab3120 con 0x55bf6bb5da30 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:57886/0 (socket says 172.21.15.28:57886) 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 -- 172.21.15.28:0/2220506665 learned_addr learned my addr 172.21.15.28:0/2220506665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928affd640 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb53170 0x55bf6bb5d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 -- 172.21.15.28:0/2220506665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb53170 msgr2=0x55bf6bb5d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb53170 0x55bf6bb5d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 -- 172.21.15.28:0/2220506665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb52850 msgr2=0x55bf6bb52c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb52850 0x55bf6bb52c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.702+0000 7f928a7fc640 1 -- 172.21.15.28:0/2220506665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf6b828ba0 con 0x55bf6bb5da30 2024-09-15T22:02:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.703+0000 7f928a7fc640 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5fe20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f92840cdbc0 tx=0x7f92840d6ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e95fbf14f478bc97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.703+0000 7f9271ffb640 1 -- 172.21.15.28:0/2220506665 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92840df070 con 0x55bf6bb5da30 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.703+0000 7f9271ffb640 1 -- 172.21.15.28:0/2220506665 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92840d3e50 con 0x55bf6bb5da30 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f92909fb640 1 --2- 172.21.15.28:0/1890268898 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f92740a35c0 0x7f92740a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f92909fb640 1 -- 172.21.15.28:0/1890268898 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f92840d08f0 con 0x7f928c074eb0 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f9293dd6740 1 -- 172.21.15.28:0/2220506665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 msgr2=0x55bf6bb5fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f9293dd6740 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5fe20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f92840cdbc0 tx=0x7f92840d6ef0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f9293dd6740 1 -- 172.21.15.28:0/2220506665 shutdown_connections 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f9293dd6740 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5fe20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f9293dd6740 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb53170 0x55bf6bb5d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f9293dd6740 1 --2- 172.21.15.28:0/2220506665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb52850 0x55bf6bb52c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f9293dd6740 1 -- 172.21.15.28:0/2220506665 >> 172.21.15.28:0/2220506665 conn(0x55bf6bb4ede0 msgr2=0x55bf6bb4f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.704+0000 7f928b7fe640 1 --2- 172.21.15.28:0/1890268898 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f92740a35c0 0x7f92740a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.705+0000 7f9293dd6740 1 -- 172.21.15.28:0/2220506665 shutdown_connections 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.705+0000 7f9293dd6740 1 -- 172.21.15.28:0/2220506665 wait complete. 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.705+0000 7f9293dd6740 1 -- start start 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 1 Processor -- start 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f928b7fe640 1 --2- 172.21.15.28:0/1890268898 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f92740a35c0 0x7f92740a5a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f927c009900 tx=0x7f927c0056d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb5d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb5a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf6bb606a0 con 0x55bf6bb53170 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bf6ba3f900 con 0x55bf6bb52850 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bf6bb60820 con 0x55bf6bb5da30 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.706+0000 7f9293dd6740 10 client.0 _ll_register_callbacks cb 0x55bf6ba3d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.707+0000 7f928b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.707+0000 7f928b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:57894/0 (socket says 172.21.15.28:57894) 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.707+0000 7f928b7fe640 1 -- 172.21.15.28:0/1523562090 learned_addr learned my addr 172.21.15.28:0/1523562090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.707+0000 7f928affd640 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb5d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.708+0000 7f928b7fe640 1 -- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 msgr2=0x55bf6bb5d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.708+0000 7f928b7fe640 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb5d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.708+0000 7f928b7fe640 1 -- 172.21.15.28:0/1523562090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 msgr2=0x55bf6bb5a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.708+0000 7f928b7fe640 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb5a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.708+0000 7f928b7fe640 1 -- 172.21.15.28:0/1523562090 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf6ba7fc50 con 0x55bf6bb5da30 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.709+0000 7f928b7fe640 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5af20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f927c033000 tx=0x7f927c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e048d02c9796aa8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.709+0000 7f926affd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.709+0000 7f926affd640 1 -- 172.21.15.28:0/1523562090 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c040070 con 0x55bf6bb5da30 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.710+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92840d3e50 con 0x55bf6bb5da30 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.710+0000 7f926affd640 1 -- 172.21.15.28:0/1523562090 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f927c0315b0 con 0x55bf6bb5da30 2024-09-15T22:02:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.710+0000 7f926affd640 1 -- 172.21.15.28:0/1523562090 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f927c055050 con 0x55bf6bb5da30 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 10 client.25849 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bf6bb6c330 con 0x55bf6bb5da30 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f926affd640 1 -- 172.21.15.28:0/1523562090 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92840d3e50 con 0x55bf6bb5da30 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f926affd640 1 -- 172.21.15.28:0/1523562090 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f927c047b00 con 0x55bf6bb5da30 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 10 client.25849 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 10 client.25849 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 20 client.25849 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bb6d800 0x55bf6bb6fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 4 client.25849 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.711+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf6bac4890 con 0x55bf6bb6d800 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.712+0000 7f928affd640 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bb6d800 0x55bf6bb6fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.713+0000 7f926affd640 10 client.25849 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.713+0000 7f928affd640 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bb6d800 0x55bf6bb6fca0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fd262fedb7e5c6 server_cookie=27179ded62a498e7 in_seq=0 out_seq=0 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.714+0000 7f926affd640 1 -- 172.21.15.28:0/1523562090 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bf6bac4890 con 0x55bf6bb6d800 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.714+0000 7f926affd640 10 client.25849 handle_command_reply: tid=0 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.714+0000 7f9293dd6740 1 client.25849 shutdown 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 msgr2=0x55bf6bb5af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5af20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f927c033000 tx=0x7f927c005fd0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 shutdown_connections 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bb6d800 0x55bf6bb6fca0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb5af20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb5a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 --2- 172.21.15.28:0/1523562090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb5d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 >> 172.21.15.28:0/1523562090 conn(0x55bf6bb4ede0 msgr2=0x55bf6bb54b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f926affd640 0 client.25849 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 shutdown_connections 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 1 -- 172.21.15.28:0/1523562090 wait complete. 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.715+0000 7f9293dd6740 20 client.25849 trim_cache size 0 max 16384 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 Processor -- start 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 -- start start 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb52850 0x55bf6bb49f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb75570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb5da30 0x55bf6bb75ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f927800fe30 con 0x55bf6bb53170 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9278011970 con 0x55bf6bb5da30 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.749+0000 7f9293dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9278010500 con 0x55bf6bb52850 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb75570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb75570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40424/0 (socket says 172.21.15.28:40424) 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 -- 172.21.15.28:0/687438950 learned_addr learned my addr 172.21.15.28:0/687438950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928b7fe640 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb5da30 0x55bf6bb75ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928affd640 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb52850 0x55bf6bb49f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 -- 172.21.15.28:0/687438950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb52850 msgr2=0x55bf6bb49f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb52850 0x55bf6bb49f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 -- 172.21.15.28:0/687438950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb5da30 msgr2=0x55bf6bb75ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb5da30 0x55bf6bb75ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.750+0000 7f928a7fc640 1 -- 172.21.15.28:0/687438950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf6bb4ec10 con 0x55bf6bb53170 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.752+0000 7f928a7fc640 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb75570 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f92840d8730 tx=0x7f9284005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f424edf9788e579 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.752+0000 7f92717fa640 1 -- 172.21.15.28:0/687438950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92840df070 con 0x55bf6bb53170 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.752+0000 7f92717fa640 1 -- 172.21.15.28:0/687438950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92840d3e50 con 0x55bf6bb53170 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.753+0000 7f9293dd6740 1 -- 172.21.15.28:0/687438950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 msgr2=0x55bf6bb75570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:33.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.753+0000 7f9293dd6740 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb75570 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f92840d8730 tx=0x7f9284005a50 comp rx=0 tx=0).stop 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.760+0000 7f9293dd6740 1 -- 172.21.15.28:0/687438950 shutdown_connections 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.760+0000 7f9293dd6740 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb5da30 0x55bf6bb75ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.760+0000 7f9293dd6740 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb75570 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.760+0000 7f9293dd6740 1 --2- 172.21.15.28:0/687438950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb52850 0x55bf6bb49f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.760+0000 7f9293dd6740 1 -- 172.21.15.28:0/687438950 >> 172.21.15.28:0/687438950 conn(0x55bf6bb4ede0 msgr2=0x55bf6bb59080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.760+0000 7f9293dd6740 1 -- 172.21.15.28:0/687438950 shutdown_connections 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.761+0000 7f9293dd6740 1 -- 172.21.15.28:0/687438950 wait complete. 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.761+0000 7f9293dd6740 1 -- start start 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.761+0000 7f9293dd6740 1 Processor -- start 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.762+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb65f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.762+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb66480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.762+0000 7f9293dd6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb57c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.762+0000 7f9293dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92780112a0 con 0x55bf6bb53170 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.762+0000 7f9293dd6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9278010bd0 con 0x55bf6bb52850 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.762+0000 7f9293dd6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f927800de10 con 0x55bf6bb5da30 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.762+0000 7f9293dd6740 10 client.0 _ll_register_callbacks cb 0x7f92840093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.764+0000 7f928affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb65f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.764+0000 7f928b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb57c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.764+0000 7f928affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb65f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:43448/0 (socket says 172.21.15.28:43448) 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.764+0000 7f928affd640 1 -- 172.21.15.28:0/762460303 learned_addr learned my addr 172.21.15.28:0/762460303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.765+0000 7f928affd640 1 -- 172.21.15.28:0/762460303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 msgr2=0x55bf6bb57c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.765+0000 7f928affd640 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb57c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.765+0000 7f928affd640 1 -- 172.21.15.28:0/762460303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 msgr2=0x55bf6bb66480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.765+0000 7f928affd640 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb66480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.765+0000 7f928affd640 1 -- 172.21.15.28:0/762460303 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf6bb58460 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.766+0000 7f928affd640 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb65f40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9280009d90 tx=0x7f92800168c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1a564c01c60b69f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.767+0000 7f926affd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.767+0000 7f926affd640 1 -- 172.21.15.28:0/762460303 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000f040 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.767+0000 7f926affd640 1 -- 172.21.15.28:0/762460303 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9280015890 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.768+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92840d3e50 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.769+0000 7f926affd640 1 -- 172.21.15.28:0/762460303 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9280015a90 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.771+0000 7f9293dd6740 10 client.25892 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.771+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bf6bb5f3e0 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f926affd640 1 -- 172.21.15.28:0/762460303 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92840d3e50 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f926affd640 1 -- 172.21.15.28:0/762460303 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9280025410 con 0x55bf6bb52850 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f9293dd6740 10 client.25892 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:33.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f9293dd6740 10 client.25892 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:02:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f9293dd6740 20 client.25892 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f9293dd6740 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bbd8ad0 0x55bf6bbdaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f9293dd6740 4 client.25892 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-15T22:02:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.775+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x55bf6babebe0 con 0x55bf6bbd8ad0 2024-09-15T22:02:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.778+0000 7f928a7fc640 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bbd8ad0 0x55bf6bbdaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.782+0000 7f928a7fc640 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bbd8ad0 0x55bf6bbdaef0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d417c1c3798742 server_cookie=43cc95bc49c3c87c in_seq=0 out_seq=0 2024-09-15T22:02:33.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:32.784+0000 7f926affd640 10 client.25892 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:02:33.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:33 smithi028 ceph-mon[30969]: pgmap v159: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 27 MiB/s wr, 4.62k op/s 2024-09-15T22:02:33.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:33 smithi203 ceph-mon[35546]: pgmap v159: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 27 MiB/s wr, 4.62k op/s 2024-09-15T22:02:34.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:33 smithi174 ceph-mon[35244]: pgmap v159: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 27 MiB/s wr, 4.62k op/s 2024-09-15T22:02:35.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:35 smithi028 ceph-mon[30969]: pgmap v160: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 25 MiB/s wr, 4.21k op/s 2024-09-15T22:02:35.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:35 smithi203 ceph-mon[35546]: pgmap v160: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 25 MiB/s wr, 4.21k op/s 2024-09-15T22:02:36.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:35 smithi174 ceph-mon[35244]: pgmap v160: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 25 MiB/s wr, 4.21k op/s 2024-09-15T22:02:37.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:37 smithi028 ceph-mon[30969]: pgmap v161: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 MiB/s wr, 3.56k op/s 2024-09-15T22:02:37.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:37 smithi203 ceph-mon[35546]: pgmap v161: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 MiB/s wr, 3.56k op/s 2024-09-15T22:02:38.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:37 smithi174 ceph-mon[35244]: pgmap v161: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 MiB/s wr, 3.56k op/s 2024-09-15T22:02:39.819 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:39 smithi028 ceph-mon[30969]: pgmap v162: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 28 MiB/s wr, 4.73k op/s 2024-09-15T22:02:39.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:39 smithi203 ceph-mon[35546]: pgmap v162: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 28 MiB/s wr, 4.73k op/s 2024-09-15T22:02:40.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:39 smithi174 ceph-mon[35244]: pgmap v162: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 28 MiB/s wr, 4.73k op/s 2024-09-15T22:02:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 -- 172.21.15.28:0/46307439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938101420 msgr2=0x7f093810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 --2- 172.21.15.28:0/46307439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938101420 0x7f093810d950 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0930009cd0 tx=0x7f093002a8e0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 -- 172.21.15.28:0/46307439 shutdown_connections 2024-09-15T22:02:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 --2- 172.21.15.28:0/46307439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938101420 0x7f093810d950 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 --2- 172.21.15.28:0/46307439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0938100a80 0x7f0938100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 --2- 172.21.15.28:0/46307439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0938106a80 0x7f0938106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 -- 172.21.15.28:0/46307439 >> 172.21.15.28:0/46307439 conn(0x7f09380fc7a0 msgr2=0x7f09380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 -- 172.21.15.28:0/46307439 shutdown_connections 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.792+0000 7f093f7a9640 1 -- 172.21.15.28:0/46307439 wait complete. 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 Processor -- start 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 -- start start 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0938100a80 0x7f0938102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0938101420 0x7f0938102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938106a80 0x7f09381030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0938110e80 con 0x7f0938106a80 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0938110d00 con 0x7f0938100a80 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093f7a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0938111000 con 0x7f0938101420 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093dd46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938106a80 0x7f09381030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093d545640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0938100a80 0x7f0938102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093dd46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938106a80 0x7f09381030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40450/0 (socket says 172.21.15.28:40450) 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093dd46640 1 -- 172.21.15.28:0/1738535397 learned_addr learned my addr 172.21.15.28:0/1738535397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.793+0000 7f093cd44640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0938101420 0x7f0938102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093dd46640 1 -- 172.21.15.28:0/1738535397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0938101420 msgr2=0x7f0938102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093dd46640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0938101420 0x7f0938102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093dd46640 1 -- 172.21.15.28:0/1738535397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0938100a80 msgr2=0x7f0938102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093dd46640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0938100a80 0x7f0938102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093dd46640 1 -- 172.21.15.28:0/1738535397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09381a6890 con 0x7f0938106a80 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093dd46640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938106a80 0x7f09381030d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0930005cc0 tx=0x7f0930003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f092e7fc640 1 -- 172.21.15.28:0/1738535397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930004040 con 0x7f0938106a80 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09381a6b20 con 0x7f0938106a80 2024-09-15T22:02:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f092e7fc640 1 -- 172.21.15.28:0/1738535397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09300046c0 con 0x7f0938106a80 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f092e7fc640 1 -- 172.21.15.28:0/1738535397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930046480 con 0x7f0938106a80 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.794+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09381a7060 con 0x7f0938106a80 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.796+0000 7f092e7fc640 1 -- 172.21.15.28:0/1738535397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f09300305a0 con 0x7f0938106a80 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 Processor -- start 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 -- start start 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f61199e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199e5d0 0x55f61199ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119ab760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6119042f0 con 0x55f61199e5d0 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6118abcf0 con 0x55f6119a9370 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f0941b82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f6119002b0 con 0x55f61199fb40 2024-09-15T22:02:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.797+0000 7f093d545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199e5d0 0x55f61199ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199e5d0 0x55f61199ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40464/0 (socket says 172.21.15.28:40464) 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 -- 172.21.15.28:0/4055843320 learned_addr learned my addr 172.21.15.28:0/4055843320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093cd44640 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f61199e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093cd44640 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f092e7fc640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f09240a33b0 0x7f09240a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093dd46640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f09240a33b0 0x7f09240a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 -- 172.21.15.28:0/4055843320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 msgr2=0x55f61199e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f61199e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 -- 172.21.15.28:0/4055843320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 msgr2=0x55f6119ab760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119ab760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 -- 172.21.15.28:0/4055843320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6118903a0 con 0x55f61199e5d0 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f092e7fc640 1 -- 172.21.15.28:0/1738535397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f093002e030 con 0x7f0938106a80 2024-09-15T22:02:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093d545640 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199e5d0 0x55f61199ea10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f093400bc90 tx=0x7f09340107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6226cb60394a96ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f09237fe640 1 -- 172.21.15.28:0/4055843320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093401c480 con 0x55f61199e5d0 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f09237fe640 1 -- 172.21.15.28:0/4055843320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f093401cb00 con 0x55f61199e5d0 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f093dd46640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f09240a33b0 0x7f09240a5870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09300eec90 tx=0x7f09300095c0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.798+0000 7f09237fe640 1 -- 172.21.15.28:0/4055843320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934011b40 con 0x55f61199e5d0 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 -- 172.21.15.28:0/4055843320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199e5d0 msgr2=0x55f61199ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199e5d0 0x55f61199ea10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f093400bc90 tx=0x7f09340107a0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 -- 172.21.15.28:0/4055843320 shutdown_connections 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199e5d0 0x55f61199ea10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 --2- 172.21.15.28:0/4055843320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f61199e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 -- 172.21.15.28:0/4055843320 >> 172.21.15.28:0/4055843320 conn(0x55f611967310 msgr2=0x55f61199ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 -- 172.21.15.28:0/4055843320 shutdown_connections 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.799+0000 7f0941b82740 1 -- 172.21.15.28:0/4055843320 wait complete. 2024-09-15T22:02:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 -- start start 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 Processor -- start 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119a8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6119a9370 0x55f6119a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6119ac070 con 0x55f61199fb40 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6118bf060 con 0x55f61199e5d0 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f6119ac1f0 con 0x55f6119a9370 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f0941b82740 10 client.0 _ll_register_callbacks cb 0x55f6116128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f093d545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f093cd44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119a8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f093cd44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119a8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:43500/0 (socket says 172.21.15.28:43500) 2024-09-15T22:02:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f093cd44640 1 -- 172.21.15.28:0/2332542898 learned_addr learned my addr 172.21.15.28:0/2332542898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.800+0000 7f093cd44640 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6119a9370 0x55f6119a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f093d545640 1 -- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6119a9370 msgr2=0x55f6119a65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f093d545640 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6119a9370 0x55f6119a65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f093d545640 1 -- 172.21.15.28:0/2332542898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 msgr2=0x55f6119a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f093d545640 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119a8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f093d545640 1 -- 172.21.15.28:0/2332542898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6116753c0 con 0x55f61199fb40 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f093cd44640 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119a8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f093d545640 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a5ff0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0934011480 tx=0x7f093401cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974d8401e6f79552 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0920ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0920ff9640 1 -- 172.21.15.28:0/2332542898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934029020 con 0x55f61199fb40 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f093401cb00 con 0x55f61199fb40 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0920ff9640 1 -- 172.21.15.28:0/2332542898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f093401c690 con 0x55f61199fb40 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0920ff9640 1 -- 172.21.15.28:0/2332542898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0934031050 con 0x55f61199fb40 2024-09-15T22:02:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0941b82740 10 client.16341 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f6119b7b10 con 0x55f61199fb40 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.801+0000 7f0920ff9640 1 -- 172.21.15.28:0/2332542898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f093401cb00 con 0x55f61199fb40 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0920ff9640 1 -- 172.21.15.28:0/2332542898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f0934024070 con 0x55f61199fb40 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0941b82740 10 client.16341 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0941b82740 10 client.16341 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0941b82740 20 client.16341 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0941b82740 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6119b8e60 0x55f6119bb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0941b82740 4 client.16341 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f611911860 con 0x55f6119b8e60 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f093cd44640 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6119b8e60 0x55f6119bb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f093cd44640 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6119b8e60 0x55f6119bb250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d0726582cb1d584 server_cookie=4e585ba85c730f2b in_seq=0 out_seq=0 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.802+0000 7f0920ff9640 10 client.16341 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:02:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.803+0000 7f0920ff9640 1 -- 172.21.15.28:0/2332542898 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f611911860 con 0x55f6119b8e60 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.803+0000 7f0920ff9640 10 client.16341 handle_command_reply: tid=0 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.803+0000 7f0941b82740 1 client.16341 shutdown 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 msgr2=0x55f6119a5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a5ff0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0934011480 tx=0x7f093401cf60 comp rx=0 tx=0).stop 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 shutdown_connections 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f6119b8e60 0x55f6119bb250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6119a9370 0x55f6119a65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a5ff0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 --2- 172.21.15.28:0/2332542898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119a8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 >> 172.21.15.28:0/2332542898 conn(0x55f611967310 msgr2=0x55f6119a0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0920ff9640 0 client.16341 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:02:41.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 shutdown_connections 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 1 -- 172.21.15.28:0/2332542898 wait complete. 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.804+0000 7f0941b82740 20 client.16341 trim_cache size 0 max 16384 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 Processor -- start 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 -- start start 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199e5d0 0x55f6119b5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119a1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09381b10c0 con 0x55f61199fb40 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09381b2c00 con 0x55f6119a9370 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f0941b82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09381b1790 con 0x55f61199e5d0 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f093d545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f093d545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40474/0 (socket says 172.21.15.28:40474) 2024-09-15T22:02:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f093d545640 1 -- 172.21.15.28:0/3672649222 learned_addr learned my addr 172.21.15.28:0/3672649222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f093cd44640 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199e5d0 0x55f6119b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.817+0000 7f093cd44640 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f093d545640 1 -- 172.21.15.28:0/3672649222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199e5d0 msgr2=0x55f6119b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f093d545640 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199e5d0 0x55f6119b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f093d545640 1 -- 172.21.15.28:0/3672649222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 msgr2=0x55f6119a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f093d545640 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f093d545640 1 -- 172.21.15.28:0/3672649222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6119a1ba0 con 0x55f61199fb40 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f093cd44640 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f093d545640 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a1120 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f093400b620 tx=0x7f093400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3de26ba0c0c1bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f0922ffd640 1 -- 172.21.15.28:0/3672649222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934029070 con 0x55f61199fb40 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f0922ffd640 1 -- 172.21.15.28:0/3672649222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0934005040 con 0x55f61199fb40 2024-09-15T22:02:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.818+0000 7f0922ffd640 1 -- 172.21.15.28:0/3672649222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934011710 con 0x55f61199fb40 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- 172.21.15.28:0/3672649222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 msgr2=0x55f6119a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a1120 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f093400b620 tx=0x7f093400b730 comp rx=0 tx=0).stop 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- 172.21.15.28:0/3672649222 shutdown_connections 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6119a9370 0x55f6119a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f61199fb40 0x55f6119a1120 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 --2- 172.21.15.28:0/3672649222 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199e5d0 0x55f6119b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- 172.21.15.28:0/3672649222 >> 172.21.15.28:0/3672649222 conn(0x55f611967310 msgr2=0x55f6119a8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- 172.21.15.28:0/3672649222 shutdown_connections 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- 172.21.15.28:0/3672649222 wait complete. 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- start start 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 Processor -- start 2024-09-15T22:02:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f6119b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6119a9370 0x55f6119a2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09381b2530 con 0x55f6119a9370 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09381b1e60 con 0x55f61199e5d0 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.819+0000 7f0941b82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09381af0a0 con 0x55f61199fb40 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f0941b82740 10 client.0 _ll_register_callbacks cb 0x7f092800be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093d545640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f6119b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119b14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:43526/0 (socket says 172.21.15.28:43526) 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 -- 172.21.15.28:0/782904431 learned_addr learned my addr 172.21.15.28:0/782904431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6119a9370 0x55f6119a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 -- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 msgr2=0x55f6119b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f6119b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 -- 172.21.15.28:0/782904431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 msgr2=0x55f6119b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 -- 172.21.15.28:0/782904431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6119a2c60 con 0x55f6119a9370 2024-09-15T22:02:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093d545640 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f6119b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f093cd44640 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6119a9370 0x55f6119a2620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55f6119bb880 tx=0x7f092800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d474243ed841ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f0920ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f0920ff9640 1 -- 172.21.15.28:0/782904431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928032030 con 0x55f6119a9370 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f0920ff9640 1 -- 172.21.15.28:0/782904431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f092801b040 con 0x55f6119a9370 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.820+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0934005040 con 0x55f6119a9370 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0920ff9640 1 -- 172.21.15.28:0/782904431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f092803c050 con 0x55f6119a9370 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 10 client.16347 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f611a231f0 con 0x55f6119a9370 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0920ff9640 1 -- 172.21.15.28:0/782904431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0934005040 con 0x55f6119a9370 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0920ff9640 1 -- 172.21.15.28:0/782904431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f092802b070 con 0x55f6119a9370 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 10 client.16347 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 10 client.16347 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 20 client.16347 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f611a239a0 0x55f611a25dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 4 client.16347 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f61190bd70 con 0x55f611a239a0 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.821+0000 7f093d545640 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f611a239a0 0x55f611a25dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f093d545640 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f611a239a0 0x55f611a25dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efec5ac4ccc8d42f server_cookie=78907261c44afa3e in_seq=0 out_seq=0 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0920ff9640 10 client.16347 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0920ff9640 1 -- 172.21.15.28:0/782904431 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f093403b020 con 0x55f611a239a0 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0920ff9640 10 client.16347 handle_command_reply: tid=0 2024-09-15T22:02:41.721 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 client.16347 shutdown 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6119a9370 msgr2=0x55f6119a2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6119a9370 0x55f6119a2620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55f6119bb880 tx=0x7f092800b390 comp rx=0 tx=0).stop 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 shutdown_connections 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55f611a239a0 0x55f611a25dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6119a9370 0x55f6119a2620 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f61199fb40 0x55f6119b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0920ff9640 0 client.16347 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 --2- 172.21.15.28:0/782904431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f61199e5d0 0x55f6119b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 >> 172.21.15.28:0/782904431 conn(0x55f611967310 msgr2=0x55f6119994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 shutdown_connections 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 1 -- 172.21.15.28:0/782904431 wait complete. 2024-09-15T22:02:41.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.822+0000 7f0941b82740 20 client.16347 trim_cache size 0 max 16384 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f09240a33b0 msgr2=0x7f09240a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f09240a33b0 0x7f09240a5870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09300eec90 tx=0x7f09300095c0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938106a80 msgr2=0x7f09381030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938106a80 0x7f09381030d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0930005cc0 tx=0x7f0930003970 comp rx=0 tx=0).stop 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 shutdown_connections 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f09240a33b0 0x7f09240a5870 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0938106a80 0x7f09381030d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0938101420 0x7f0938102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 --2- 172.21.15.28:0/1738535397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0938100a80 0x7f0938102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 >> 172.21.15.28:0/1738535397 conn(0x7f09380fc7a0 msgr2=0x7f093806a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 shutdown_connections 2024-09-15T22:02:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:02:39.823+0000 7f093f7a9640 1 -- 172.21.15.28:0/1738535397 wait complete. 2024-09-15T22:02:41.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:41 smithi028 ceph-mon[30969]: pgmap v163: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 30 MiB/s wr, 5.03k op/s 2024-09-15T22:02:41.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:41 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:02:41.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:41 smithi203 ceph-mon[35546]: pgmap v163: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 30 MiB/s wr, 5.03k op/s 2024-09-15T22:02:41.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:41 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:02:42.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:41 smithi174 ceph-mon[35244]: pgmap v163: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 30 MiB/s wr, 5.03k op/s 2024-09-15T22:02:42.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:41 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:02:43.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:43 smithi028 ceph-mon[30969]: pgmap v164: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 41 MiB/s wr, 6.92k op/s 2024-09-15T22:02:43.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:43 smithi203 ceph-mon[35546]: pgmap v164: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 41 MiB/s wr, 6.92k op/s 2024-09-15T22:02:44.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:43 smithi174 ceph-mon[35244]: pgmap v164: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 41 MiB/s wr, 6.92k op/s 2024-09-15T22:02:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 -- 172.21.15.28:0/3799503231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 msgr2=0x7fc7f8106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/3799503231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f8106da0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8009930 tx=0x7fc7e802a420 comp rx=0 tx=0).stop 2024-09-15T22:02:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 -- 172.21.15.28:0/3799503231 shutdown_connections 2024-09-15T22:02:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/3799503231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8101350 0x7fc7f810d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/3799503231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 0x7fc7f8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/3799503231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f8106da0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 -- 172.21.15.28:0/3799503231 >> 172.21.15.28:0/3799503231 conn(0x7fc7f8077990 msgr2=0x7fc7f80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 -- 172.21.15.28:0/3799503231 shutdown_connections 2024-09-15T22:02:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 -- 172.21.15.28:0/3799503231 wait complete. 2024-09-15T22:02:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.990+0000 7fc7fd01e640 1 Processor -- start 2024-09-15T22:02:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7fd01e640 1 -- start start 2024-09-15T22:02:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7fd01e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 0x7fc7f810e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7fd01e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8101350 0x7fc7f810e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7fd01e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f810ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7fd01e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8111ef0 con 0x7fc7f81069c0 2024-09-15T22:02:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7fd01e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8111d70 con 0x7fc7f81009b0 2024-09-15T22:02:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7fd01e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8112070 con 0x7fc7f8101350 2024-09-15T22:02:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f810ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 0x7fc7f810e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 0x7fc7f810e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:34892/0 (socket says 172.21.15.28:34892) 2024-09-15T22:02:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f810ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49874/0 (socket says 172.21.15.28:49874) 2024-09-15T22:02:44.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f6d9d640 1 -- 172.21.15.28:0/992889603 learned_addr learned my addr 172.21.15.28:0/992889603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f659c640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8101350 0x7fc7f810e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f6d9d640 1 -- 172.21.15.28:0/992889603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8101350 msgr2=0x7fc7f810e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8101350 0x7fc7f810e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f6d9d640 1 -- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 msgr2=0x7fc7f810ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.991+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f810ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:44.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7f6d9d640 1 -- 172.21.15.28:0/992889603 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f81a68c0 con 0x7fc7f81009b0 2024-09-15T22:02:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7f759e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f810ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 0x7fc7f810e310 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8009230 tx=0x7fc7e8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7dffff640 1 -- 172.21.15.28:0/992889603 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e803f030 con 0x7fc7f81009b0 2024-09-15T22:02:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f81a6b50 con 0x7fc7f81009b0 2024-09-15T22:02:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7dffff640 1 -- 172.21.15.28:0/992889603 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e8004440 con 0x7fc7f81009b0 2024-09-15T22:02:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7dffff640 1 -- 172.21.15.28:0/992889603 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8035070 con 0x7fc7f81009b0 2024-09-15T22:02:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.992+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f81a7090 con 0x7fc7f81009b0 2024-09-15T22:02:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.994+0000 7fc7ffe8d740 1 Processor -- start 2024-09-15T22:02:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7ffe8d740 1 -- start start 2024-09-15T22:02:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7dffff640 1 -- 172.21.15.28:0/992889603 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fc7e8042030 con 0x7fc7f81009b0 2024-09-15T22:02:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aac9ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aaca7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacaa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591aac012f0 con 0x5591aac9d410 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5591aaba9270 con 0x5591aaca7cb0 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5591aabfd2b0 con 0x5591aac9caf0 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aaca7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacaa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aaca7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49882/0 (socket says 172.21.15.28:49882) 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f6d9d640 1 -- 172.21.15.28:0/3500236639 learned_addr learned my addr 172.21.15.28:0/3500236639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f759e640 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aac9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f659c640 1 -- 172.21.15.28:0/3500236639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 msgr2=0x5591aac9ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f659c640 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aac9ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f659c640 1 -- 172.21.15.28:0/3500236639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 msgr2=0x5591aaca7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f659c640 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aaca7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f659c640 1 -- 172.21.15.28:0/3500236639 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591aaba8340 con 0x5591aaca7cb0 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.995+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aaca7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.996+0000 7fc7dffff640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7e00a3460 0x7fc7e00a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.996+0000 7fc7f659c640 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacaa0a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec009bf0 tx=0x7fc7ec011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc1b514d5a82f70b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.996+0000 7fc7f759e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7e00a3460 0x7fc7e00a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.996+0000 7fc7dffff640 1 -- 172.21.15.28:0/992889603 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc7e80f0370 con 0x7fc7f81009b0 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.996+0000 7fc7dd7fa640 1 -- 172.21.15.28:0/3500236639 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec004050 con 0x5591aaca7cb0 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.996+0000 7fc7dd7fa640 1 -- 172.21.15.28:0/3500236639 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec005500 con 0x5591aaca7cb0 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.996+0000 7fc7dd7fa640 1 -- 172.21.15.28:0/3500236639 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec0035f0 con 0x5591aaca7cb0 2024-09-15T22:02:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3500236639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 msgr2=0x5591aacaa0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7f759e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7e00a3460 0x7fc7e00a5920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009fd0 tx=0x7fc7f0009500 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacaa0a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec009bf0 tx=0x7fc7ec011810 comp rx=0 tx=0).stop 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3500236639 shutdown_connections 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacaa0a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aaca7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3500236639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aac9ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3500236639 >> 172.21.15.28:0/3500236639 conn(0x5591aac99060 msgr2=0x5591aac99440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3500236639 shutdown_connections 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3500236639 wait complete. 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- start start 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 Processor -- start 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9caf0 0x5591aaca4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9d410 0x5591aaca5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aaca7cb0 0x5591aaca5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591aacaa880 con 0x5591aaca7cb0 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5591aab8a290 con 0x5591aac9caf0 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5591aacaaa00 con 0x5591aac9d410 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.997+0000 7fc7ffe8d740 10 client.0 _ll_register_callbacks cb 0x5591aabbb140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aaca7cb0 0x5591aaca5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9caf0 0x5591aaca4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aaca7cb0 0x5591aaca5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49892/0 (socket says 172.21.15.28:49892) 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 -- 172.21.15.28:0/1263679548 learned_addr learned my addr 172.21.15.28:0/1263679548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f659c640 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9d410 0x5591aaca5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 -- 172.21.15.28:0/1263679548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9d410 msgr2=0x5591aaca5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9d410 0x5591aaca5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 -- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9caf0 msgr2=0x5591aaca4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9caf0 0x5591aaca4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 -- 172.21.15.28:0/1263679548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591aa914210 con 0x5591aaca7cb0 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9caf0 0x5591aaca4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7f759e640 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aaca7cb0 0x5591aaca5540 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0013010 tx=0x7fc7f0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce530d0623d9cb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7daff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7daff5640 1 -- 172.21.15.28:0/1263679548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00045c0 con 0x5591aaca7cb0 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7daff5640 1 -- 172.21.15.28:0/1263679548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc7f0004760 con 0x5591aaca7cb0 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7ec005500 con 0x5591aaca7cb0 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7daff5640 1 -- 172.21.15.28:0/1263679548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7f0012020 con 0x5591aaca7cb0 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7ffe8d740 10 client.16359 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.998+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5591aa9e0620 con 0x5591aaca7cb0 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7daff5640 1 -- 172.21.15.28:0/1263679548 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec005500 con 0x5591aaca7cb0 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7daff5640 1 -- 172.21.15.28:0/1263679548 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fc7f0027070 con 0x5591aaca7cb0 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7ffe8d740 10 client.16359 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7ffe8d740 10 client.16359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7ffe8d740 20 client.16359 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aacb7bb0 0x5591aacba020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7ffe8d740 4 client.16359 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591aac0ea20 con 0x5591aacb7bb0 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aacb7bb0 0x5591aacba020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aacb7bb0 0x5591aacba020 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57cff47fcc3b8dac server_cookie=36c34de845533a2e in_seq=0 out_seq=0 2024-09-15T22:02:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:44.999+0000 7fc7daff5640 10 client.16359 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7daff5640 1 -- 172.21.15.28:0/1263679548 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591aac0ea20 con 0x5591aacb7bb0 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7daff5640 10 client.16359 handle_command_reply: tid=0 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 client.16359 shutdown 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aaca7cb0 msgr2=0x5591aaca5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aaca7cb0 0x5591aaca5540 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0013010 tx=0x7fc7f0004ab0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 shutdown_connections 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aacb7bb0 0x5591aacba020 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aaca7cb0 0x5591aaca5540 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9d410 0x5591aaca5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7daff5640 0 client.16359 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1263679548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9caf0 0x5591aaca4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 >> 172.21.15.28:0/1263679548 conn(0x5591aac99060 msgr2=0x5591aac9edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 shutdown_connections 2024-09-15T22:02:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.000+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1263679548 wait complete. 2024-09-15T22:02:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.001+0000 7fc7ffe8d740 20 client.16359 trim_cache size 0 max 16384 2024-09-15T22:02:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.022+0000 7fc7ffe8d740 1 Processor -- start 2024-09-15T22:02:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.022+0000 7fc7ffe8d740 1 -- start start 2024-09-15T22:02:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.022+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aaca5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aacbf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacbfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b1000 con 0x5591aac9d410 2024-09-15T22:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b2b40 con 0x5591aaca7cb0 2024-09-15T22:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b16d0 con 0x5591aac9caf0 2024-09-15T22:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacbfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacbfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:34916/0 (socket says 172.21.15.28:34916) 2024-09-15T22:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.023+0000 7fc7f759e640 1 -- 172.21.15.28:0/3090957936 learned_addr learned my addr 172.21.15.28:0/3090957936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.024+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aaca5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.024+0000 7fc7f659c640 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aacbf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.024+0000 7fc7f659c640 1 -- 172.21.15.28:0/3090957936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 msgr2=0x5591aaca5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.024+0000 7fc7f659c640 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aaca5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.024+0000 7fc7f659c640 1 -- 172.21.15.28:0/3090957936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 msgr2=0x5591aacbfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.024+0000 7fc7f659c640 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacbfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.024+0000 7fc7f659c640 1 -- 172.21.15.28:0/3090957936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591aac98e90 con 0x5591aac9d410 2024-09-15T22:02:45.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.025+0000 7fc7f659c640 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aacbf980 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec01b040 tx=0x7fc7ec005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72556712c0a2c6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.026+0000 7fc7dcff9640 1 -- 172.21.15.28:0/3090957936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec009ea0 con 0x5591aac9d410 2024-09-15T22:02:45.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.026+0000 7fc7dcff9640 1 -- 172.21.15.28:0/3090957936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec0035f0 con 0x5591aac9d410 2024-09-15T22:02:45.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.027+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3090957936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 msgr2=0x5591aacbf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:02:45.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.027+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aacbf980 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec01b040 tx=0x7fc7ec005d30 comp rx=0 tx=0).stop 2024-09-15T22:02:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.029+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3090957936 shutdown_connections 2024-09-15T22:02:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.029+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aaca7cb0 0x5591aacbfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.029+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9d410 0x5591aacbf980 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.029+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/3090957936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aac9caf0 0x5591aaca5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.029+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3090957936 >> 172.21.15.28:0/3090957936 conn(0x5591aac99060 msgr2=0x5591aaca3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:02:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.030+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3090957936 shutdown_connections 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.034+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/3090957936 wait complete. 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.034+0000 7fc7ffe8d740 1 -- start start 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.034+0000 7fc7ffe8d740 1 Processor -- start 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.034+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9caf0 0x5591aacaffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.034+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9d410 0x5591aacb0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7ffe8d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aaca7cb0 0x5591aaca88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b2470 con 0x5591aac9caf0 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81b1da0 con 0x5591aac9d410 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7ffe8d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81aefe0 con 0x5591aaca7cb0 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7ffe8d740 10 client.0 _ll_register_callbacks cb 0x7fc7f00125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aaca7cb0 0x5591aaca88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aaca7cb0 0x5591aaca88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:44754/0 (socket says 172.21.15.28:44754) 2024-09-15T22:02:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 -- 172.21.15.28:0/1168446676 learned_addr learned my addr 172.21.15.28:0/1168446676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:02:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f659c640 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9d410 0x5591aacb0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 -- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9d410 msgr2=0x5591aacb0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9d410 0x5591aacb0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 -- 172.21.15.28:0/1168446676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9caf0 msgr2=0x5591aacaffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:02:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9caf0 0x5591aacaffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:02:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.035+0000 7fc7f759e640 1 -- 172.21.15.28:0/1168446676 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591aac9a1f0 con 0x5591aaca7cb0 2024-09-15T22:02:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.036+0000 7fc7f759e640 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aaca7cb0 0x5591aaca88b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc7f001f6f0 tx=0x7fc7f00123f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ef4b8b02d43ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:02:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.037+0000 7fc7daff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:02:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.037+0000 7fc7daff5640 1 -- 172.21.15.28:0/1168446676 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f005b020 con 0x5591aaca7cb0 2024-09-15T22:02:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.037+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7ec0035f0 con 0x5591aaca7cb0 2024-09-15T22:02:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.037+0000 7fc7daff5640 1 -- 172.21.15.28:0/1168446676 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc7f0004410 con 0x5591aaca7cb0 2024-09-15T22:02:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.039+0000 7fc7daff5640 1 -- 172.21.15.28:0/1168446676 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7f0004610 con 0x5591aaca7cb0 2024-09-15T22:02:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.039+0000 7fc7ffe8d740 10 client.25891 fetch_fsmap learned FSMap version 32 2024-09-15T22:02:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.039+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5591aacaf800 con 0x5591aaca7cb0 2024-09-15T22:02:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7daff5640 1 -- 172.21.15.28:0/1168446676 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec0035f0 con 0x5591aaca7cb0 2024-09-15T22:02:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7daff5640 1 -- 172.21.15.28:0/1168446676 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fc7f0027050 con 0x5591aaca7cb0 2024-09-15T22:02:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7ffe8d740 10 client.25891 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:02:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7ffe8d740 10 client.25891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:02:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7ffe8d740 20 client.25891 populate_metadata read hostname 'smithi028' 2024-09-15T22:02:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aad22b30 0x5591aad24f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:02:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7ffe8d740 4 client.25891 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:02:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.041+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5591aac08d70 con 0x5591aad22b30 2024-09-15T22:02:45.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.044+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aad22b30 0x5591aad24f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:02:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.055+0000 7fc7f6d9d640 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aad22b30 0x5591aad24f90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72d593747c32fb36 server_cookie=4f317c8737086bda in_seq=0 out_seq=0 2024-09-15T22:02:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:02:45.055+0000 7fc7daff5640 10 client.25891 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:02:45.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:45 smithi028 ceph-mon[30969]: pgmap v165: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 39 MiB/s wr, 6.61k op/s 2024-09-15T22:02:45.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:45 smithi203 ceph-mon[35546]: pgmap v165: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 39 MiB/s wr, 6.61k op/s 2024-09-15T22:02:46.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:45 smithi174 ceph-mon[35244]: pgmap v165: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 39 MiB/s wr, 6.61k op/s 2024-09-15T22:02:47.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:47 smithi028 ceph-mon[30969]: pgmap v166: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 41 MiB/s wr, 6.90k op/s 2024-09-15T22:02:47.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:47 smithi203 ceph-mon[35546]: pgmap v166: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 41 MiB/s wr, 6.90k op/s 2024-09-15T22:02:48.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:47 smithi174 ceph-mon[35244]: pgmap v166: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 41 MiB/s wr, 6.90k op/s 2024-09-15T22:02:49.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:49 smithi028 ceph-mon[30969]: pgmap v167: 129 pgs: 129 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 49 MiB/s wr, 8.27k op/s 2024-09-15T22:02:49.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:49 smithi203 ceph-mon[35546]: pgmap v167: 129 pgs: 129 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 49 MiB/s wr, 8.27k op/s 2024-09-15T22:02:50.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:49 smithi174 ceph-mon[35244]: pgmap v167: 129 pgs: 129 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 49 MiB/s wr, 8.27k op/s 2024-09-15T22:02:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:51 smithi028 ceph-mon[30969]: pgmap v168: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 MiB/s wr, 7.34k op/s 2024-09-15T22:02:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:51 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:02:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:51 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:02:51.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:51 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:02:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:51 smithi203 ceph-mon[35546]: pgmap v168: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 MiB/s wr, 7.34k op/s 2024-09-15T22:02:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:51 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:02:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:51 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:02:51.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:51 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:02:52.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:51 smithi174 ceph-mon[35244]: pgmap v168: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 MiB/s wr, 7.34k op/s 2024-09-15T22:02:52.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:51 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:02:52.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:51 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:02:52.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:51 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:02:53.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:53 smithi028 ceph-mon[30969]: pgmap v169: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 MiB/s wr, 7.78k op/s 2024-09-15T22:02:53.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:53 smithi203 ceph-mon[35546]: pgmap v169: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 MiB/s wr, 7.78k op/s 2024-09-15T22:02:54.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:53 smithi174 ceph-mon[35244]: pgmap v169: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 46 MiB/s wr, 7.78k op/s 2024-09-15T22:02:55.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:02:55.579 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:02:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:55 smithi028 ceph-mon[30969]: pgmap v170: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 35 MiB/s wr, 5.87k op/s 2024-09-15T22:02:55.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:55 smithi203 ceph-mon[35546]: pgmap v170: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 35 MiB/s wr, 5.87k op/s 2024-09-15T22:02:56.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:55 smithi174 ceph-mon[35244]: pgmap v170: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 35 MiB/s wr, 5.87k op/s 2024-09-15T22:02:57.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:57 smithi028 ceph-mon[30969]: pgmap v171: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 6.29k op/s 2024-09-15T22:02:57.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:57 smithi203 ceph-mon[35546]: pgmap v171: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 6.29k op/s 2024-09-15T22:02:58.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:57 smithi174 ceph-mon[35244]: pgmap v171: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 6.29k op/s 2024-09-15T22:02:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:02:59 smithi028 ceph-mon[30969]: pgmap v172: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 26 KiB/s rd, 38 MiB/s wr, 6.35k op/s 2024-09-15T22:02:59.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:02:59 smithi203 ceph-mon[35546]: pgmap v172: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 26 KiB/s rd, 38 MiB/s wr, 6.35k op/s 2024-09-15T22:03:00.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:02:59 smithi174 ceph-mon[35244]: pgmap v172: 129 pgs: 129 active+clean; 18 GiB data, 49 GiB used, 1023 GiB / 1.0 TiB avail; 26 KiB/s rd, 38 MiB/s wr, 6.35k op/s 2024-09-15T22:03:01.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:01 smithi028 ceph-mon[30969]: pgmap v173: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 31 MiB/s wr, 5.29k op/s 2024-09-15T22:03:01.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:01 smithi203 ceph-mon[35546]: pgmap v173: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 31 MiB/s wr, 5.29k op/s 2024-09-15T22:03:02.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:01 smithi174 ceph-mon[35244]: pgmap v173: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 16 KiB/s rd, 31 MiB/s wr, 5.29k op/s 2024-09-15T22:03:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.957+0000 7fd97527a640 1 -- 172.21.15.28:0/2146268267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd97010ea80 msgr2=0x7fd9701154f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.957+0000 7fd97527a640 1 --2- 172.21.15.28:0/2146268267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd97010ea80 0x7fd9701154f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd96400af00 tx=0x7fd96402e630 comp rx=0 tx=0).stop 2024-09-15T22:03:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.958+0000 7fd97527a640 1 -- 172.21.15.28:0/2146268267 shutdown_connections 2024-09-15T22:03:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.958+0000 7fd97527a640 1 --2- 172.21.15.28:0/2146268267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd97010ea80 0x7fd9701154f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.958+0000 7fd97527a640 1 --2- 172.21.15.28:0/2146268267 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd970109e00 0x7fd97010e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.958+0000 7fd97527a640 1 --2- 172.21.15.28:0/2146268267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd970109450 0x7fd970109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.958+0000 7fd97527a640 1 -- 172.21.15.28:0/2146268267 >> 172.21.15.28:0/2146268267 conn(0x7fd97006e2d0 msgr2=0x7fd97006e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.958+0000 7fd97527a640 1 -- 172.21.15.28:0/2146268267 shutdown_connections 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.958+0000 7fd97527a640 1 -- 172.21.15.28:0/2146268267 wait complete. 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.959+0000 7fd97527a640 1 Processor -- start 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.959+0000 7fd97527a640 1 -- start start 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.960+0000 7fd97527a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd970109450 0x7fd97011f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.960+0000 7fd97527a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd970109e00 0x7fd970118690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.960+0000 7fd97527a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd97010ea80 0x7fd970118bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.960+0000 7fd97527a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd970120da0 con 0x7fd970109450 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.960+0000 7fd97527a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd970120c20 con 0x7fd970109e00 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.960+0000 7fd97527a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd970120f20 con 0x7fd97010ea80 2024-09-15T22:03:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.961+0000 7fd96e7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd970109e00 0x7fd970118690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.961+0000 7fd96effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd970109450 0x7fd97011f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.961+0000 7fd96effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd970109450 0x7fd97011f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35234/0 (socket says 172.21.15.28:35234) 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.961+0000 7fd96effd640 1 -- 172.21.15.28:0/4015533867 learned_addr learned my addr 172.21.15.28:0/4015533867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.961+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd97010ea80 0x7fd970118bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.961+0000 7fd96f7fe640 1 -- 172.21.15.28:0/4015533867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd970109e00 msgr2=0x7fd970118690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.961+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd970109e00 0x7fd970118690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.962+0000 7fd96f7fe640 1 -- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd970109450 msgr2=0x7fd97011f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.962+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd970109450 0x7fd97011f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.962+0000 7fd96f7fe640 1 -- 172.21.15.28:0/4015533867 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd970119210 con 0x7fd97010ea80 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.962+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd97010ea80 0x7fd970118bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd964037040 tx=0x7fd96400a980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.962+0000 7fd957fff640 1 -- 172.21.15.28:0/4015533867 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964004450 con 0x7fd97010ea80 2024-09-15T22:03:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.962+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9701194a0 con 0x7fd97010ea80 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.962+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd97011d460 con 0x7fd97010ea80 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.963+0000 7fd957fff640 1 -- 172.21.15.28:0/4015533867 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd964043070 con 0x7fd97010ea80 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.963+0000 7fd957fff640 1 -- 172.21.15.28:0/4015533867 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9640035f0 con 0x7fd97010ea80 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.966+0000 7fd957fff640 1 -- 172.21.15.28:0/4015533867 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd96400f070 con 0x7fd97010ea80 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 Processor -- start 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 -- start start 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0ae090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0aea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ad2c0142f0 con 0x55ad2c0ae5d0 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ad2bfbbcf0 con 0x55ad2c0b9370 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd977653740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ad2c0102b0 con 0x55ad2c0afb40 2024-09-15T22:03:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.967+0000 7fd96e7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.968+0000 7fd96e7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:53378/0 (socket says 172.21.15.28:53378) 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.968+0000 7fd96e7fc640 1 -- 172.21.15.28:0/279632480 learned_addr learned my addr 172.21.15.28:0/279632480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.968+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96effd640 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96e7fc640 1 -- 172.21.15.28:0/279632480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 msgr2=0x55ad2c0bb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96e7fc640 1 -- 172.21.15.28:0/279632480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 msgr2=0x55ad2c0aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96e7fc640 1 -- 172.21.15.28:0/279632480 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad2bfa03a0 con 0x55ad2c0afb40 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96effd640 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0ae090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd96000ab00 tx=0x7fd96000a540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e7c78f64217b86 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.969+0000 7fd957fff640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9580a3500 0x7fd9580a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.970+0000 7fd957fff640 1 -- 172.21.15.28:0/4015533867 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd9640f0930 con 0x7fd97010ea80 2024-09-15T22:03:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.970+0000 7fd96effd640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9580a3500 0x7fd9580a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.971+0000 7fd96effd640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9580a3500 0x7fd9580a59c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55ad2c0abac0 tx=0x7fd96800b040 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.972+0000 7fd9557fa640 1 -- 172.21.15.28:0/279632480 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96000cc60 con 0x55ad2c0afb40 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.972+0000 7fd9557fa640 1 -- 172.21.15.28:0/279632480 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd960019070 con 0x55ad2c0afb40 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.972+0000 7fd9557fa640 1 -- 172.21.15.28:0/279632480 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960013690 con 0x55ad2c0afb40 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.974+0000 7fd977653740 1 -- 172.21.15.28:0/279632480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 msgr2=0x55ad2c0ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.974+0000 7fd977653740 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0ae090 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd96000ab00 tx=0x7fd96000a540 comp rx=0 tx=0).stop 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.976+0000 7fd977653740 1 -- 172.21.15.28:0/279632480 shutdown_connections 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.976+0000 7fd977653740 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.976+0000 7fd977653740 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.976+0000 7fd977653740 1 --2- 172.21.15.28:0/279632480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0ae090 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.976+0000 7fd977653740 1 -- 172.21.15.28:0/279632480 >> 172.21.15.28:0/279632480 conn(0x55ad2c077310 msgr2=0x55ad2c0aac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.977+0000 7fd977653740 1 -- 172.21.15.28:0/279632480 shutdown_connections 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.977+0000 7fd977653740 1 -- 172.21.15.28:0/279632480 wait complete. 2024-09-15T22:03:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.977+0000 7fd977653740 1 -- start start 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.978+0000 7fd977653740 1 Processor -- start 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.978+0000 7fd977653740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd977653740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd96effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd96effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35252/0 (socket says 172.21.15.28:35252) 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd977653740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd977653740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ad2c0bc070 con 0x55ad2c0ae5d0 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd977653740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ad2bfcf060 con 0x55ad2c0b9370 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd977653740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ad2c0bc1f0 con 0x55ad2c0afb40 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd96effd640 1 -- 172.21.15.28:0/1268181358 learned_addr learned my addr 172.21.15.28:0/1268181358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd977653740 10 client.0 _ll_register_callbacks cb 0x55ad2bd228c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.979+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd96e7fc640 1 -- 172.21.15.28:0/1268181358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 msgr2=0x55ad2c0b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd96e7fc640 1 -- 172.21.15.28:0/1268181358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 msgr2=0x55ad2c0b8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd96e7fc640 1 -- 172.21.15.28:0/1268181358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad2bd853c0 con 0x55ad2c0afb40 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd96effd640 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b6030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c5e0 tx=0x7fd960006e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2448c9383c027ab4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.980+0000 7fd952ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd952ff5640 1 -- 172.21.15.28:0/1268181358 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960027610 con 0x55ad2c0afb40 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd960019070 con 0x55ad2c0afb40 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd952ff5640 1 -- 172.21.15.28:0/1268181358 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd9600277b0 con 0x55ad2c0afb40 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd952ff5640 1 -- 172.21.15.28:0/1268181358 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd960030050 con 0x55ad2c0afb40 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd977653740 10 client.25900 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad2c0c7b20 con 0x55ad2c0afb40 2024-09-15T22:03:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd952ff5640 1 -- 172.21.15.28:0/1268181358 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96000a610 con 0x55ad2c0afb40 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.981+0000 7fd952ff5640 1 -- 172.21.15.28:0/1268181358 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd960013410 con 0x55ad2c0afb40 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.982+0000 7fd977653740 10 client.25900 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.982+0000 7fd977653740 10 client.25900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.982+0000 7fd977653740 20 client.25900 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.982+0000 7fd977653740 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c0c92d0 0x55ad2c0cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.982+0000 7fd977653740 4 client.25900 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.982+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd960019070 con 0x55ad2c0c92d0 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.983+0000 7fd96effd640 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c0c92d0 0x55ad2c0cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.992+0000 7fd96effd640 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c0c92d0 0x55ad2c0cb730 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf59ed65747cf505 server_cookie=5dafd7268fee8cf9 in_seq=0 out_seq=0 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.993+0000 7fd952ff5640 10 client.25900 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.994+0000 7fd952ff5640 1 -- 172.21.15.28:0/1268181358 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fd960019070 con 0x55ad2c0c92d0 2024-09-15T22:03:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.994+0000 7fd952ff5640 10 client.25900 handle_command_reply: tid=0 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:01.994+0000 7fd977653740 1 client.25900 shutdown 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.002+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 msgr2=0x55ad2c0b6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.002+0000 7fd977653740 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b6030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c5e0 tx=0x7fd960006e40 comp rx=0 tx=0).stop 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.003+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 shutdown_connections 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.003+0000 7fd977653740 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c0c92d0 0x55ad2c0cb730 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.003+0000 7fd977653740 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.003+0000 7fd977653740 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b6030 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.003+0000 7fd977653740 1 --2- 172.21.15.28:0/1268181358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.003+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 >> 172.21.15.28:0/1268181358 conn(0x55ad2c077310 msgr2=0x55ad2c0b0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.003+0000 7fd952ff5640 0 client.25900 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.005+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 shutdown_connections 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.005+0000 7fd977653740 1 -- 172.21.15.28:0/1268181358 wait complete. 2024-09-15T22:03:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.005+0000 7fd977653740 20 client.25900 trim_cache size 0 max 16384 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 Processor -- start 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 -- start start 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c00fec0 con 0x55ad2c0ae5d0 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c011a00 con 0x55ad2c0afb40 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd977653740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c010590 con 0x55ad2c0b9370 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd96e7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd96f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd96f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:53400/0 (socket says 172.21.15.28:53400) 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd96f7fe640 1 -- 172.21.15.28:0/219288436 learned_addr learned my addr 172.21.15.28:0/219288436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.035+0000 7fd96effd640 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.036+0000 7fd96effd640 1 -- 172.21.15.28:0/219288436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0b9370 msgr2=0x55ad2c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.036+0000 7fd96effd640 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.036+0000 7fd96effd640 1 -- 172.21.15.28:0/219288436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0afb40 msgr2=0x55ad2c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.036+0000 7fd96effd640 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.036+0000 7fd96effd640 1 -- 172.21.15.28:0/219288436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad2c0b69b0 con 0x55ad2c0ae5d0 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.037+0000 7fd96effd640 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b7fa0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x55ad2c0cbf80 tx=0x7fd968012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189a40c18bba1752 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.038+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.038+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.039+0000 7fd954ff9640 1 -- 172.21.15.28:0/219288436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968016a00 con 0x55ad2c0ae5d0 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.039+0000 7fd954ff9640 1 -- 172.21.15.28:0/219288436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd968028030 con 0x55ad2c0ae5d0 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.040+0000 7fd977653740 1 -- 172.21.15.28:0/219288436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 msgr2=0x55ad2c0b7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.040+0000 7fd977653740 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b7fa0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x55ad2c0cbf80 tx=0x7fd968012000 comp rx=0 tx=0).stop 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.041+0000 7fd977653740 1 -- 172.21.15.28:0/219288436 shutdown_connections 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.041+0000 7fd977653740 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0b9370 0x55ad2c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.041+0000 7fd977653740 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0afb40 0x55ad2c0b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.041+0000 7fd977653740 1 --2- 172.21.15.28:0/219288436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0b7fa0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.041+0000 7fd977653740 1 -- 172.21.15.28:0/219288436 >> 172.21.15.28:0/219288436 conn(0x55ad2c077310 msgr2=0x55ad2c0b7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.041+0000 7fd977653740 1 -- 172.21.15.28:0/219288436 shutdown_connections 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.041+0000 7fd977653740 1 -- 172.21.15.28:0/219288436 wait complete. 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 -- start start 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 Processor -- start 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0bc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0bca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c011330 con 0x55ad2c0b9370 2024-09-15T22:03:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c010c60 con 0x55ad2c0ae5d0 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd95c00dea0 con 0x55ad2c0afb40 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.042+0000 7fd977653740 10 client.0 _ll_register_callbacks cb 0x7fd96000a960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0bc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96effd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0bc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:45814/0 (socket says 172.21.15.28:45814) 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96effd640 1 -- 172.21.15.28:0/1774281666 learned_addr learned my addr 172.21.15.28:0/1774281666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96e7fc640 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0bca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96f7fe640 1 -- 172.21.15.28:0/1774281666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 msgr2=0x55ad2c0bca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0bca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96f7fe640 1 -- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0ae5d0 msgr2=0x55ad2c0bc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0bc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.043+0000 7fd96f7fe640 1 -- 172.21.15.28:0/1774281666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad2c132100 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.044+0000 7fd96effd640 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0bc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.044+0000 7fd96f7fe640 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bcf50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd9640084c0 tx=0x7fd964005130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5e699723568f3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.044+0000 7fd952ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.044+0000 7fd952ff5640 1 -- 172.21.15.28:0/1774281666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9640d5060 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.044+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd968028030 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.044+0000 7fd952ff5640 1 -- 172.21.15.28:0/1774281666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd9640bdd80 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.045+0000 7fd952ff5640 1 -- 172.21.15.28:0/1774281666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd9640ba990 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.045+0000 7fd977653740 10 client.16377 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.045+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ad2c133450 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.045+0000 7fd952ff5640 1 -- 172.21.15.28:0/1774281666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd968028030 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd952ff5640 1 -- 172.21.15.28:0/1774281666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd9640bbca0 con 0x55ad2c0b9370 2024-09-15T22:03:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd977653740 10 client.16377 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd977653740 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd977653740 20 client.16377 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd977653740 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c133ad0 0x55ad2c135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd977653740 4 client.16377 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:03:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad2c021860 con 0x55ad2c133ad0 2024-09-15T22:03:03.698 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.046+0000 7fd96effd640 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c133ad0 0x55ad2c135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.048+0000 7fd96effd640 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c133ad0 0x55ad2c135ef0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf90bbdc6ffaf4ed server_cookie=3b63f08d4efd76a in_seq=0 out_seq=0 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.048+0000 7fd952ff5640 10 client.16377 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.048+0000 7fd952ff5640 1 -- 172.21.15.28:0/1774281666 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad2c021860 con 0x55ad2c133ad0 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.048+0000 7fd952ff5640 10 client.16377 handle_command_reply: tid=0 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.048+0000 7fd977653740 1 client.16377 shutdown 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.048+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0b9370 msgr2=0x55ad2c0bcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.048+0000 7fd977653740 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bcf50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd9640084c0 tx=0x7fd964005130 comp rx=0 tx=0).stop 2024-09-15T22:03:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.051+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 shutdown_connections 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.051+0000 7fd977653740 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ad2c133ad0 0x55ad2c135ef0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd977653740 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad2c0b9370 0x55ad2c0bcf50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd977653740 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ad2c0afb40 0x55ad2c0bca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd977653740 1 --2- 172.21.15.28:0/1774281666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad2c0ae5d0 0x55ad2c0bc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 >> 172.21.15.28:0/1774281666 conn(0x55ad2c077310 msgr2=0x55ad2c0b7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd952ff5640 0 client.16377 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 shutdown_connections 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd977653740 1 -- 172.21.15.28:0/1774281666 wait complete. 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.052+0000 7fd977653740 20 client.16377 trim_cache size 0 max 16384 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9580a3500 msgr2=0x7fd9580a59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9580a3500 0x7fd9580a59c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55ad2c0abac0 tx=0x7fd96800b040 comp rx=0 tx=0).stop 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd97010ea80 msgr2=0x7fd970118bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd97010ea80 0x7fd970118bd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd964037040 tx=0x7fd96400a980 comp rx=0 tx=0).stop 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 shutdown_connections 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9580a3500 0x7fd9580a59c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd97010ea80 0x7fd970118bd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd970109e00 0x7fd970118690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 --2- 172.21.15.28:0/4015533867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd970109450 0x7fd97011f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.053+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 >> 172.21.15.28:0/4015533867 conn(0x7fd97006e2d0 msgr2=0x7fd97010c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.054+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 shutdown_connections 2024-09-15T22:03:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:02.054+0000 7fd97527a640 1 -- 172.21.15.28:0/4015533867 wait complete. 2024-09-15T22:03:03.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:03 smithi028 ceph-mon[30969]: pgmap v174: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 20 KiB/s rd, 38 MiB/s wr, 6.47k op/s 2024-09-15T22:03:03.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:03 smithi203 ceph-mon[35546]: pgmap v174: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 20 KiB/s rd, 38 MiB/s wr, 6.47k op/s 2024-09-15T22:03:04.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:03 smithi174 ceph-mon[35244]: pgmap v174: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 20 KiB/s rd, 38 MiB/s wr, 6.47k op/s 2024-09-15T22:03:05.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:05 smithi028 ceph-mon[30969]: pgmap v175: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 32 MiB/s wr, 5.49k op/s 2024-09-15T22:03:05.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:05 smithi203 ceph-mon[35546]: pgmap v175: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 32 MiB/s wr, 5.49k op/s 2024-09-15T22:03:06.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:05 smithi174 ceph-mon[35244]: pgmap v175: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 32 MiB/s wr, 5.49k op/s 2024-09-15T22:03:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:07 smithi028 ceph-mon[30969]: pgmap v176: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1022 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 34 MiB/s wr, 5.76k op/s 2024-09-15T22:03:07.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:07 smithi203 ceph-mon[35546]: pgmap v176: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1022 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 34 MiB/s wr, 5.76k op/s 2024-09-15T22:03:08.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:07 smithi174 ceph-mon[35244]: pgmap v176: 129 pgs: 129 active+clean; 18 GiB data, 50 GiB used, 1022 GiB / 1.0 TiB avail; 9.7 KiB/s rd, 34 MiB/s wr, 5.76k op/s 2024-09-15T22:03:09.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:09 smithi028 ceph-mon[30969]: pgmap v177: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 19 KiB/s rd, 34 MiB/s wr, 5.77k op/s 2024-09-15T22:03:09.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:09 smithi203 ceph-mon[35546]: pgmap v177: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 19 KiB/s rd, 34 MiB/s wr, 5.77k op/s 2024-09-15T22:03:10.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:09 smithi174 ceph-mon[35244]: pgmap v177: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 19 KiB/s rd, 34 MiB/s wr, 5.77k op/s 2024-09-15T22:03:11.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.101+0000 7fc7daff5640 1 -- 172.21.15.28:0/1168446676 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5591aac08d70 con 0x5591aad22b30 2024-09-15T22:03:11.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.101+0000 7fc7daff5640 10 client.25891 handle_command_reply: tid=0 2024-09-15T22:03:11.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.101+0000 7fc7ffe8d740 1 client.25891 shutdown 2024-09-15T22:03:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aaca7cb0 msgr2=0x5591aaca88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aaca7cb0 0x5591aaca88b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc7f001f6f0 tx=0x7fc7f00123f0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 shutdown_connections 2024-09-15T22:03:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5591aad22b30 0x5591aad24f90 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591aaca7cb0 0x5591aaca88b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591aac9d410 0x5591aacb0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 --2- 172.21.15.28:0/1168446676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591aac9caf0 0x5591aacaffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 >> 172.21.15.28:0/1168446676 conn(0x5591aac99060 msgr2=0x5591aac979a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7daff5640 0 client.25891 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 shutdown_connections 2024-09-15T22:03:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 1 -- 172.21.15.28:0/1168446676 wait complete. 2024-09-15T22:03:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.102+0000 7fc7ffe8d740 20 client.25891 trim_cache size 0 max 16384 2024-09-15T22:03:11.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:03:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:03:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:03:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7e00a3460 msgr2=0x7fc7e00a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7e00a3460 0x7fc7e00a5920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009fd0 tx=0x7fc7f0009500 comp rx=0 tx=0).stop 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 msgr2=0x7fc7f810e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 0x7fc7f810e310 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8009230 tx=0x7fc7e8004040 comp rx=0 tx=0).stop 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 shutdown_connections 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7e00a3460 0x7fc7e00a5920 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7f81069c0 0x7fc7f810ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7f8101350 0x7fc7f810e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 --2- 172.21.15.28:0/992889603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81009b0 0x7fc7f810e310 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.103+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 >> 172.21.15.28:0/992889603 conn(0x7fc7f8077990 msgr2=0x7fc7f8103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.104+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 shutdown_connections 2024-09-15T22:03:11.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:11.104+0000 7fc7fd01e640 1 -- 172.21.15.28:0/992889603 wait complete. 2024-09-15T22:03:12.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:11 smithi028 ceph-mon[30969]: pgmap v178: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 16 KiB/s rd, 30 MiB/s wr, 5.16k op/s 2024-09-15T22:03:12.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:11 smithi203 ceph-mon[35546]: pgmap v178: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 16 KiB/s rd, 30 MiB/s wr, 5.16k op/s 2024-09-15T22:03:12.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:11 smithi174 ceph-mon[35244]: pgmap v178: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 16 KiB/s rd, 30 MiB/s wr, 5.16k op/s 2024-09-15T22:03:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:13 smithi028 ceph-mon[30969]: pgmap v179: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1021 GiB / 1.0 TiB avail; 19 KiB/s rd, 34 MiB/s wr, 5.99k op/s 2024-09-15T22:03:14.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:13 smithi203 ceph-mon[35546]: pgmap v179: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1021 GiB / 1.0 TiB avail; 19 KiB/s rd, 34 MiB/s wr, 5.99k op/s 2024-09-15T22:03:14.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:13 smithi174 ceph-mon[35244]: pgmap v179: 129 pgs: 129 active+clean; 18 GiB data, 51 GiB used, 1021 GiB / 1.0 TiB avail; 19 KiB/s rd, 34 MiB/s wr, 5.99k op/s 2024-09-15T22:03:16.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:15 smithi028 ceph-mon[30969]: pgmap v180: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-09-15T22:03:16.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:15 smithi203 ceph-mon[35546]: pgmap v180: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-09-15T22:03:16.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:15 smithi174 ceph-mon[35244]: pgmap v180: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-09-15T22:03:17.203 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T22:03:17.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T22:03:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:03:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f926affd640 1 -- 172.21.15.28:0/762460303 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: -115 ) ==== 8+0+593 (crc 0 0 0) 0x55bf6babebe0 con 0x55bf6bbd8ad0 2024-09-15T22:03:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f926affd640 10 client.25892 handle_command_reply: tid=0 2024-09-15T22:03:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 client.25892 shutdown 2024-09-15T22:03:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 msgr2=0x55bf6bb65f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb65f40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9280009d90 tx=0x7f92800168c0 comp rx=0 tx=0).stop 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 shutdown_connections 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55bf6bbd8ad0 0x55bf6bbdaef0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bf6bb5da30 0x55bf6bb57c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6bb53170 0x55bf6bb66480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 --2- 172.21.15.28:0/762460303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf6bb52850 0x55bf6bb65f40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.783+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 >> 172.21.15.28:0/762460303 conn(0x55bf6bb4ede0 msgr2=0x55bf6bb4d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.784+0000 7f926affd640 0 client.25892 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.784+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 shutdown_connections 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.784+0000 7f9293dd6740 1 -- 172.21.15.28:0/762460303 wait complete. 2024-09-15T22:03:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:17.784+0000 7f9293dd6740 20 client.25892 trim_cache size 0 max 16384 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 32, 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 9, 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "ebf3369a": { 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 9, 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:03:17.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCING", 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.0 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 90.0, 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 139.6, 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCING", 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 45.0 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:03:17.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:03:17.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:03:17.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:03:17.791 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:03:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINPROGRESS: 2024-09-15T22:03:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:17 smithi028 ceph-mon[30969]: pgmap v181: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 4.57k op/s 2024-09-15T22:03:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:17 smithi203 ceph-mon[35546]: pgmap v181: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 4.57k op/s 2024-09-15T22:03:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:17 smithi174 ceph-mon[35244]: pgmap v181: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 4.57k op/s 2024-09-15T22:03:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:19 smithi028 ceph-mon[30969]: pgmap v182: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 42 KiB/s rd, 28 MiB/s wr, 4.82k op/s 2024-09-15T22:03:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:19 smithi203 ceph-mon[35546]: pgmap v182: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 42 KiB/s rd, 28 MiB/s wr, 4.82k op/s 2024-09-15T22:03:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:19 smithi174 ceph-mon[35244]: pgmap v182: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 42 KiB/s rd, 28 MiB/s wr, 4.82k op/s 2024-09-15T22:03:22.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:21 smithi028 ceph-mon[30969]: pgmap v183: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 4.27k op/s 2024-09-15T22:03:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:21 smithi203 ceph-mon[35546]: pgmap v183: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 4.27k op/s 2024-09-15T22:03:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:21 smithi174 ceph-mon[35244]: pgmap v183: 129 pgs: 129 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 32 KiB/s rd, 24 MiB/s wr, 4.27k op/s 2024-09-15T22:03:24.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:23 smithi028 ceph-mon[30969]: pgmap v184: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 56 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-09-15T22:03:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:23 smithi203 ceph-mon[35546]: pgmap v184: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 56 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-09-15T22:03:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.257+0000 7f17944ba640 1 -- 172.21.15.28:0/5740119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1095d0 msgr2=0x7f178c109a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.257+0000 7f17944ba640 1 --2- 172.21.15.28:0/5740119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1095d0 0x7f178c109a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1784009930 tx=0x7f178402a420 comp rx=0 tx=0).stop 2024-09-15T22:03:24.259 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:23 smithi174 ceph-mon[35244]: pgmap v184: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 56 KiB/s rd, 28 MiB/s wr, 4.87k op/s 2024-09-15T22:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.258+0000 7f17944ba640 1 -- 172.21.15.28:0/5740119 shutdown_connections 2024-09-15T22:03:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.258+0000 7f17944ba640 1 --2- 172.21.15.28:0/5740119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178c11b140 0x7f178c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.258+0000 7f17944ba640 1 --2- 172.21.15.28:0/5740119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1095d0 0x7f178c109a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.258+0000 7f17944ba640 1 --2- 172.21.15.28:0/5740119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f178c1091f0 0x7f178c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.258+0000 7f17944ba640 1 -- 172.21.15.28:0/5740119 >> 172.21.15.28:0/5740119 conn(0x7f178c06ce00 msgr2=0x7f178c06d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.259+0000 7f17944ba640 1 -- 172.21.15.28:0/5740119 shutdown_connections 2024-09-15T22:03:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.259+0000 7f17944ba640 1 -- 172.21.15.28:0/5740119 wait complete. 2024-09-15T22:03:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.259+0000 7f17944ba640 1 Processor -- start 2024-09-15T22:03:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f17944ba640 1 -- start start 2024-09-15T22:03:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f17944ba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1091f0 0x7f178c118a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f17944ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178c1095d0 0x7f178c1109f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f17944ba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f178c11b140 0x7f178c110f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f17944ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f178c120cf0 con 0x7f178c1095d0 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f17944ba640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f178c120b70 con 0x7f178c11b140 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f17944ba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f178c120e70 con 0x7f178c1091f0 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f1792256640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1091f0 0x7f178c118a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f1792256640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1091f0 0x7f178c118a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58892/0 (socket says 172.21.15.28:58892) 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f1792256640 1 -- 172.21.15.28:0/1430191754 learned_addr learned my addr 172.21.15.28:0/1430191754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.260+0000 7f1792a57640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f178c11b140 0x7f178c110f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.261+0000 7f1792256640 1 -- 172.21.15.28:0/1430191754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f178c11b140 msgr2=0x7f178c110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.261+0000 7f1792256640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f178c11b140 0x7f178c110f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.261+0000 7f1792256640 1 -- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178c1095d0 msgr2=0x7f178c1109f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.261+0000 7f1792256640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178c1095d0 0x7f178c1109f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.261+0000 7f1792256640 1 -- 172.21.15.28:0/1430191754 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f178c111750 con 0x7f178c1091f0 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.261+0000 7f1792256640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1091f0 0x7f178c118a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f178800eb90 tx=0x7f178800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.261+0000 7f17837fe640 1 -- 172.21.15.28:0/1430191754 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788004d20 con 0x7f178c1091f0 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.262+0000 7f17837fe640 1 -- 172.21.15.28:0/1430191754 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f178800ee90 con 0x7f178c1091f0 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.262+0000 7f17837fe640 1 -- 172.21.15.28:0/1430191754 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788010830 con 0x7f178c1091f0 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.262+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f178c1bbf50 con 0x7f178c1091f0 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.262+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f178c1bc3c0 con 0x7f178c1091f0 2024-09-15T22:03:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.265+0000 7f17837fe640 1 -- 172.21.15.28:0/1430191754 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f17880109d0 con 0x7f178c1091f0 2024-09-15T22:03:24.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.267+0000 7f1797255740 1 Processor -- start 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.267+0000 7f1797255740 1 -- start start 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1797255740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1797255740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1797255740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddd10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1797255740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff8dd282f0 con 0x55ff8ddc3af0 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1797255740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ff8dcd0270 con 0x55ff8ddc4410 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1797255740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ff8dd242b0 con 0x55ff8ddcecb0 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f17837fe640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17780a34b0 0x7f17780a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1791a55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddd10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f17837fe640 1 -- 172.21.15.28:0/1430191754 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f1788005020 con 0x7f178c1091f0 2024-09-15T22:03:24.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1792a57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1792a57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55562/0 (socket says 172.21.15.28:55562) 2024-09-15T22:03:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1792a57640 1 -- 172.21.15.28:0/670170013 learned_addr learned my addr 172.21.15.28:0/670170013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.268+0000 7f1792a57640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17780a34b0 0x7f17780a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.269+0000 7f1792256640 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.270+0000 7f1792256640 1 -- 172.21.15.28:0/670170013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddcecb0 msgr2=0x55ff8ddd10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.270+0000 7f1792256640 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddd10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.270+0000 7f1792256640 1 -- 172.21.15.28:0/670170013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddc3af0 msgr2=0x55ff8ddc3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.270+0000 7f1792256640 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.270+0000 7f1792256640 1 -- 172.21.15.28:0/670170013 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff8dccf340 con 0x55ff8ddc4410 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.270+0000 7f1792256640 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddce770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f17880ceb80 tx=0x7f178809d250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f84c327043d9f6d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.271+0000 7f1780bfe640 1 -- 172.21.15.28:0/670170013 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17880b5070 con 0x55ff8ddc4410 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.271+0000 7f1780bfe640 1 -- 172.21.15.28:0/670170013 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f178809d710 con 0x55ff8ddc4410 2024-09-15T22:03:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.271+0000 7f1780bfe640 1 -- 172.21.15.28:0/670170013 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788099d80 con 0x55ff8ddc4410 2024-09-15T22:03:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.272+0000 7f1797255740 1 -- 172.21.15.28:0/670170013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 msgr2=0x55ff8ddce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.272+0000 7f1797255740 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddce770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f17880ceb80 tx=0x7f178809d250 comp rx=0 tx=0).stop 2024-09-15T22:03:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.273+0000 7f1792a57640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17780a34b0 0x7f17780a5970 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f177c011980 tx=0x7f177c005f90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.273+0000 7f1797255740 1 -- 172.21.15.28:0/670170013 shutdown_connections 2024-09-15T22:03:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.273+0000 7f1797255740 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddd10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.274+0000 7f1797255740 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddce770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.274+0000 7f1797255740 1 --2- 172.21.15.28:0/670170013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.274+0000 7f1797255740 1 -- 172.21.15.28:0/670170013 >> 172.21.15.28:0/670170013 conn(0x55ff8ddc0060 msgr2=0x55ff8ddc0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.274+0000 7f1797255740 1 -- 172.21.15.28:0/670170013 shutdown_connections 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.274+0000 7f1797255740 1 -- 172.21.15.28:0/670170013 wait complete. 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 -- start start 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 Processor -- start 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc4410 0x55ff8ddcc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddcc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff8ddd1880 con 0x55ff8ddcecb0 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ff8dcb1290 con 0x55ff8ddc3af0 2024-09-15T22:03:24.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ff8ddd1a00 con 0x55ff8ddc4410 2024-09-15T22:03:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.275+0000 7f1797255740 10 client.0 _ll_register_callbacks cb 0x55ff8dce2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.276+0000 7f1792256640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.276+0000 7f1791a55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc4410 0x55ff8ddcc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.276+0000 7f1791a55640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc4410 0x55ff8ddcc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58916/0 (socket says 172.21.15.28:58916) 2024-09-15T22:03:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.276+0000 7f1791a55640 1 -- 172.21.15.28:0/1066254518 learned_addr learned my addr 172.21.15.28:0/1066254518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1792256640 1 -- 172.21.15.28:0/1066254518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc4410 msgr2=0x55ff8ddcc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1792256640 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc4410 0x55ff8ddcc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1792256640 1 -- 172.21.15.28:0/1066254518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 msgr2=0x55ff8ddcc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1792256640 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddcc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1792256640 1 -- 172.21.15.28:0/1066254518 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff8da3b210 con 0x55ff8ddc3af0 2024-09-15T22:03:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1792256640 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcbbc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55ff8ddc4a30 tx=0x7f1788005b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ba1df2ef201473c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1775ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1775ffb640 1 -- 172.21.15.28:0/1066254518 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17880b5070 con 0x55ff8ddc3af0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.277+0000 7f1775ffb640 1 -- 172.21.15.28:0/1066254518 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f17880a5ef0 con 0x55ff8ddc3af0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.278+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f178809d710 con 0x55ff8ddc3af0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.278+0000 7f1775ffb640 1 -- 172.21.15.28:0/1066254518 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f17880a1ac0 con 0x55ff8ddc3af0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.278+0000 7f1797255740 10 client.25946 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.278+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ff8db07620 con 0x55ff8ddc3af0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.278+0000 7f1775ffb640 1 -- 172.21.15.28:0/1066254518 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17880a60f0 con 0x55ff8ddc3af0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.278+0000 7f1775ffb640 1 -- 172.21.15.28:0/1066254518 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f178807d060 con 0x55ff8ddc3af0 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.279+0000 7f1797255740 10 client.25946 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.279+0000 7f1797255740 10 client.25946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:24.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.279+0000 7f1797255740 20 client.25946 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.279+0000 7f1797255740 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8dddeab0 0x55ff8dde0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.279+0000 7f1797255740 4 client.25946 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.279+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f178809d710 con 0x55ff8dddeab0 2024-09-15T22:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.280+0000 7f1791a55640 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8dddeab0 0x55ff8dde0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.282+0000 7f1775ffb640 10 client.25946 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.282+0000 7f1791a55640 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8dddeab0 0x55ff8dde0ea0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=633c372effa6ae9f server_cookie=2b13506755bae3e7 in_seq=0 out_seq=0 2024-09-15T22:03:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1775ffb640 1 -- 172.21.15.28:0/1066254518 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f178809d710 con 0x55ff8dddeab0 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1775ffb640 10 client.25946 handle_command_reply: tid=0 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 client.25946 shutdown 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc3af0 msgr2=0x55ff8ddcbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcbbc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55ff8ddc4a30 tx=0x7f1788005b40 comp rx=0 tx=0).stop 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 shutdown_connections 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8dddeab0 0x55ff8dde0ea0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1775ffb640 0 client.25946 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddcc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc4410 0x55ff8ddcc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 --2- 172.21.15.28:0/1066254518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcbbc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 >> 172.21.15.28:0/1066254518 conn(0x55ff8ddc0060 msgr2=0x55ff8ddc5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.284+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 shutdown_connections 2024-09-15T22:03:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.285+0000 7f1797255740 1 -- 172.21.15.28:0/1066254518 wait complete. 2024-09-15T22:03:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.285+0000 7f1797255740 20 client.25946 trim_cache size 0 max 16384 2024-09-15T22:03:24.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.315+0000 7f1797255740 1 Processor -- start 2024-09-15T22:03:24.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.315+0000 7f1797255740 1 -- start start 2024-09-15T22:03:24.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.315+0000 7f1797255740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1797255740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8dde6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1797255740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8dde6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1797255740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f178c1c6640 con 0x55ff8ddcecb0 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1792256640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1791a55640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8dde6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1792256640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58932/0 (socket says 172.21.15.28:58932) 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1791a55640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8dde6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:42960/0 (socket says 172.21.15.28:42960) 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1792256640 1 -- 172.21.15.28:0/2303529376 learned_addr learned my addr 172.21.15.28:0/2303529376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1797255740 1 -- 172.21.15.28:0/2303529376 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f178c1c8180 con 0x55ff8ddc4410 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1797255740 1 -- 172.21.15.28:0/2303529376 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f178c1c6d10 con 0x55ff8ddc3af0 2024-09-15T22:03:24.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1792a57640 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8dde6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1791a55640 1 -- 172.21.15.28:0/2303529376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 msgr2=0x55ff8ddcc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1791a55640 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1791a55640 1 -- 172.21.15.28:0/2303529376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 msgr2=0x55ff8dde6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1791a55640 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8dde6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.316+0000 7f1791a55640 1 -- 172.21.15.28:0/2303529376 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff8ddbb2d0 con 0x55ff8ddc4410 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.317+0000 7f1791a55640 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8dde6550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55ff8ddcb430 tx=0x7f1784004350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98eec22a62624833 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.317+0000 7f1777fff640 1 -- 172.21.15.28:0/2303529376 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784049070 con 0x55ff8ddc4410 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.317+0000 7f1777fff640 1 -- 172.21.15.28:0/2303529376 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1784004440 con 0x55ff8ddc4410 2024-09-15T22:03:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.317+0000 7f1777fff640 1 -- 172.21.15.28:0/2303529376 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784035040 con 0x55ff8ddc4410 2024-09-15T22:03:24.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 -- 172.21.15.28:0/2303529376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 msgr2=0x55ff8dde6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8dde6550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55ff8ddcb430 tx=0x7f1784004350 comp rx=0 tx=0).stop 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 -- 172.21.15.28:0/2303529376 shutdown_connections 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8dde6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8dde6550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 --2- 172.21.15.28:0/2303529376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddcc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 -- 172.21.15.28:0/2303529376 >> 172.21.15.28:0/2303529376 conn(0x55ff8ddc0060 msgr2=0x55ff8ddcdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 -- 172.21.15.28:0/2303529376 shutdown_connections 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.318+0000 7f1797255740 1 -- 172.21.15.28:0/2303529376 wait complete. 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.319+0000 7f1797255740 1 -- start start 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.319+0000 7f1797255740 1 Processor -- start 2024-09-15T22:03:24.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.320+0000 7f1797255740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.320+0000 7f1797255740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddc9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.320+0000 7f1797255740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddc97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.320+0000 7f1792256640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.320+0000 7f1792256640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58934/0 (socket says 172.21.15.28:58934) 2024-09-15T22:03:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1792256640 1 -- 172.21.15.28:0/8856244 learned_addr learned my addr 172.21.15.28:0/8856244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1791a55640 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddc9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f178c1c7ab0 con 0x55ff8ddcecb0 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f178c1c73e0 con 0x55ff8ddc4410 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f178c1c4620 con 0x55ff8ddc3af0 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1797255740 10 client.0 _ll_register_callbacks cb 0x7f177c009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1792a57640 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddc97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1791a55640 1 -- 172.21.15.28:0/8856244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 msgr2=0x55ff8ddc8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1791a55640 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1791a55640 1 -- 172.21.15.28:0/8856244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 msgr2=0x55ff8ddc97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1791a55640 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddc97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1791a55640 1 -- 172.21.15.28:0/8856244 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff8de486c0 con 0x55ff8ddc4410 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1791a55640 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddc9280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55ff8ddcb430 tx=0x7f1784005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2734b00b4b533cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.321+0000 7f1775ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:03:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1775ffb640 1 -- 172.21.15.28:0/8856244 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784049070 con 0x55ff8ddc4410 2024-09-15T22:03:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1784004440 con 0x55ff8ddc4410 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1775ffb640 1 -- 172.21.15.28:0/8856244 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f17840317c0 con 0x55ff8ddc4410 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1775ffb640 1 -- 172.21.15.28:0/8856244 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1784035040 con 0x55ff8ddc4410 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1797255740 10 client.25952 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ff8de49970 con 0x55ff8ddc4410 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1775ffb640 1 -- 172.21.15.28:0/8856244 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1784004440 con 0x55ff8ddc4410 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.322+0000 7f1775ffb640 1 -- 172.21.15.28:0/8856244 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f178403a040 con 0x55ff8ddc4410 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.323+0000 7f1797255740 10 client.25952 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.323+0000 7f1797255740 10 client.25952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.323+0000 7f1797255740 20 client.25952 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:24.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.323+0000 7f1797255740 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8de4a030 0x55ff8de4c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.323+0000 7f1797255740 4 client.25952 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:03:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.323+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ff8dd35a20 con 0x55ff8de4a030 2024-09-15T22:03:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.324+0000 7f1792256640 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8de4a030 0x55ff8de4c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.325+0000 7f1792256640 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8de4a030 0x55ff8de4c450 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9fd5061835e805d server_cookie=c8a52aa48493df5a in_seq=0 out_seq=0 2024-09-15T22:03:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:24.326+0000 7f1775ffb640 10 client.25952 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:25 smithi028 ceph-mon[30969]: pgmap v185: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 53 KiB/s rd, 20 MiB/s wr, 3.53k op/s 2024-09-15T22:03:26.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:25 smithi203 ceph-mon[35546]: pgmap v185: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 53 KiB/s rd, 20 MiB/s wr, 3.53k op/s 2024-09-15T22:03:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:25 smithi174 ceph-mon[35244]: pgmap v185: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 53 KiB/s rd, 20 MiB/s wr, 3.53k op/s 2024-09-15T22:03:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1775ffb640 1 -- 172.21.15.28:0/8856244 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ff8dd35a20 con 0x55ff8de4a030 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1775ffb640 10 client.25952 handle_command_reply: tid=0 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 client.25952 shutdown 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 msgr2=0x55ff8ddc9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddc9280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55ff8ddcb430 tx=0x7f1784005be0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 shutdown_connections 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ff8de4a030 0x55ff8de4c450 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff8ddcecb0 0x55ff8ddc97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1775ffb640 0 client.25952 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ff8ddc4410 0x55ff8ddc9280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 --2- 172.21.15.28:0/8856244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ff8ddc3af0 0x55ff8ddc8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.395+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 >> 172.21.15.28:0/8856244 conn(0x55ff8ddc0060 msgr2=0x55ff8ddcd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 shutdown_connections 2024-09-15T22:03:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f1797255740 1 -- 172.21.15.28:0/8856244 wait complete. 2024-09-15T22:03:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f1797255740 20 client.25952 trim_cache size 0 max 16384 2024-09-15T22:03:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:03:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:03:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:03:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:03:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17780a34b0 msgr2=0x7f17780a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17780a34b0 0x7f17780a5970 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f177c011980 tx=0x7f177c005f90 comp rx=0 tx=0).stop 2024-09-15T22:03:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1091f0 msgr2=0x7f178c118a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1091f0 0x7f178c118a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f178800eb90 tx=0x7f178800c6a0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 shutdown_connections 2024-09-15T22:03:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f17780a34b0 0x7f17780a5970 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f178c11b140 0x7f178c110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.396+0000 7f17944ba640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f178c1095d0 0x7f178c1109f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.397+0000 7f17944ba640 1 --2- 172.21.15.28:0/1430191754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f178c1091f0 0x7f178c118a30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.397+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 >> 172.21.15.28:0/1430191754 conn(0x7f178c06ce00 msgr2=0x7f178c11b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.397+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 shutdown_connections 2024-09-15T22:03:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:27.397+0000 7f17944ba640 1 -- 172.21.15.28:0/1430191754 wait complete. 2024-09-15T22:03:28.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:27 smithi028 ceph-mon[30969]: pgmap v186: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 54 KiB/s rd, 16 MiB/s wr, 2.75k op/s 2024-09-15T22:03:28.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:27 smithi203 ceph-mon[35546]: pgmap v186: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 54 KiB/s rd, 16 MiB/s wr, 2.75k op/s 2024-09-15T22:03:28.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:27 smithi174 ceph-mon[35244]: pgmap v186: 129 pgs: 129 active+clean; 18 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 54 KiB/s rd, 16 MiB/s wr, 2.75k op/s 2024-09-15T22:03:30.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:29 smithi028 ceph-mon[30969]: pgmap v187: 129 pgs: 129 active+clean; 19 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 85 KiB/s rd, 22 MiB/s wr, 3.80k op/s 2024-09-15T22:03:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:29 smithi203 ceph-mon[35546]: pgmap v187: 129 pgs: 129 active+clean; 19 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 85 KiB/s rd, 22 MiB/s wr, 3.80k op/s 2024-09-15T22:03:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:29 smithi174 ceph-mon[35244]: pgmap v187: 129 pgs: 129 active+clean; 19 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 85 KiB/s rd, 22 MiB/s wr, 3.80k op/s 2024-09-15T22:03:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 115 2024-09-15T22:03:31.478 WARNING:tasks.quiescer.fs.[cephfs]:Set 'ebf3369a' hasn't quiesced after 124.4 seconds (timeout: 90). Dumping ops with locks from all ranks. 2024-09-15T22:03:31.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:03:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:31 smithi028 ceph-mon[30969]: pgmap v188: 129 pgs: 129 active+clean; 19 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 57 KiB/s rd, 18 MiB/s wr, 3.21k op/s 2024-09-15T22:03:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:31 smithi203 ceph-mon[35546]: pgmap v188: 129 pgs: 129 active+clean; 19 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 57 KiB/s rd, 18 MiB/s wr, 3.21k op/s 2024-09-15T22:03:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:31 smithi174 ceph-mon[35244]: pgmap v188: 129 pgs: 129 active+clean; 19 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 57 KiB/s rd, 18 MiB/s wr, 3.21k op/s 2024-09-15T22:03:34.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: pgmap v189: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 4.24k op/s 2024-09-15T22:03:34.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T22:03:34.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:03:34.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:03:34.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:03:34.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:33 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 -- 172.21.15.28:0/2819310289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c106f10 msgr2=0x7f036c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 --2- 172.21.15.28:0/2819310289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c106f10 0x7f036c10ea90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0364009ca0 tx=0x7f036402a8c0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 -- 172.21.15.28:0/2819310289 shutdown_connections 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 --2- 172.21.15.28:0/2819310289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c106f10 0x7f036c10ea90 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 --2- 172.21.15.28:0/2819310289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c102790 0x7f036c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 --2- 172.21.15.28:0/2819310289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f036c101de0 0x7f036c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 -- 172.21.15.28:0/2819310289 >> 172.21.15.28:0/2819310289 conn(0x7f036c0fd8c0 msgr2=0x7f036c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 -- 172.21.15.28:0/2819310289 shutdown_connections 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.187+0000 7f0373a0b640 1 -- 172.21.15.28:0/2819310289 wait complete. 2024-09-15T22:03:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 Processor -- start 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 -- start start 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c101de0 0x7f036c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102790 0x7f036c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f036c106f10 0x7f036c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f036c111f70 con 0x7f036c102790 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f036c111df0 con 0x7f036c101de0 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0373a0b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f036c1120f0 con 0x7f036c106f10 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102790 0x7f036c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102790 0x7f036c198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41660/0 (socket says 172.21.15.28:41660) 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 -- 172.21.15.28:0/4291488349 learned_addr learned my addr 172.21.15.28:0/4291488349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0371fa8640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f036c106f10 0x7f036c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f03717a7640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c101de0 0x7f036c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 -- 172.21.15.28:0/4291488349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f036c106f10 msgr2=0x7f036c19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f036c106f10 0x7f036c19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 -- 172.21.15.28:0/4291488349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c101de0 msgr2=0x7f036c1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c101de0 0x7f036c1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f0370fa6640 1 -- 172.21.15.28:0/4291488349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036c19d390 con 0x7f036c102790 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.188+0000 7f03717a7640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c101de0 0x7f036c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.189+0000 7f0370fa6640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102790 0x7f036c198920 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f035c00e9f0 tx=0x7f035c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.189+0000 7f03627fc640 1 -- 172.21.15.28:0/4291488349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c00cd80 con 0x7f036c102790 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.189+0000 7f03627fc640 1 -- 172.21.15.28:0/4291488349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f035c010910 con 0x7f036c102790 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.189+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036c19d680 con 0x7f036c102790 2024-09-15T22:03:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.189+0000 7f03627fc640 1 -- 172.21.15.28:0/4291488349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c018990 con 0x7f036c102790 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.189+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f036c1a4f60 con 0x7f036c102790 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.191+0000 7f03627fc640 1 -- 172.21.15.28:0/4291488349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f035c010ac0 con 0x7f036c102790 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 Processor -- start 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 -- start start 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f3b60 0x5604b48f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48f4480 0x5604b48fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48fed40 0x5604b4901130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604b4859280 con 0x5604b48fed40 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5604b4800b10 con 0x5604b48f4480 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0375de4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604b4855240 con 0x5604b48f3b60 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48fed40 0x5604b4901130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48fed40 0x5604b4901130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41666/0 (socket says 172.21.15.28:41666) 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 -- 172.21.15.28:0/1152736246 learned_addr learned my addr 172.21.15.28:0/1152736246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0371fa8640 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f3b60 0x5604b48f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f03717a7640 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48f4480 0x5604b48fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 -- 172.21.15.28:0/1152736246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f3b60 msgr2=0x5604b48f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f3b60 0x5604b48f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 -- 172.21.15.28:0/1152736246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48f4480 msgr2=0x5604b48fe800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48f4480 0x5604b48fe800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f0370fa6640 1 -- 172.21.15.28:0/1152736246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604b480d1f0 con 0x5604b48fed40 2024-09-15T22:03:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f03627fc640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03580a3430 0x7f03580a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.192+0000 7f03717a7640 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48f4480 0x5604b48fe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.193+0000 7f0371fa8640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03580a3430 0x7f03580a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.193+0000 7f0370fa6640 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48fed40 0x5604b4901130 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f035c0cd710 tx=0x7f035c0d6a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e968c95b175786dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.193+0000 7f03627fc640 1 -- 172.21.15.28:0/4291488349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f035c0d03b0 con 0x7f036c102790 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.193+0000 7f03577fe640 1 -- 172.21.15.28:0/1152736246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c026030 con 0x5604b48fed40 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.193+0000 7f03577fe640 1 -- 172.21.15.28:0/1152736246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f035c09ca30 con 0x5604b48fed40 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.193+0000 7f03577fe640 1 -- 172.21.15.28:0/1152736246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c09bc90 con 0x5604b48fed40 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 -- 172.21.15.28:0/1152736246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48fed40 msgr2=0x5604b4901130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48fed40 0x5604b4901130 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f035c0cd710 tx=0x7f035c0d6a30 comp rx=0 tx=0).stop 2024-09-15T22:03:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0371fa8640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03580a3430 0x7f03580a58f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0364005e30 tx=0x7f0364005c30 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 -- 172.21.15.28:0/1152736246 shutdown_connections 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48fed40 0x5604b4901130 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48f4480 0x5604b48fe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 --2- 172.21.15.28:0/1152736246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f3b60 0x5604b48f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 -- 172.21.15.28:0/1152736246 >> 172.21.15.28:0/1152736246 conn(0x5604b48f00f0 msgr2=0x5604b48f04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 -- 172.21.15.28:0/1152736246 shutdown_connections 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 -- 172.21.15.28:0/1152736246 wait complete. 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 -- start start 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.194+0000 7f0375de4740 1 Processor -- start 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0375de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0375de4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b48fc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0375de4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b48fc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0375de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604b4901910 con 0x5604b48f3b60 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0375de4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5604b4813f10 con 0x5604b48fed40 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0375de4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5604b4901a90 con 0x5604b48f4480 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0375de4740 10 client.0 _ll_register_callbacks cb 0x5604b47dca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f03717a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0370fa6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b48fc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f03717a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fbc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41668/0 (socket says 172.21.15.28:41668) 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f03717a7640 1 -- 172.21.15.28:0/3838741651 learned_addr learned my addr 172.21.15.28:0/3838741651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0371fa8640 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b48fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0370fa6640 1 -- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 msgr2=0x5604b48fc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0370fa6640 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b48fc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0370fa6640 1 -- 172.21.15.28:0/3838741651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 msgr2=0x5604b48fbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0370fa6640 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0370fa6640 1 -- 172.21.15.28:0/3838741651 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604b45ca6a0 con 0x5604b48f4480 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f03717a7640 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0370fa6640 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b48fc150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f035c0cd800 tx=0x7f035c00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71d2c13a3fce8769 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0354ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.195+0000 7f0354ff9640 1 -- 172.21.15.28:0/3838741651 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c09bc90 con 0x5604b48f4480 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0354ff9640 1 -- 172.21.15.28:0/3838741651 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f035c0d0130 con 0x5604b48f4480 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f035c09ca30 con 0x5604b48f4480 2024-09-15T22:03:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0354ff9640 1 -- 172.21.15.28:0/3838741651 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f035c09c100 con 0x5604b48f4480 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 10 client.25936 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5604b4901910 con 0x5604b48f4480 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0354ff9640 1 -- 172.21.15.28:0/3838741651 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f035c09ca30 con 0x5604b48f4480 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0354ff9640 1 -- 172.21.15.28:0/3838741651 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f035c09c590 con 0x5604b48f4480 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 10 client.25936 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 10 client.25936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 20 client.25936 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b490e6c0 0x5604b4910ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 4 client.25936 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604b48667f0 con 0x5604b490e6c0 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.196+0000 7f03717a7640 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b490e6c0 0x5604b4910ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.197+0000 7f03717a7640 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b490e6c0 0x5604b4910ab0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5607f75971e1ab37 server_cookie=f37156bb5e28d08c in_seq=0 out_seq=0 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.197+0000 7f0354ff9640 10 client.25936 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.197+0000 7f0354ff9640 1 -- 172.21.15.28:0/3838741651 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604b48667f0 con 0x5604b490e6c0 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.197+0000 7f0354ff9640 10 client.25936 handle_command_reply: tid=0 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.197+0000 7f0375de4740 1 client.25936 shutdown 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.197+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 msgr2=0x5604b48fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.197+0000 7f0375de4740 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b48fc150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f035c0cd800 tx=0x7f035c00cf90 comp rx=0 tx=0).stop 2024-09-15T22:03:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 shutdown_connections 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b490e6c0 0x5604b4910ab0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b48fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b48fc150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0354ff9640 0 client.25936 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 --2- 172.21.15.28:0/3838741651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 >> 172.21.15.28:0/3838741651 conn(0x5604b48f00f0 msgr2=0x5604b48f5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 shutdown_connections 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 1 -- 172.21.15.28:0/3838741651 wait complete. 2024-09-15T22:03:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.198+0000 7f0375de4740 20 client.25936 trim_cache size 0 max 16384 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.219+0000 7f0375de4740 1 Processor -- start 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.219+0000 7f0375de4740 1 -- start start 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0375de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0375de4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4916050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0375de4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4916590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0375de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f036c1af000 con 0x5604b48f3b60 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41682/0 (socket says 172.21.15.28:41682) 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 -- 172.21.15.28:0/1633231814 learned_addr learned my addr 172.21.15.28:0/1633231814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0370fa6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4916050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0375de4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f036c1b0b40 con 0x5604b48fed40 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0375de4740 1 -- 172.21.15.28:0/1633231814 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f036c1af6d0 con 0x5604b48f4480 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f0371fa8640 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4916590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 -- 172.21.15.28:0/1633231814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 msgr2=0x5604b4916050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4916050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 -- 172.21.15.28:0/1633231814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 msgr2=0x5604b4916590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4916590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.220+0000 7f03717a7640 1 -- 172.21.15.28:0/1633231814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604b48eb830 con 0x5604b48f3b60 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.221+0000 7f03717a7640 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fda70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0368009eb0 tx=0x7f0368013980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f370f7f92db17cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.221+0000 7f0356ffd640 1 -- 172.21.15.28:0/1633231814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368023070 con 0x5604b48f3b60 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.221+0000 7f0356ffd640 1 -- 172.21.15.28:0/1633231814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0368005500 con 0x5604b48f3b60 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.222+0000 7f0375de4740 1 -- 172.21.15.28:0/1633231814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 msgr2=0x5604b48fda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.222+0000 7f0375de4740 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fda70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0368009eb0 tx=0x7f0368013980 comp rx=0 tx=0).stop 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 -- 172.21.15.28:0/1633231814 shutdown_connections 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4916590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4916050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 --2- 172.21.15.28:0/1633231814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b48fda70 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 -- 172.21.15.28:0/1633231814 >> 172.21.15.28:0/1633231814 conn(0x5604b48f00f0 msgr2=0x5604b48fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 -- 172.21.15.28:0/1633231814 shutdown_connections 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 -- 172.21.15.28:0/1633231814 wait complete. 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.224+0000 7f0375de4740 1 -- start start 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 1 Processor -- start 2024-09-15T22:03:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b4902e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4903340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4980830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f036c1b0470 con 0x5604b48f3b60 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f036c1afda0 con 0x5604b48fed40 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f036c1acfe0 con 0x5604b48f4480 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.225+0000 7f0375de4740 10 client.0 _ll_register_callbacks cb 0x7f035c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0371fa8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4980830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0371fa8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4980830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:58032/0 (socket says 172.21.15.28:58032) 2024-09-15T22:03:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0371fa8640 1 -- 172.21.15.28:0/1653202958 learned_addr learned my addr 172.21.15.28:0/1653202958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0370fa6640 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4903340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f03717a7640 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b4902e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0370fa6640 1 -- 172.21.15.28:0/1653202958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 msgr2=0x5604b4980830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0370fa6640 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4980830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0370fa6640 1 -- 172.21.15.28:0/1653202958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 msgr2=0x5604b4902e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:34.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0370fa6640 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b4902e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.226+0000 7f0370fa6640 1 -- 172.21.15.28:0/1653202958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604b4970fe0 con 0x5604b48f4480 2024-09-15T22:03:34.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0370fa6640 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4903340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f035c09c2c0 tx=0x7f035c0d6a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bc3ba194ed04d02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:34.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0354ff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:03:34.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0354ff9640 1 -- 172.21.15.28:0/1653202958 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c09bc90 con 0x5604b48f4480 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0368005500 con 0x5604b48f4480 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0354ff9640 1 -- 172.21.15.28:0/1653202958 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f035c0bc390 con 0x5604b48f4480 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0354ff9640 1 -- 172.21.15.28:0/1653202958 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f035c09be70 con 0x5604b48f4480 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0375de4740 10 client.25942 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5604b4972260 con 0x5604b48f4480 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.227+0000 7f0354ff9640 1 -- 172.21.15.28:0/1653202958 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0368005500 con 0x5604b48f4480 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0354ff9640 1 -- 172.21.15.28:0/1653202958 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f035c0b50d0 con 0x5604b48f4480 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0375de4740 10 client.25942 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0375de4740 10 client.25942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0375de4740 20 client.25942 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0375de4740 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b4972880 0x5604b4974ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0375de4740 4 client.25942 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T22:03:34.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5604b4860d00 con 0x5604b4972880 2024-09-15T22:03:34.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f03717a7640 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b4972880 0x5604b4974ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:34.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f03717a7640 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b4972880 0x5604b4974ca0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f3f7cada5288d1 server_cookie=5d458b6f0c75c301 in_seq=0 out_seq=0 2024-09-15T22:03:34.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.228+0000 7f0354ff9640 10 client.25942 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:34.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.229+0000 7f0354ff9640 1 -- 172.21.15.28:0/1653202958 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5604b4860d00 con 0x5604b4972880 2024-09-15T22:03:34.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.229+0000 7f0354ff9640 10 client.25942 handle_command_reply: tid=0 2024-09-15T22:03:34.198 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "a032fa29-3a4d-4c03-b896-5eaf89fc646f", 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.229+0000 7f0375de4740 1 client.25942 shutdown 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 msgr2=0x5604b4903340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4903340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f035c09c2c0 tx=0x7f035c0d6a30 comp rx=0 tx=0).stop 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 shutdown_connections 2024-09-15T22:03:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5604b4972880 0x5604b4974ca0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5604b48fed40 0x5604b4980830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5604b48f4480 0x5604b4903340 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 --2- 172.21.15.28:0/1653202958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604b48f3b60 0x5604b4902e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0354ff9640 0 client.25942 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 >> 172.21.15.28:0/1653202958 conn(0x5604b48f00f0 msgr2=0x5604b48fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 shutdown_connections 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 1 -- 172.21.15.28:0/1653202958 wait complete. 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.230+0000 7f0375de4740 20 client.25942 trim_cache size 0 max 16384 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03580a3430 msgr2=0x7f03580a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03580a3430 0x7f03580a58f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0364005e30 tx=0x7f0364005c30 comp rx=0 tx=0).stop 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102790 msgr2=0x7f036c198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102790 0x7f036c198920 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f035c00e9f0 tx=0x7f035c00eeb0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 shutdown_connections 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03580a3430 0x7f03580a58f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f036c106f10 0x7f036c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f036c102790 0x7f036c198920 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 --2- 172.21.15.28:0/4291488349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c101de0 0x7f036c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 >> 172.21.15.28:0/4291488349 conn(0x7f036c0fd8c0 msgr2=0x7f036c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 shutdown_connections 2024-09-15T22:03:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:33.231+0000 7f0373a0b640 1 -- 172.21.15.28:0/4291488349 wait complete. 2024-09-15T22:03:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: pgmap v189: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 4.24k op/s 2024-09-15T22:03:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T22:03:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:03:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:03:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:03:34.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:33 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: pgmap v189: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 74 KiB/s rd, 24 MiB/s wr, 4.24k op/s 2024-09-15T22:03:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T22:03:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:03:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:03:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:03:34.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:33 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:35.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:34 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:03:35.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:34 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:35.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:34 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:34 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:03:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:34 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:35.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:34 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:35.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:34 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:03:35.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:34 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:35.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:34 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:03:36.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:35 smithi028 ceph-mon[30969]: pgmap v190: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 56 KiB/s rd, 17 MiB/s wr, 3.03k op/s 2024-09-15T22:03:36.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:35 smithi203 ceph-mon[35546]: pgmap v190: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 56 KiB/s rd, 17 MiB/s wr, 3.03k op/s 2024-09-15T22:03:36.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:35 smithi174 ceph-mon[35244]: pgmap v190: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 56 KiB/s rd, 17 MiB/s wr, 3.03k op/s 2024-09-15T22:03:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:37 smithi028 ceph-mon[30969]: pgmap v191: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 2.97k op/s 2024-09-15T22:03:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:37 smithi203 ceph-mon[35546]: pgmap v191: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 2.97k op/s 2024-09-15T22:03:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:37 smithi174 ceph-mon[35244]: pgmap v191: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 2.97k op/s 2024-09-15T22:03:40.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:39 smithi028 ceph-mon[30969]: pgmap v192: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 123 KiB/s rd, 19 MiB/s wr, 3.30k op/s 2024-09-15T22:03:40.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:39 smithi203 ceph-mon[35546]: pgmap v192: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 123 KiB/s rd, 19 MiB/s wr, 3.30k op/s 2024-09-15T22:03:40.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:39 smithi174 ceph-mon[35244]: pgmap v192: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 123 KiB/s rd, 19 MiB/s wr, 3.30k op/s 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 -- 172.21.15.28:0/645498661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd914101250 msgr2=0x7fd914107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/645498661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd914101250 0x7fd914107990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd904009900 tx=0x7fd90402a410 comp rx=0 tx=0).stop 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 -- 172.21.15.28:0/645498661 shutdown_connections 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/645498661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9141080f0 0x7fd91410ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/645498661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd914101250 0x7fd914107990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/645498661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9141008a0 0x7fd914100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 -- 172.21.15.28:0/645498661 >> 172.21.15.28:0/645498661 conn(0x7fd914075ef0 msgr2=0x7fd914076300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 -- 172.21.15.28:0/645498661 shutdown_connections 2024-09-15T22:03:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 -- 172.21.15.28:0/645498661 wait complete. 2024-09-15T22:03:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.504+0000 7fd91b7e4640 1 Processor -- start 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd91b7e4640 1 -- start start 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd91b7e4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9141008a0 0x7fd914198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd91b7e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9141080f0 0x7fd914198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd91b7e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd91419cbf0 0x7fd91419d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd91b7e4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd914111f20 con 0x7fd91419cbf0 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd91b7e4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd914111da0 con 0x7fd9141080f0 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd91b7e4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141120a0 con 0x7fd9141008a0 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd91419cbf0 0x7fd91419d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd91419cbf0 0x7fd91419d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38456/0 (socket says 172.21.15.28:38456) 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 -- 172.21.15.28:0/355009781 learned_addr learned my addr 172.21.15.28:0/355009781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd918d7f640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9141080f0 0x7fd914198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919580640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9141008a0 0x7fd914198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 -- 172.21.15.28:0/355009781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9141008a0 msgr2=0x7fd914198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9141008a0 0x7fd914198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 -- 172.21.15.28:0/355009781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9141080f0 msgr2=0x7fd914198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9141080f0 0x7fd914198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 -- 172.21.15.28:0/355009781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91419d6e0 con 0x7fd91419cbf0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd918d7f640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9141080f0 0x7fd914198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.505+0000 7fd919d81640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd91419cbf0 0x7fd91419d0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00eaa0 tx=0x7fd90c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.506+0000 7fd90a7fc640 1 -- 172.21.15.28:0/355009781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c00cdb0 con 0x7fd91419cbf0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.506+0000 7fd90a7fc640 1 -- 172.21.15.28:0/355009781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90c005020 con 0x7fd91419cbf0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.506+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91419d9d0 con 0x7fd91419cbf0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.506+0000 7fd90a7fc640 1 -- 172.21.15.28:0/355009781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0116b0 con 0x7fd91419cbf0 2024-09-15T22:03:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.506+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9141a51e0 con 0x7fd91419cbf0 2024-09-15T22:03:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.508+0000 7fd90a7fc640 1 -- 172.21.15.28:0/355009781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd90c005560 con 0x7fd91419cbf0 2024-09-15T22:03:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 Processor -- start 2024-09-15T22:03:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 -- start start 2024-09-15T22:03:40.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe321ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5636fe2862f0 con 0x5636fe32ccb0 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5636fe22e270 con 0x5636fe321af0 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd91e656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5636fe2822b0 con 0x5636fe322410 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd918d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd90a7fc640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9000a34e0 0x7fd9000a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd918d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38468/0 (socket says 172.21.15.28:38468) 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.509+0000 7fd918d7f640 1 -- 172.21.15.28:0/3346159585 learned_addr learned my addr 172.21.15.28:0/3346159585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd919580640 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd919d81640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9000a34e0 0x7fd9000a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd918d7f640 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe321ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd918d7f640 1 -- 172.21.15.28:0/3346159585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 msgr2=0x5636fe32c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd90a7fc640 1 -- 172.21.15.28:0/355009781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd90c0d0340 con 0x7fd91419cbf0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd918d7f640 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd918d7f640 1 -- 172.21.15.28:0/3346159585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 msgr2=0x5636fe321ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd918d7f640 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe321ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd918d7f640 1 -- 172.21.15.28:0/3346159585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636fe22d340 con 0x5636fe32ccb0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd918d7f640 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32f0a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd9040096c0 tx=0x7fd904009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59064014ee9aa6fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd919d81640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9000a34e0 0x7fd9000a59a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00efd0 tx=0x7fd90c0bc040 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd8ff7fe640 1 -- 172.21.15.28:0/3346159585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904035040 con 0x5636fe32ccb0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd8ff7fe640 1 -- 172.21.15.28:0/3346159585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd904004030 con 0x5636fe32ccb0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.510+0000 7fd8ff7fe640 1 -- 172.21.15.28:0/3346159585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904003c70 con 0x5636fe32ccb0 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 -- 172.21.15.28:0/3346159585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 msgr2=0x5636fe32f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32f0a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd9040096c0 tx=0x7fd904009560 comp rx=0 tx=0).stop 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 -- 172.21.15.28:0/3346159585 shutdown_connections 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32f0a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 --2- 172.21.15.28:0/3346159585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe321ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 -- 172.21.15.28:0/3346159585 >> 172.21.15.28:0/3346159585 conn(0x5636fe31e060 msgr2=0x5636fe31e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 -- 172.21.15.28:0/3346159585 shutdown_connections 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 -- 172.21.15.28:0/3346159585 wait complete. 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.511+0000 7fd91e656740 1 -- start start 2024-09-15T22:03:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 1 Processor -- start 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe329ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5636fe32f880 con 0x5636fe32ccb0 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5636fe20f290 con 0x5636fe321af0 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5636fe32fa00 con 0x5636fe322410 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd91e656740 10 client.0 _ll_register_callbacks cb 0x5636fe240140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd918d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd918d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38478/0 (socket says 172.21.15.28:38478) 2024-09-15T22:03:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd918d7f640 1 -- 172.21.15.28:0/2208182558 learned_addr learned my addr 172.21.15.28:0/2208182558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd919580640 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.512+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe329ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd918d7f640 1 -- 172.21.15.28:0/2208182558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 msgr2=0x5636fe32a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd918d7f640 1 -- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 msgr2=0x5636fe329ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe329ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd918d7f640 1 -- 172.21.15.28:0/2208182558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636fdf99210 con 0x5636fe32ccb0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32a620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd904000c00 tx=0x7fd90402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9776e4ff1f8f159e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd8fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2208182558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904035030 con 0x5636fe32ccb0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2208182558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd904031940 con 0x5636fe32ccb0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd904004030 con 0x5636fe32ccb0 2024-09-15T22:03:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2208182558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd90405c050 con 0x5636fe32ccb0 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd91e656740 10 client.16416 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5636fe065620 con 0x5636fe32ccb0 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.513+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2208182558 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd904004030 con 0x5636fe32ccb0 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2208182558 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd90403a040 con 0x5636fe32ccb0 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd91e656740 10 client.16416 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd91e656740 10 client.16416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd91e656740 20 client.16416 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd91e656740 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe33caf0 0x5636fe33eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd91e656740 4 client.16416 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636fe293a20 con 0x5636fe33caf0 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd919580640 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe33caf0 0x5636fe33eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd8fcff9640 10 client.16416 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.514+0000 7fd919580640 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe33caf0 0x5636fe33eee0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=911b4a8ee90d1ab9 server_cookie=e3ff52b58f27158 in_seq=0 out_seq=0 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2208182558 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5636fe293a20 con 0x5636fe33caf0 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd8fcff9640 10 client.16416 handle_command_reply: tid=0 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 client.16416 shutdown 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 msgr2=0x5636fe32a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32a620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd904000c00 tx=0x7fd90402a410 comp rx=0 tx=0).stop 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 shutdown_connections 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe33caf0 0x5636fe33eee0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe32ccb0 0x5636fe32a620 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe322410 0x5636fe32a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd8fcff9640 0 client.16416 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 --2- 172.21.15.28:0/2208182558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe329ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 >> 172.21.15.28:0/2208182558 conn(0x5636fe31e060 msgr2=0x5636fe323d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.515+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 shutdown_connections 2024-09-15T22:03:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.516+0000 7fd91e656740 1 -- 172.21.15.28:0/2208182558 wait complete. 2024-09-15T22:03:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.516+0000 7fd91e656740 20 client.16416 trim_cache size 0 max 16384 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 Processor -- start 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 -- start start 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe321af0 0x5636fe32a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe322410 0x5636fe3350d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 0x5636fe335610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141aed50 con 0x5636fe321af0 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141b0890 con 0x5636fe322410 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.530+0000 7fd91e656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141af420 con 0x5636fe32ccb0 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe321af0 0x5636fe32a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe321af0 0x5636fe32a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38494/0 (socket says 172.21.15.28:38494) 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 -- 172.21.15.28:0/1389399211 learned_addr learned my addr 172.21.15.28:0/1389399211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd919580640 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe322410 0x5636fe3350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 -- 172.21.15.28:0/1389399211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 msgr2=0x5636fe335610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 0x5636fe335610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 -- 172.21.15.28:0/1389399211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe322410 msgr2=0x5636fe3350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe322410 0x5636fe3350d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 -- 172.21.15.28:0/1389399211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636fe322240 con 0x5636fe321af0 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd919580640 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe322410 0x5636fe3350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd918d7f640 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe321af0 0x5636fe32a450 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd9040095a0 tx=0x7fd904003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f13b75f728a37d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd8feffd640 1 -- 172.21.15.28:0/1389399211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904035070 con 0x5636fe321af0 2024-09-15T22:03:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd8feffd640 1 -- 172.21.15.28:0/1389399211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd904005520 con 0x5636fe321af0 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.531+0000 7fd8feffd640 1 -- 172.21.15.28:0/1389399211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90403d6b0 con 0x5636fe321af0 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 -- 172.21.15.28:0/1389399211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe321af0 msgr2=0x5636fe32a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe321af0 0x5636fe32a450 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd9040095a0 tx=0x7fd904003880 comp rx=0 tx=0).stop 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 -- 172.21.15.28:0/1389399211 shutdown_connections 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 0x5636fe335610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe322410 0x5636fe3350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 --2- 172.21.15.28:0/1389399211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe321af0 0x5636fe32a450 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 -- 172.21.15.28:0/1389399211 >> 172.21.15.28:0/1389399211 conn(0x5636fe31e060 msgr2=0x5636fe32bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 -- 172.21.15.28:0/1389399211 shutdown_connections 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 -- 172.21.15.28:0/1389399211 wait complete. 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 -- start start 2024-09-15T22:03:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.532+0000 7fd91e656740 1 Processor -- start 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd91e656740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe3444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd91e656740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe322410 0x5636fe3449e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd91e656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 0x5636fe3a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd91e656740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141b01c0 con 0x5636fe322410 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd91e656740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141afaf0 con 0x5636fe321af0 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd91e656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9141acd30 con 0x5636fe32ccb0 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd91e656740 10 client.0 _ll_register_callbacks cb 0x7fd91000b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe322410 0x5636fe3449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe322410 0x5636fe3449e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38508/0 (socket says 172.21.15.28:38508) 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 -- 172.21.15.28:0/2091583167 learned_addr learned my addr 172.21.15.28:0/2091583167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 0x5636fe3a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe3444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 -- 172.21.15.28:0/2091583167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 msgr2=0x5636fe3a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 0x5636fe3a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 -- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 msgr2=0x5636fe3444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe3444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 -- 172.21.15.28:0/2091583167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636fe3a3210 con 0x5636fe322410 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd919580640 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe322410 0x5636fe3449e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd91000bdd0 tx=0x7fd910004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c007c10afa0981ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.533+0000 7fd8fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:03:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2091583167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91001d580 con 0x5636fe322410 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2091583167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd91001d720 con 0x5636fe322410 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd904005520 con 0x5636fe322410 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2091583167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd91002d050 con 0x5636fe322410 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 10 client.16422 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5636fe3ab800 con 0x5636fe322410 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2091583167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd904005520 con 0x5636fe322410 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2091583167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd910011c70 con 0x5636fe322410 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 10 client.16422 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 10 client.16422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 20 client.16422 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe3abf30 0x5636fe3ae350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 4 client.16422 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:03:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.534+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5636fe28dd70 con 0x5636fe3abf30 2024-09-15T22:03:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.535+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe3abf30 0x5636fe3ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.535+0000 7fd918d7f640 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe3abf30 0x5636fe3ae350 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=695eb6246c970861 server_cookie=9a28ccd6f09fa4d2 in_seq=0 out_seq=0 2024-09-15T22:03:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.535+0000 7fd8fcff9640 10 client.16422 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd8fcff9640 1 -- 172.21.15.28:0/2091583167 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5636fe28dd70 con 0x5636fe3abf30 2024-09-15T22:03:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd8fcff9640 10 client.16422 handle_command_reply: tid=0 2024-09-15T22:03:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 client.16422 shutdown 2024-09-15T22:03:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe322410 msgr2=0x5636fe3449e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe322410 0x5636fe3449e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd91000bdd0 tx=0x7fd910004540 comp rx=0 tx=0).stop 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 shutdown_connections 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5636fe3abf30 0x5636fe3ae350 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5636fe32ccb0 0x5636fe3a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5636fe322410 0x5636fe3449e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd8fcff9640 0 client.16422 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 --2- 172.21.15.28:0/2091583167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5636fe321af0 0x5636fe3444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 >> 172.21.15.28:0/2091583167 conn(0x5636fe31e060 msgr2=0x5636fe323550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 shutdown_connections 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.553+0000 7fd91e656740 1 -- 172.21.15.28:0/2091583167 wait complete. 2024-09-15T22:03:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91e656740 20 client.16422 trim_cache size 0 max 16384 2024-09-15T22:03:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:03:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:03:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:03:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9000a34e0 msgr2=0x7fd9000a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9000a34e0 0x7fd9000a59a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00efd0 tx=0x7fd90c0bc040 comp rx=0 tx=0).stop 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd91419cbf0 msgr2=0x7fd91419d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd91419cbf0 0x7fd91419d0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00eaa0 tx=0x7fd90c00ef60 comp rx=0 tx=0).stop 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 shutdown_connections 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd9000a34e0 0x7fd9000a59a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd91419cbf0 0x7fd91419d0a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9141080f0 0x7fd914198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 --2- 172.21.15.28:0/355009781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9141008a0 0x7fd914198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 >> 172.21.15.28:0/355009781 conn(0x7fd914075ef0 msgr2=0x7fd9140ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 shutdown_connections 2024-09-15T22:03:40.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:40.554+0000 7fd91b7e4640 1 -- 172.21.15.28:0/355009781 wait complete. 2024-09-15T22:03:42.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:41 smithi028 ceph-mon[30969]: pgmap v193: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 91 KiB/s rd, 10 MiB/s wr, 1.88k op/s 2024-09-15T22:03:42.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:41 smithi203 ceph-mon[35546]: pgmap v193: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 91 KiB/s rd, 10 MiB/s wr, 1.88k op/s 2024-09-15T22:03:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:41 smithi174 ceph-mon[35244]: pgmap v193: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 91 KiB/s rd, 10 MiB/s wr, 1.88k op/s 2024-09-15T22:03:44.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:43 smithi028 ceph-mon[30969]: pgmap v194: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 105 KiB/s rd, 9.1 MiB/s wr, 1.66k op/s 2024-09-15T22:03:44.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:43 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:03:44.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:43 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:03:44.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:43 smithi203 ceph-mon[35546]: pgmap v194: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 105 KiB/s rd, 9.1 MiB/s wr, 1.66k op/s 2024-09-15T22:03:44.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:43 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:03:44.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:43 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:03:44.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:43 smithi174 ceph-mon[35244]: pgmap v194: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 105 KiB/s rd, 9.1 MiB/s wr, 1.66k op/s 2024-09-15T22:03:44.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:43 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:03:44.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:43 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:03:44.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:03:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:03:45.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:44 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:03:45.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:44 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:03:45.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:44 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:03:46.161 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:45 smithi174 ceph-mon[35244]: pgmap v195: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 88 KiB/s rd, 1.9 MiB/s wr, 398 op/s 2024-09-15T22:03:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:45 smithi028 ceph-mon[30969]: pgmap v195: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 88 KiB/s rd, 1.9 MiB/s wr, 398 op/s 2024-09-15T22:03:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:45 smithi203 ceph-mon[35546]: pgmap v195: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 88 KiB/s rd, 1.9 MiB/s wr, 398 op/s 2024-09-15T22:03:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:47 smithi028 ceph-mon[30969]: pgmap v196: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 82 KiB/s rd, 1.9 MiB/s wr, 384 op/s 2024-09-15T22:03:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:47 smithi203 ceph-mon[35546]: pgmap v196: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 82 KiB/s rd, 1.9 MiB/s wr, 384 op/s 2024-09-15T22:03:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:47 smithi174 ceph-mon[35244]: pgmap v196: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 82 KiB/s rd, 1.9 MiB/s wr, 384 op/s 2024-09-15T22:03:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:49 smithi028 ceph-mon[30969]: pgmap v197: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 377 op/s 2024-09-15T22:03:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:49 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1642307420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:03:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:49 smithi203 ceph-mon[35546]: pgmap v197: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 377 op/s 2024-09-15T22:03:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:49 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1642307420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:03:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:49 smithi174 ceph-mon[35244]: pgmap v197: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 76 KiB/s rd, 1.9 MiB/s wr, 377 op/s 2024-09-15T22:03:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:49 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1642307420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.338+0000 7fac2e684640 1 -- 172.21.15.28:0/1492612602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28106a80 msgr2=0x7fac28106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.338+0000 7fac2e684640 1 --2- 172.21.15.28:0/1492612602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28106a80 0x7fac28106e60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fac18009930 tx=0x7fac1802a420 comp rx=0 tx=0).stop 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.338+0000 7fac2e684640 1 -- 172.21.15.28:0/1492612602 shutdown_connections 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.338+0000 7fac2e684640 1 --2- 172.21.15.28:0/1492612602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac28101420 0x7fac2810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.338+0000 7fac2e684640 1 --2- 172.21.15.28:0/1492612602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac28100a80 0x7fac28100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.338+0000 7fac2e684640 1 --2- 172.21.15.28:0/1492612602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28106a80 0x7fac28106e60 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.338+0000 7fac2e684640 1 -- 172.21.15.28:0/1492612602 >> 172.21.15.28:0/1492612602 conn(0x7fac280fc7a0 msgr2=0x7fac280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.339+0000 7fac2e684640 1 -- 172.21.15.28:0/1492612602 shutdown_connections 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.339+0000 7fac2e684640 1 -- 172.21.15.28:0/1492612602 wait complete. 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.339+0000 7fac2e684640 1 Processor -- start 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.339+0000 7fac2e684640 1 -- start start 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac2e684640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac28100a80 0x7fac2819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac2e684640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101420 0x7fac2819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac2e684640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac28106a80 0x7fac2819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac2e684640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac28110e80 con 0x7fac28101420 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac2e684640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fac28110d00 con 0x7fac28100a80 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac2e684640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac28111000 con 0x7fac28106a80 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101420 0x7fac2819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101420 0x7fac2819aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38518/0 (socket says 172.21.15.28:38518) 2024-09-15T22:03:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 -- 172.21.15.28:0/1642307420 learned_addr learned my addr 172.21.15.28:0/1642307420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac27fff640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac28100a80 0x7fac2819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:50.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:03:50.496 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"btime":"2024-09-15T22:01:08:357214+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:01:08.288094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:03:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac2cc21640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac28106a80 0x7fac2819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 -- 172.21.15.28:0/1642307420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac28106a80 msgr2=0x7fac2819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac28106a80 0x7fac2819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 -- 172.21.15.28:0/1642307420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac28100a80 msgr2=0x7fac2819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac28100a80 0x7fac2819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 -- 172.21.15.28:0/1642307420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2819f610 con 0x7fac28101420 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.340+0000 7fac277fe640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101420 0x7fac2819aba0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00e9f0 tx=0x7fac1c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.341+0000 7fac257fa640 1 -- 172.21.15.28:0/1642307420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c00cd80 con 0x7fac28101420 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.341+0000 7fac257fa640 1 -- 172.21.15.28:0/1642307420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac1c005500 con 0x7fac28101420 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.341+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2819f900 con 0x7fac28101420 2024-09-15T22:03:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.341+0000 7fac257fa640 1 -- 172.21.15.28:0/1642307420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c010690 con 0x7fac28101420 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.341+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac281a71e0 con 0x7fac28101420 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.342+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac28100ee0 con 0x7fac28101420 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.343+0000 7fac257fa640 1 -- 172.21.15.28:0/1642307420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fac1c0046c0 con 0x7fac28101420 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.344+0000 7fac257fa640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac100a33b0 0x7fac100a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.344+0000 7fac257fa640 1 -- 172.21.15.28:0/1642307420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fac1c0d0740 con 0x7fac28101420 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.344+0000 7fac27fff640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac100a33b0 0x7fac100a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.344+0000 7fac27fff640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac100a33b0 0x7fac100a5870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fac18009900 tx=0x7fac180058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.347+0000 7fac257fa640 1 -- 172.21.15.28:0/1642307420 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fac1c0541b0 con 0x7fac28101420 2024-09-15T22:03:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.560+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac2819b9c0 con 0x7fac28101420 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.561+0000 7fac257fa640 1 -- 172.21.15.28:0/1642307420 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11986 (secure 0 0 0) 0x7fac2819b9c0 con 0x7fac28101420 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac100a33b0 msgr2=0x7fac100a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac100a33b0 0x7fac100a5870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fac18009900 tx=0x7fac180058d0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101420 msgr2=0x7fac2819aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101420 0x7fac2819aba0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00e9f0 tx=0x7fac1c00edf0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 shutdown_connections 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac100a33b0 0x7fac100a5870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac28106a80 0x7fac2819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac28101420 0x7fac2819aba0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 --2- 172.21.15.28:0/1642307420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac28100a80 0x7fac2819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 >> 172.21.15.28:0/1642307420 conn(0x7fac280fc7a0 msgr2=0x7fac2806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 shutdown_connections 2024-09-15T22:03:50.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:03:49.563+0000 7fac2e684640 1 -- 172.21.15.28:0/1642307420 wait complete. 2024-09-15T22:03:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:51 smithi028 ceph-mon[30969]: pgmap v198: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-09-15T22:03:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:51 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:03:52.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:51 smithi203 ceph-mon[35546]: pgmap v198: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-09-15T22:03:52.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:51 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:03:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:51 smithi174 ceph-mon[35244]: pgmap v198: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-09-15T22:03:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:51 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.655+0000 7f9eed1cd640 1 -- 172.21.15.28:0/1853334873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee810c110 msgr2=0x7f9ee810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.655+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/1853334873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee810c110 0x7f9ee810e590 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009ca0 tx=0x7f9ee002a950 comp rx=0 tx=0).stop 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 -- 172.21.15.28:0/1853334873 shutdown_connections 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/1853334873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee810c110 0x7f9ee810e590 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/1853334873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee81035e0 0x7f9ee810bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/1853334873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee8102cc0 0x7f9ee81030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 -- 172.21.15.28:0/1853334873 >> 172.21.15.28:0/1853334873 conn(0x7f9ee80fc6b0 msgr2=0x7f9ee80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 -- 172.21.15.28:0/1853334873 shutdown_connections 2024-09-15T22:03:53.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 -- 172.21.15.28:0/1853334873 wait complete. 2024-09-15T22:03:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.656+0000 7f9eed1cd640 1 Processor -- start 2024-09-15T22:03:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9eed1cd640 1 -- start start 2024-09-15T22:03:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9eed1cd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee8102cc0 0x7f9ee81982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9eed1cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee81035e0 0x7f9ee8198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9eed1cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee810c110 0x7f9ee819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9eed1cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8111f40 con 0x7f9ee81035e0 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9eed1cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8111dc0 con 0x7f9ee810c110 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9eed1cd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee81120c0 con 0x7f9ee8102cc0 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee81035e0 0x7f9ee8198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee81035e0 0x7f9ee8198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56508/0 (socket says 172.21.15.28:56508) 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 -- 172.21.15.28:0/2823563157 learned_addr learned my addr 172.21.15.28:0/2823563157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee8102cc0 0x7f9ee81982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee759e640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee810c110 0x7f9ee819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 -- 172.21.15.28:0/2823563157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee8102cc0 msgr2=0x7f9ee81982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee8102cc0 0x7f9ee81982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 -- 172.21.15.28:0/2823563157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee810c110 msgr2=0x7f9ee819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee810c110 0x7f9ee819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee659c640 1 -- 172.21.15.28:0/2823563157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee819d2e0 con 0x7f9ee81035e0 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.657+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee8102cc0 0x7f9ee81982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.658+0000 7f9ee659c640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee81035e0 0x7f9ee8198830 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00e9f0 tx=0x7f9edc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.658+0000 7f9ecffff640 1 -- 172.21.15.28:0/2823563157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc00ccc0 con 0x7f9ee81035e0 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.658+0000 7f9ecffff640 1 -- 172.21.15.28:0/2823563157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9edc00ce60 con 0x7f9ee81035e0 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.658+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee819d5d0 con 0x7f9ee81035e0 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.658+0000 7f9ecffff640 1 -- 172.21.15.28:0/2823563157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc010690 con 0x7f9ee81035e0 2024-09-15T22:03:53.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.658+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee81a4e60 con 0x7f9ee81035e0 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.660+0000 7f9ecffff640 1 -- 172.21.15.28:0/2823563157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f9edc0046c0 con 0x7f9ee81035e0 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.660+0000 7f9ef0042740 1 Processor -- start 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ef0042740 1 -- start start 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d45bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d466770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d466cb0 0x55dd1d4690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ef0042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd1d3c02f0 con 0x55dd1d45baf0 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ef0042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd1d368270 con 0x55dd1d45c410 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ef0042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dd1d3bc2b0 con 0x55dd1d466cb0 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d45bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d45bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56516/0 (socket says 172.21.15.28:56516) 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 -- 172.21.15.28:0/1804961252 learned_addr learned my addr 172.21.15.28:0/1804961252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee659c640 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d466cb0 0x55dd1d4690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d466770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ecffff640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ed00a3460 0x7f9ed00a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 -- 172.21.15.28:0/1804961252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d466cb0 msgr2=0x55dd1d4690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d466cb0 0x55dd1d4690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 -- 172.21.15.28:0/1804961252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 msgr2=0x55dd1d466770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d466770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee759e640 1 -- 172.21.15.28:0/1804961252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd1d367340 con 0x55dd1d45baf0 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.661+0000 7f9ee659c640 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d466cb0 0x55dd1d4690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ecffff640 1 -- 172.21.15.28:0/2823563157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9edc0d0530 con 0x7f9ee81035e0 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ee759e640 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d45bed0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0035000 tx=0x7f9ee0009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295555af9d8440ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ee759e640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ed00a3460 0x7f9ed00a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ecd7fa640 1 -- 172.21.15.28:0/1804961252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0049040 con 0x55dd1d45baf0 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ecd7fa640 1 -- 172.21.15.28:0/1804961252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee0003e50 con 0x55dd1d45baf0 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ecd7fa640 1 -- 172.21.15.28:0/1804961252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0034040 con 0x55dd1d45baf0 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ef0042740 1 -- 172.21.15.28:0/1804961252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 msgr2=0x55dd1d45bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ee759e640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ed00a3460 0x7f9ed00a5920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00049d0 tx=0x7f9ee00046e0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.662+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d45bed0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0035000 tx=0x7f9ee0009a20 comp rx=0 tx=0).stop 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- 172.21.15.28:0/1804961252 shutdown_connections 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d466cb0 0x55dd1d4690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d466770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1804961252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d45bed0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- 172.21.15.28:0/1804961252 >> 172.21.15.28:0/1804961252 conn(0x55dd1d458060 msgr2=0x55dd1d458440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- 172.21.15.28:0/1804961252 shutdown_connections 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- 172.21.15.28:0/1804961252 wait complete. 2024-09-15T22:03:53.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- start start 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 Processor -- start 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d463b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d4640a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d466cb0 0x55dd1d4645e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd1d469880 con 0x55dd1d466cb0 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd1d349290 con 0x55dd1d45c410 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dd1d469a00 con 0x55dd1d45baf0 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ef0042740 10 client.0 _ll_register_callbacks cb 0x55dd1d37a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.663+0000 7f9ee759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d466cb0 0x55dd1d4645e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d466cb0 0x55dd1d4645e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56518/0 (socket says 172.21.15.28:56518) 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 -- 172.21.15.28:0/3460466407 learned_addr learned my addr 172.21.15.28:0/3460466407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d463b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee659c640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d4640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 -- 172.21.15.28:0/3460466407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 msgr2=0x55dd1d463b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d463b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 -- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 msgr2=0x55dd1d4640a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d4640a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 -- 172.21.15.28:0/3460466407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1d0d3210 con 0x55dd1d466cb0 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee659c640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d4640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ee759e640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d466cb0 0x55dd1d4645e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9ee002ae70 tx=0x7f9ee0042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed32043d28c7db4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ecaff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3460466407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee005e060 con 0x55dd1d466cb0 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee0003e50 con 0x55dd1d466cb0 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3460466407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9ee002f940 con 0x55dd1d466cb0 2024-09-15T22:03:53.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3460466407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9ee0034040 con 0x55dd1d466cb0 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.664+0000 7f9ef0042740 10 client.16440 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dd1d19f620 con 0x55dd1d466cb0 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3460466407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee0003e50 con 0x55dd1d466cb0 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3460466407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9ee0042070 con 0x55dd1d466cb0 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ef0042740 10 client.16440 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ef0042740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ef0042740 20 client.16440 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d476a50 0x55dd1d478e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ef0042740 4 client.16440 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd1d3cda20 con 0x55dd1d476a50 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.665+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d476a50 0x55dd1d478e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.666+0000 7f9ecaff5640 10 client.16440 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:53.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.666+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d476a50 0x55dd1d478e40 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe5e2f0fb98b4e0 server_cookie=aae5c356f962b8a3 in_seq=0 out_seq=0 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.666+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3460466407 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dd1d3cda20 con 0x55dd1d476a50 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.666+0000 7f9ecaff5640 10 client.16440 handle_command_reply: tid=0 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.666+0000 7f9ef0042740 1 client.16440 shutdown 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d466cb0 msgr2=0x55dd1d4645e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d466cb0 0x55dd1d4645e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9ee002ae70 tx=0x7f9ee0042000 comp rx=0 tx=0).stop 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 shutdown_connections 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d476a50 0x55dd1d478e40 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d466cb0 0x55dd1d4645e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d45c410 0x55dd1d4640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ecaff5640 0 client.16440 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3460466407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d463b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 >> 172.21.15.28:0/3460466407 conn(0x55dd1d458060 msgr2=0x55dd1d45ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 shutdown_connections 2024-09-15T22:03:53.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 1 -- 172.21.15.28:0/3460466407 wait complete. 2024-09-15T22:03:53.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.667+0000 7f9ef0042740 20 client.16440 trim_cache size 0 max 16384 2024-09-15T22:03:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 Processor -- start 2024-09-15T22:03:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 -- start start 2024-09-15T22:03:53.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d464410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45c410 0x55dd1d47e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d47e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee81aee80 con 0x55dd1d45c410 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee81b09c0 con 0x55dd1d466cb0 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ef0042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee81af550 con 0x55dd1d45baf0 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45c410 0x55dd1d47e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45c410 0x55dd1d47e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56534/0 (socket says 172.21.15.28:56534) 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 -- 172.21.15.28:0/1874091917 learned_addr learned my addr 172.21.15.28:0/1874091917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d464410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee759e640 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d47e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 -- 172.21.15.28:0/1874091917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 msgr2=0x55dd1d464410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d464410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 -- 172.21.15.28:0/1874091917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 msgr2=0x55dd1d47e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d47e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 -- 172.21.15.28:0/1874091917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd1d458cf0 con 0x55dd1d45c410 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.682+0000 7f9ee659c640 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45c410 0x55dd1d47e3e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9edc098e20 tx=0x7f9edc004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7792ca1eb19322 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9eccff9640 1 -- 172.21.15.28:0/1874091917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc09bf90 con 0x55dd1d45c410 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9eccff9640 1 -- 172.21.15.28:0/1874091917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9edc09cb50 con 0x55dd1d45c410 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9eccff9640 1 -- 172.21.15.28:0/1874091917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc09bdb0 con 0x55dd1d45c410 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 -- 172.21.15.28:0/1874091917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45c410 msgr2=0x55dd1d47e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45c410 0x55dd1d47e3e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9edc098e20 tx=0x7f9edc004b00 comp rx=0 tx=0).stop 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 -- 172.21.15.28:0/1874091917 shutdown_connections 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d47e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45c410 0x55dd1d47e3e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 --2- 172.21.15.28:0/1874091917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45baf0 0x55dd1d464410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 -- 172.21.15.28:0/1874091917 >> 172.21.15.28:0/1874091917 conn(0x55dd1d458060 msgr2=0x55dd1d465be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 -- 172.21.15.28:0/1874091917 shutdown_connections 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.683+0000 7f9ef0042740 1 -- 172.21.15.28:0/1874091917 wait complete. 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 -- start start 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 Processor -- start 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d46f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45c410 0x55dd1d460960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d460f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee81b02f0 con 0x55dd1d45baf0 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee81afc20 con 0x55dd1d466cb0 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee81ace60 con 0x55dd1d45c410 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ef0042740 10 client.0 _ll_register_callbacks cb 0x7f9ed800b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d46f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d46f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56538/0 (socket says 172.21.15.28:56538) 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 -- 172.21.15.28:0/3322577416 learned_addr learned my addr 172.21.15.28:0/3322577416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee759e640 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d460f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee659c640 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45c410 0x55dd1d460960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 -- 172.21.15.28:0/3322577416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45c410 msgr2=0x55dd1d460960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45c410 0x55dd1d460960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 -- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 msgr2=0x55dd1d460f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d460f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.684+0000 7f9ee6d9d640 1 -- 172.21.15.28:0/3322577416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd1d461750 con 0x55dd1d45baf0 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ee6d9d640 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d46f4d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x55dd1d3c0400 tx=0x7f9ed8013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc76380378eed7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ecaff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3322577416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8010b00 con 0x55dd1d45baf0 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3322577416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9ed8010ca0 con 0x55dd1d45baf0 2024-09-15T22:03:53.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9edc09cb50 con 0x55dd1d45baf0 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3322577416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9ed802f050 con 0x55dd1d45baf0 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 10 client.16446 fetch_fsmap learned FSMap version 32 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dd1d4e1500 con 0x55dd1d45baf0 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3322577416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9edc09cb50 con 0x55dd1d45baf0 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3322577416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9ed8026450 con 0x55dd1d45baf0 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 10 client.16446 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 20 client.16446 populate_metadata read hostname 'smithi028' 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d4e1c00 0x55dd1d4e4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 4 client.16446 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.685+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd1d3c7d70 con 0x55dd1d4e1c00 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.686+0000 7f9ee659c640 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d4e1c00 0x55dd1d4e4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:03:53.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.686+0000 7f9ecaff5640 10 client.16446 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:53.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.686+0000 7f9ee659c640 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d4e1c00 0x55dd1d4e4020 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3783526b3f2391d1 server_cookie=33249eba85d2581 in_seq=0 out_seq=0 2024-09-15T22:03:53.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ecaff5640 1 -- 172.21.15.28:0/3322577416 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd1d3c7d70 con 0x55dd1d4e1c00 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ecaff5640 10 client.16446 handle_command_reply: tid=0 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 client.16446 shutdown 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 msgr2=0x55dd1d46f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d46f4d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x55dd1d3c0400 tx=0x7f9ed8013bd0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 shutdown_connections 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd1d4e1c00 0x55dd1d4e4020 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd1d466cb0 0x55dd1d460f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ecaff5640 0 client.16446 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd1d45c410 0x55dd1d460960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 --2- 172.21.15.28:0/3322577416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd1d45baf0 0x55dd1d46f4d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.699+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 >> 172.21.15.28:0/3322577416 conn(0x55dd1d458060 msgr2=0x55dd1d4569a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:53.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 shutdown_connections 2024-09-15T22:03:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9ef0042740 1 -- 172.21.15.28:0/3322577416 wait complete. 2024-09-15T22:03:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9ef0042740 20 client.16446 trim_cache size 0 max 16384 2024-09-15T22:03:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:03:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:03:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:03:53.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ed00a3460 msgr2=0x7f9ed00a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ed00a3460 0x7f9ed00a5920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00049d0 tx=0x7f9ee00046e0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee81035e0 msgr2=0x7f9ee8198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee81035e0 0x7f9ee8198830 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00e9f0 tx=0x7f9edc00eeb0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 shutdown_connections 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ed00a3460 0x7f9ed00a5920 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee810c110 0x7f9ee819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee81035e0 0x7f9ee8198830 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 --2- 172.21.15.28:0/2823563157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee8102cc0 0x7f9ee81982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:03:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 >> 172.21.15.28:0/2823563157 conn(0x7f9ee80fc6b0 msgr2=0x7f9ee80fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:03:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 shutdown_connections 2024-09-15T22:03:53.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:03:53.700+0000 7f9eed1cd640 1 -- 172.21.15.28:0/2823563157 wait complete. 2024-09-15T22:03:54.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:53 smithi028 ceph-mon[30969]: pgmap v199: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-09-15T22:03:54.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:53 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:03:54.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:53 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:03:54.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:53 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:03:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:53 smithi203 ceph-mon[35546]: pgmap v199: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-09-15T22:03:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:53 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:03:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:53 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:03:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:53 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:03:54.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:53 smithi174 ceph-mon[35244]: pgmap v199: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-09-15T22:03:54.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:53 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:03:54.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:53 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:03:54.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:53 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:03:56.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:55 smithi028 ceph-mon[30969]: pgmap v200: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T22:03:56.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:55 smithi203 ceph-mon[35546]: pgmap v200: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T22:03:56.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:55 smithi174 ceph-mon[35244]: pgmap v200: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T22:03:57.714 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 4 (f) to a remote file /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.f.json 2024-09-15T22:03:57.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 ops --flags=locks --path=/var/run/ceph/ops-ebf3369a-1-mds.f.json 2024-09-15T22:03:57.719 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:03:58.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:57 smithi028 ceph-mon[30969]: pgmap v201: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T22:03:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:57 smithi203 ceph-mon[35546]: pgmap v201: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T22:03:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:57 smithi174 ceph-mon[35244]: pgmap v201: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T22:04:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:03:59 smithi028 ceph-mon[30969]: pgmap v202: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T22:04:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:03:59 smithi203 ceph-mon[35546]: pgmap v202: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T22:04:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:03:59 smithi174 ceph-mon[35244]: pgmap v202: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 -- 172.21.15.28:0/3739845972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc103660 msgr2=0x7fd7dc10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 --2- 172.21.15.28:0/3739845972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc103660 0x7fd7dc10bb50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009900 tx=0x7fd7d002a480 comp rx=0 tx=0).stop 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 -- 172.21.15.28:0/3739845972 shutdown_connections 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 --2- 172.21.15.28:0/3739845972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7dc10c170 0x7fd7dc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 --2- 172.21.15.28:0/3739845972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc103660 0x7fd7dc10bb50 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 --2- 172.21.15.28:0/3739845972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7dc102d40 0x7fd7dc103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 -- 172.21.15.28:0/3739845972 >> 172.21.15.28:0/3739845972 conn(0x7fd7dc0fc7a0 msgr2=0x7fd7dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 -- 172.21.15.28:0/3739845972 shutdown_connections 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.826+0000 7fd7e2497640 1 -- 172.21.15.28:0/3739845972 wait complete. 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 Processor -- start 2024-09-15T22:04:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 -- start start 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7dc102d40 0x7fd7dc1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7dc103660 0x7fd7dc100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc10c170 0x7fd7dc101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc111f70 con 0x7fd7dc10c170 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc111df0 con 0x7fd7dc103660 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e2497640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1120f0 con 0x7fd7dc102d40 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.827+0000 7fd7e0a34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc10c170 0x7fd7dc101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc10c170 0x7fd7dc101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45350/0 (socket says 172.21.15.28:45350) 2024-09-15T22:04:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 -- 172.21.15.28:0/1214602473 learned_addr learned my addr 172.21.15.28:0/1214602473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7dc102d40 0x7fd7dc1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7dc103660 0x7fd7dc100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 -- 172.21.15.28:0/1214602473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7dc102d40 msgr2=0x7fd7dc1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7dc102d40 0x7fd7dc1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 -- 172.21.15.28:0/1214602473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7dc103660 msgr2=0x7fd7dc100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7dc103660 0x7fd7dc100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 -- 172.21.15.28:0/1214602473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc1a6910 con 0x7fd7dc10c170 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7dc103660 0x7fd7dc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7dc102d40 0x7fd7dc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e0a34640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc10c170 0x7fd7dc101420 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400eaa0 tx=0x7fd7d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7d97fa640 1 -- 172.21.15.28:0/1214602473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d400cdb0 con 0x7fd7dc10c170 2024-09-15T22:04:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7d97fa640 1 -- 172.21.15.28:0/1214602473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d4005020 con 0x7fd7dc10c170 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7dc1a6c00 con 0x7fd7dc10c170 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7d97fa640 1 -- 172.21.15.28:0/1214602473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40116b0 con 0x7fd7dc10c170 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.828+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7dc1a70f0 con 0x7fd7dc10c170 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.831+0000 7fd7d97fa640 1 -- 172.21.15.28:0/1214602473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd7d4005560 con 0x7fd7dc10c170 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.831+0000 7fd7e4870740 1 Processor -- start 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.831+0000 7fd7e4870740 1 -- start start 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985f4a0 0x5558c985f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c986a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c986ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7e4870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558c97c52e0 con 0x5558c985f4a0 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7e4870740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5558c976cce0 con 0x5558c986a680 2024-09-15T22:04:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7e4870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5558c97c12a0 con 0x5558c985fdc0 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985f4a0 0x5558c985f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985f4a0 0x5558c985f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45366/0 (socket says 172.21.15.28:45366) 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 -- 172.21.15.28:0/604470292 learned_addr learned my addr 172.21.15.28:0/604470292 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7d97fa640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd7c40a33b0 0x7fd7c40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c986a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c986ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7e0a34640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd7c40a33b0 0x7fd7c40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 -- 172.21.15.28:0/604470292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 msgr2=0x5558c986a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c986a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 -- 172.21.15.28:0/604470292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 msgr2=0x5558c986ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c986ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 -- 172.21.15.28:0/604470292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c9751440 con 0x5558c985f4a0 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7d97fa640 1 -- 172.21.15.28:0/1214602473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd7d40d0340 con 0x7fd7dc10c170 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.832+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985f4a0 0x5558c985f880 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0005940 tx=0x7fd7d0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8f51f0ec5fd9ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7c27fc640 1 -- 172.21.15.28:0/604470292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0035050 con 0x5558c985f4a0 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7c27fc640 1 -- 172.21.15.28:0/604470292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d0031770 con 0x5558c985f4a0 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7c27fc640 1 -- 172.21.15.28:0/604470292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0005610 con 0x5558c985f4a0 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 -- 172.21.15.28:0/604470292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985f4a0 msgr2=0x5558c985f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e0a34640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd7c40a33b0 0x7fd7c40a5870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009860 tx=0x7fd7d400af70 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985f4a0 0x5558c985f880 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0005940 tx=0x7fd7d0005bd0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 -- 172.21.15.28:0/604470292 shutdown_connections 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c986ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c986a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 --2- 172.21.15.28:0/604470292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985f4a0 0x5558c985f880 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 -- 172.21.15.28:0/604470292 >> 172.21.15.28:0/604470292 conn(0x5558c985ba30 msgr2=0x5558c985be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 -- 172.21.15.28:0/604470292 shutdown_connections 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.833+0000 7fd7e4870740 1 -- 172.21.15.28:0/604470292 wait complete. 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 -- start start 2024-09-15T22:04:01.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 Processor -- start 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c986a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985fdc0 0x5558c9867310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c98678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558c986d250 con 0x5558c986a680 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5558c9780050 con 0x5558c985fdc0 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5558c986d3d0 con 0x5558c985f4a0 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.834+0000 7fd7e4870740 10 client.0 _ll_register_callbacks cb 0x5558c97c0160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c98678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c98678e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45374/0 (socket says 172.21.15.28:45374) 2024-09-15T22:04:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 -- 172.21.15.28:0/3075899400 learned_addr learned my addr 172.21.15.28:0/3075899400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c986a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985fdc0 0x5558c9867310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 -- 172.21.15.28:0/3075899400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 msgr2=0x5558c986a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c986a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 -- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985fdc0 msgr2=0x5558c9867310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985fdc0 0x5558c9867310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 -- 172.21.15.28:0/3075899400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c95363c0 con 0x5558c986a680 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985fdc0 0x5558c9867310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c98678e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0006010 tx=0x7fd7d003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c547ed4dd53c9bc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/3075899400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0035070 con 0x5558c986a680 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/3075899400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd7d0004260 con 0x5558c986a680 2024-09-15T22:04:01.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7d0031770 con 0x5558c986a680 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/3075899400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd7d0032dd0 con 0x5558c986a680 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.835+0000 7fd7e4870740 10 client.16458 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5558c9878c90 con 0x5558c986a680 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/3075899400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d0031770 con 0x5558c986a680 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/3075899400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd7d00038f0 con 0x5558c986a680 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7e4870740 10 client.16458 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7e4870740 10 client.16458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7e4870740 20 client.16458 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7e4870740 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c9879fe0 0x5558c987c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7e4870740 4 client.16458 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558c97d2850 con 0x5558c9879fe0 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.836+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c9879fe0 0x5558c987c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.837+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c9879fe0 0x5558c987c3d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c380e57278af43a0 server_cookie=d96986acd86ee482 in_seq=0 out_seq=0 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.837+0000 7fd7bb7fe640 10 client.16458 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.837+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/3075899400 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5558c97d2850 con 0x5558c9879fe0 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.837+0000 7fd7bb7fe640 10 client.16458 handle_command_reply: tid=0 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.837+0000 7fd7e4870740 1 client.16458 shutdown 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 msgr2=0x5558c98678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c98678e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0006010 tx=0x7fd7d003f040 comp rx=0 tx=0).stop 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 shutdown_connections 2024-09-15T22:04:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c9879fe0 0x5558c987c3d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c98678e0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985fdc0 0x5558c9867310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7bb7fe640 0 client.16458 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 --2- 172.21.15.28:0/3075899400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c986a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 >> 172.21.15.28:0/3075899400 conn(0x5558c985ba30 msgr2=0x5558c9861760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 shutdown_connections 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 1 -- 172.21.15.28:0/3075899400 wait complete. 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.838+0000 7fd7e4870740 20 client.16458 trim_cache size 0 max 16384 2024-09-15T22:04:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.855+0000 7fd7e4870740 1 Processor -- start 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.855+0000 7fd7e4870740 1 -- start start 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985f4a0 0x5558c9865be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c987f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c987f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7e4870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1b1040 con 0x5558c986a680 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7e4870740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1b2b80 con 0x5558c985f4a0 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7e4870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1b1710 con 0x5558c985fdc0 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c987f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c987f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45378/0 (socket says 172.21.15.28:45378) 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 -- 172.21.15.28:0/4052886054 learned_addr learned my addr 172.21.15.28:0/4052886054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c987f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985f4a0 0x5558c9865be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 -- 172.21.15.28:0/4052886054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 msgr2=0x5558c987f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c987f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 -- 172.21.15.28:0/4052886054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985f4a0 msgr2=0x5558c9865be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985f4a0 0x5558c9865be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 -- 172.21.15.28:0/4052886054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c985c6c0 con 0x5558c986a680 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c987f940 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0003cc0 tx=0x7fd7d002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616af5ed64c509ad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7c1ffb640 1 -- 172.21.15.28:0/4052886054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0031670 con 0x5558c986a680 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7c1ffb640 1 -- 172.21.15.28:0/4052886054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d0003ad0 con 0x5558c986a680 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.856+0000 7fd7c1ffb640 1 -- 172.21.15.28:0/4052886054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0031e30 con 0x5558c986a680 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 -- 172.21.15.28:0/4052886054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 msgr2=0x5558c987f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c987f940 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0003cc0 tx=0x7fd7d002a480 comp rx=0 tx=0).stop 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 -- 172.21.15.28:0/4052886054 shutdown_connections 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c986a680 0x5558c987f940 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985fdc0 0x5558c987f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 --2- 172.21.15.28:0/4052886054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c985f4a0 0x5558c9865be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 -- 172.21.15.28:0/4052886054 >> 172.21.15.28:0/4052886054 conn(0x5558c985ba30 msgr2=0x5558c9869390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 -- 172.21.15.28:0/4052886054 shutdown_connections 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 -- 172.21.15.28:0/4052886054 wait complete. 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.857+0000 7fd7e4870740 1 -- start start 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 1 Processor -- start 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c98d88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985fdc0 0x5558c98d8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c98d9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1b24b0 con 0x5558c985fdc0 2024-09-15T22:04:01.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1b1de0 con 0x5558c986a680 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc1af020 con 0x5558c985f4a0 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7e4870740 10 client.0 _ll_register_callbacks cb 0x7fd7cc00b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985fdc0 0x5558c98d8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985fdc0 0x5558c98d8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45392/0 (socket says 172.21.15.28:45392) 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 -- 172.21.15.28:0/2831128642 learned_addr learned my addr 172.21.15.28:0/2831128642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c98d88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c98d9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 -- 172.21.15.28:0/2831128642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 msgr2=0x5558c98d88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c98d88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 -- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 msgr2=0x5558c98d9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:01.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c98d9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.858+0000 7fd7dbfff640 1 -- 172.21.15.28:0/2831128642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c98e3c20 con 0x5558c985fdc0 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7dbfff640 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985fdc0 0x5558c98d8df0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x5558c985c890 tx=0x7fd7cc00bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f92ce1a5fe7d742 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/2831128642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc013d70 con 0x5558c985fdc0 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/2831128642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fd7cc01d420 con 0x5558c985fdc0 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7d0003ad0 con 0x5558c985fdc0 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:01.549 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:04:01.550 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:04:01.550 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/2831128642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd7cc02e050 con 0x5558c985fdc0 2024-09-15T22:04:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 10 client.16464 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5558c98e4f90 con 0x5558c985fdc0 2024-09-15T22:04:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/2831128642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d0003ad0 con 0x5558c985fdc0 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/2831128642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd7cc011960 con 0x5558c985fdc0 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 10 client.16464 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 10 client.16464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 20 client.16464 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c98e5710 0x5558c98e7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 4 client.16464 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.859+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5558c97ccd60 con 0x5558c98e5710 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.860+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c98e5710 0x5558c98e7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.860+0000 7fd7bb7fe640 10 client.16464 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.860+0000 7fd7db7fe640 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c98e5710 0x5558c98e7b30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18b7e4ef3f4f16bd server_cookie=de77e456a883fad6 in_seq=0 out_seq=0 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.860+0000 7fd7bb7fe640 1 -- 172.21.15.28:0/2831128642 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5558c97ccd60 con 0x5558c98e5710 2024-09-15T22:04:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.860+0000 7fd7bb7fe640 10 client.16464 handle_command_reply: tid=0 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.860+0000 7fd7e4870740 1 client.16464 shutdown 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985fdc0 msgr2=0x5558c98d8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985fdc0 0x5558c98d8df0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x5558c985c890 tx=0x7fd7cc00bfc0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 shutdown_connections 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5558c98e5710 0x5558c98e7b30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5558c986a680 0x5558c98d9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558c985fdc0 0x5558c98d8df0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 --2- 172.21.15.28:0/2831128642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558c985f4a0 0x5558c98d88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7bb7fe640 0 client.16464 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 >> 172.21.15.28:0/2831128642 conn(0x5558c985ba30 msgr2=0x5558c985a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 shutdown_connections 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 1 -- 172.21.15.28:0/2831128642 wait complete. 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e4870740 20 client.16464 trim_cache size 0 max 16384 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd7c40a33b0 msgr2=0x7fd7c40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd7c40a33b0 0x7fd7c40a5870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009860 tx=0x7fd7d400af70 comp rx=0 tx=0).stop 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc10c170 msgr2=0x7fd7dc101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc10c170 0x7fd7dc101420 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400eaa0 tx=0x7fd7d400ef60 comp rx=0 tx=0).stop 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 shutdown_connections 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd7c40a33b0 0x7fd7c40a5870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc10c170 0x7fd7dc101420 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd7dc103660 0x7fd7dc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 --2- 172.21.15.28:0/1214602473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7dc102d40 0x7fd7dc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.861+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 >> 172.21.15.28:0/1214602473 conn(0x7fd7dc0fc7a0 msgr2=0x7fd7dc0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.862+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 shutdown_connections 2024-09-15T22:04:01.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:00.862+0000 7fd7e2497640 1 -- 172.21.15.28:0/1214602473 wait complete. 2024-09-15T22:04:02.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:01 smithi203 ceph-mon[35546]: pgmap v203: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-09-15T22:04:02.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:01 smithi174 ceph-mon[35244]: pgmap v203: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-09-15T22:04:02.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:01 smithi028 ceph-mon[30969]: pgmap v203: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-09-15T22:04:02.733 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:04:02.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:04:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.308+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/750399927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 msgr2=0x7f9f94107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.308+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/750399927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94107b80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009930 tx=0x7f9f8402a420 comp rx=0 tx=0).stop 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/750399927 shutdown_connections 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/750399927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f94108290 0x7f9f9410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/750399927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94107b80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/750399927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f94102d90 0x7f9f94103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/750399927 >> 172.21.15.28:0/750399927 conn(0x7f9f940fc7d0 msgr2=0x7f9f940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/750399927 shutdown_connections 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/750399927 wait complete. 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 Processor -- start 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.309+0000 7f9f9c5c2640 1 -- start start 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9c5c2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f94102d90 0x7f9f94198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9c5c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9c5c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f94108290 0x7f9f9419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9c5c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f94112090 con 0x7f9f94103740 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9c5c2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f94111f10 con 0x7f9f94108290 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9c5c2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f94112210 con 0x7f9f94102d90 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45404/0 (socket says 172.21.15.28:45404) 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 -- 172.21.15.28:0/3995090733 learned_addr learned my addr 172.21.15.28:0/3995090733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f94102d90 0x7f9f94198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f94108290 0x7f9f9419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 -- 172.21.15.28:0/3995090733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f94102d90 msgr2=0x7f9f94198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f94102d90 0x7f9f94198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 -- 172.21.15.28:0/3995090733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f94108290 msgr2=0x7f9f9419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f94108290 0x7f9f9419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 -- 172.21.15.28:0/3995090733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9419d540 con 0x7f9f94103740 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f94102d90 0x7f9f94198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f94108290 0x7f9f9419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.310+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94198ad0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9f84005940 tx=0x7f9f840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.311+0000 7f9f8b7fe640 1 -- 172.21.15.28:0/3995090733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8403f030 con 0x7f9f94103740 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.311+0000 7f9f8b7fe640 1 -- 172.21.15.28:0/3995090733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f84003ce0 con 0x7f9f94103740 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.311+0000 7f9f8b7fe640 1 -- 172.21.15.28:0/3995090733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84035070 con 0x7f9f94103740 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.311+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9419d7d0 con 0x7f9f94103740 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.311+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f941a5110 con 0x7f9f94103740 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.313+0000 7f9f8b7fe640 1 -- 172.21.15.28:0/3995090733 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f9f84042030 con 0x7f9f94103740 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.313+0000 7f9f9e99b740 1 Processor -- start 2024-09-15T22:04:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9e99b740 1 -- start start 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d181cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d18c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d18eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e0d0e7c10 con 0x558e0d1818f0 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e0d08f520 con 0x558e0d18cad0 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558e0d0e3bd0 con 0x558e0d182210 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9ab5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d181cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9ab5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d181cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45416/0 (socket says 172.21.15.28:45416) 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9ab5f640 1 -- 172.21.15.28:0/2606805041 learned_addr learned my addr 172.21.15.28:0/2606805041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f8b7fe640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9f800a33b0 0x7f9f800a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.314+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d18c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d18eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9f800a33b0 0x7f9f800a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f8b7fe640 1 -- 172.21.15.28:0/3995090733 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9f840f0700 con 0x7f9f94103740 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9a35e640 1 -- 172.21.15.28:0/2606805041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 msgr2=0x558e0d18eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d18eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9a35e640 1 -- 172.21.15.28:0/2606805041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 msgr2=0x558e0d181cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d181cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9a35e640 1 -- 172.21.15.28:0/2606805041 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e0cebc490 con 0x558e0d182210 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d181cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9f800a33b0 0x7f9f800a5870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c005cc0 tx=0x7f9f8c004ab0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d18c590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000bc80 tx=0x7f9f9000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a682c8698e5ea2fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.315+0000 7f9f88ff9640 1 -- 172.21.15.28:0/2606805041 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90015070 con 0x558e0d182210 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f88ff9640 1 -- 172.21.15.28:0/2606805041 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9001a940 con 0x558e0d182210 2024-09-15T22:04:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f88ff9640 1 -- 172.21.15.28:0/2606805041 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f900119a0 con 0x558e0d182210 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 -- 172.21.15.28:0/2606805041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 msgr2=0x558e0d18c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d18c590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000bc80 tx=0x7f9f9000f730 comp rx=0 tx=0).stop 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 -- 172.21.15.28:0/2606805041 shutdown_connections 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d18eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d18c590 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/2606805041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d181cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 -- 172.21.15.28:0/2606805041 >> 172.21.15.28:0/2606805041 conn(0x558e0d17de80 msgr2=0x558e0d17e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 -- 172.21.15.28:0/2606805041 shutdown_connections 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.316+0000 7f9f9e99b740 1 -- 172.21.15.28:0/2606805041 wait complete. 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 -- start start 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 Processor -- start 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 0x558e0d189960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d182210 0x558e0d189ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d18cad0 0x558e0d18a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e0d18f6a0 con 0x558e0d182210 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e0d070700 con 0x558e0d1818f0 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558e0d18f820 con 0x558e0d18cad0 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f9e99b740 10 client.0 _ll_register_callbacks cb 0x558e0d091ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f99b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d182210 0x558e0d189ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f99b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d182210 0x558e0d189ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45432/0 (socket says 172.21.15.28:45432) 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.317+0000 7f9f99b5d640 1 -- 172.21.15.28:0/1224843008 learned_addr learned my addr 172.21.15.28:0/1224843008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f99b5d640 1 -- 172.21.15.28:0/1224843008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d18cad0 msgr2=0x558e0d18a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d18cad0 0x558e0d18a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f99b5d640 1 -- 172.21.15.28:0/1224843008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 msgr2=0x558e0d189960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 0x558e0d189960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f99b5d640 1 -- 172.21.15.28:0/1224843008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e0d19cf50 con 0x558e0d182210 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d182210 0x558e0d189ea0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9f840b8a80 tx=0x7f9f8403b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f4bc0abcf85875 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f7dffb640 1 -- 172.21.15.28:0/1224843008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84049050 con 0x558e0d182210 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f7dffb640 1 -- 172.21.15.28:0/1224843008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9f84003800 con 0x558e0d182210 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f9001a940 con 0x558e0d182210 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f7dffb640 1 -- 172.21.15.28:0/1224843008 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9f840044e0 con 0x558e0d182210 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f9e99b740 10 client.16476 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558e0d19fc50 con 0x558e0d182210 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f7dffb640 1 -- 172.21.15.28:0/1224843008 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9001a940 con 0x558e0d182210 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f7dffb640 1 -- 172.21.15.28:0/1224843008 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9f840b8c90 con 0x558e0d182210 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f9e99b740 10 client.16476 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.318+0000 7f9f9e99b740 10 client.16476 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.319+0000 7f9f9e99b740 20 client.16476 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.319+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d1a1060 0x558e0d1a34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.319+0000 7f9f9e99b740 4 client.16476 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.319+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e0d0f5340 con 0x558e0d1a1060 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.319+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d1a1060 0x558e0d1a34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.319+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d1a1060 0x558e0d1a34f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=979cfcf428d4a33d server_cookie=bac8e3e0cab0ced3 in_seq=0 out_seq=0 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.319+0000 7f9f7dffb640 10 client.16476 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f7dffb640 1 -- 172.21.15.28:0/1224843008 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e0d0f5340 con 0x558e0d1a1060 2024-09-15T22:04:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f7dffb640 10 client.16476 handle_command_reply: tid=0 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 client.16476 shutdown 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d182210 msgr2=0x558e0d189ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d182210 0x558e0d189ea0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9f840b8a80 tx=0x7f9f8403b000 comp rx=0 tx=0).stop 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 shutdown_connections 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d1a1060 0x558e0d1a34f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d18cad0 0x558e0d18a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d182210 0x558e0d189ea0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/1224843008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 0x558e0d189960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 >> 172.21.15.28:0/1224843008 conn(0x558e0d17de80 msgr2=0x558e0d183bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f7dffb640 0 client.16476 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 shutdown_connections 2024-09-15T22:04:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.320+0000 7f9f9e99b740 1 -- 172.21.15.28:0/1224843008 wait complete. 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.321+0000 7f9f9e99b740 20 client.16476 trim_cache size 0 max 16384 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 Processor -- start 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 -- start start 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 0x558e0d184350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d18cad0 0x558e0d1a9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f941acd90 con 0x558e0d18cad0 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f941ae8d0 con 0x558e0d1818f0 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f941ad460 con 0x558e0d182210 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9ab5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d18cad0 0x558e0d1a9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9ab5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d18cad0 0x558e0d1a9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45436/0 (socket says 172.21.15.28:45436) 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.337+0000 7f9f9ab5f640 1 -- 172.21.15.28:0/878184249 learned_addr learned my addr 172.21.15.28:0/878184249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f9ab5f640 1 -- 172.21.15.28:0/878184249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 msgr2=0x558e0d1a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 0x558e0d184350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1a8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f9ab5f640 1 -- 172.21.15.28:0/878184249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 msgr2=0x558e0d184350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 0x558e0d184350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f9ab5f640 1 -- 172.21.15.28:0/878184249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e0d17dcb0 con 0x558e0d18cad0 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d18cad0 0x558e0d1a9340 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c007300 tx=0x7f9f8c0075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78896def2afd199e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.338+0000 7f9f7ffff640 1 -- 172.21.15.28:0/878184249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c024030 con 0x558e0d18cad0 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f7ffff640 1 -- 172.21.15.28:0/878184249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f8c003e50 con 0x558e0d18cad0 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f7ffff640 1 -- 172.21.15.28:0/878184249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0206d0 con 0x558e0d18cad0 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 -- 172.21.15.28:0/878184249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d18cad0 msgr2=0x558e0d1a9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d18cad0 0x558e0d1a9340 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c007300 tx=0x7f9f8c0075f0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 -- 172.21.15.28:0/878184249 shutdown_connections 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d18cad0 0x558e0d1a9340 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/878184249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d1818f0 0x558e0d184350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 -- 172.21.15.28:0/878184249 >> 172.21.15.28:0/878184249 conn(0x558e0d17de80 msgr2=0x558e0d18b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 -- 172.21.15.28:0/878184249 shutdown_connections 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.339+0000 7f9f9e99b740 1 -- 172.21.15.28:0/878184249 wait complete. 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 -- start start 2024-09-15T22:04:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 Processor -- start 2024-09-15T22:04:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d194f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1954d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d20add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f941ae200 con 0x558e0d1818f0 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f941adb30 con 0x558e0d18cad0 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f941aad70 con 0x558e0d182210 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9e99b740 10 client.0 _ll_register_callbacks cb 0x7f9f84009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9a35e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d194f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9a35e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d194f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45440/0 (socket says 172.21.15.28:45440) 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.340+0000 7f9f9a35e640 1 -- 172.21.15.28:0/4034297905 learned_addr learned my addr 172.21.15.28:0/4034297905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1954d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d20add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9a35e640 1 -- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 msgr2=0x558e0d1954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1954d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9a35e640 1 -- 172.21.15.28:0/4034297905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 msgr2=0x558e0d20add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d20add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9a35e640 1 -- 172.21.15.28:0/4034297905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e0d18a930 con 0x558e0d1818f0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9ab5f640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d20add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9a35e640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d194f90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000bc80 tx=0x7f9f90011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51134793cd34c68b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f7dffb640 1 -- 172.21.15.28:0/4034297905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90015040 con 0x558e0d1818f0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f7dffb640 1 -- 172.21.15.28:0/4034297905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f9f90004440 con 0x558e0d1818f0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f8c003e50 con 0x558e0d1818f0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f7dffb640 1 -- 172.21.15.28:0/4034297905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9f90033050 con 0x558e0d1818f0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9e99b740 10 client.16482 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.341+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558e0d20c150 con 0x558e0d1818f0 2024-09-15T22:04:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f7dffb640 1 -- 172.21.15.28:0/4034297905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f8c003e50 con 0x558e0d1818f0 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f7dffb640 1 -- 172.21.15.28:0/4034297905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9f90011820 con 0x558e0d1818f0 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f9e99b740 10 client.16482 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f9e99b740 10 client.16482 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f9e99b740 20 client.16482 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d20c660 0x558e0d20ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f9e99b740 4 client.16482 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.f.json"}] 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.f.json"}) -- 0x558e0d0ef690 con 0x558e0d20c660 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.342+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d20c660 0x558e0d20ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.343+0000 7f9f99b5d640 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d20c660 0x558e0d20ea50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=40575b1dcce1b79f server_cookie=233cbb79437dc36b in_seq=0 out_seq=0 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.343+0000 7f9f7dffb640 10 client.16482 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f7dffb640 1 -- 172.21.15.28:0/4034297905 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x558e0d0ef690 con 0x558e0d20c660 2024-09-15T22:04:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f7dffb640 10 client.16482 handle_command_reply: tid=0 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 client.16482 shutdown 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 msgr2=0x558e0d194f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d194f90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000bc80 tx=0x7f9f90011440 comp rx=0 tx=0).stop 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 shutdown_connections 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x558e0d20c660 0x558e0d20ea50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e0d18cad0 0x558e0d20add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558e0d182210 0x558e0d1954d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 --2- 172.21.15.28:0/4034297905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e0d1818f0 0x558e0d194f90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 >> 172.21.15.28:0/4034297905 conn(0x558e0d17de80 msgr2=0x558e0d17c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f7dffb640 0 client.16482 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 shutdown_connections 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 1 -- 172.21.15.28:0/4034297905 wait complete. 2024-09-15T22:04:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.344+0000 7f9f9e99b740 20 client.16482 trim_cache size 0 max 16384 2024-09-15T22:04:03.363 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:03.363 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/var/run/ceph/ops-ebf3369a-1-mds.f.json" 2024-09-15T22:04:03.363 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9f800a33b0 msgr2=0x7f9f800a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9f800a33b0 0x7f9f800a5870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c005cc0 tx=0x7f9f8c004ab0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 msgr2=0x7f9f94198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94198ad0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f9f84005940 tx=0x7f9f840042f0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 shutdown_connections 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9f800a33b0 0x7f9f800a5870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f94108290 0x7f9f9419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f94103740 0x7f9f94198ad0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 --2- 172.21.15.28:0/3995090733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f94102d90 0x7f9f94198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 >> 172.21.15.28:0/3995090733 conn(0x7f9f940fc7d0 msgr2=0x7f9f94106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 shutdown_connections 2024-09-15T22:04:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:03.345+0000 7f9f9c5c2640 1 -- 172.21.15.28:0/3995090733 wait complete. 2024-09-15T22:04:04.085 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 3 (h) to a remote file /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.h.json 2024-09-15T22:04:04.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 ops --flags=locks --path=/var/run/ceph/ops-ebf3369a-1-mds.h.json 2024-09-15T22:04:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:04.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:03 smithi203 ceph-mon[35546]: pgmap v204: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 25 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-09-15T22:04:04.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:03 smithi028 ceph-mon[30969]: pgmap v204: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 25 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-09-15T22:04:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:03 smithi174 ceph-mon[35244]: pgmap v204: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 25 KiB/s rd, 56 KiB/s wr, 32 op/s 2024-09-15T22:04:04.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 -- 172.21.15.28:0/1653468571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 msgr2=0x7f19c01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 --2- 172.21.15.28:0/1653468571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c01067e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009900 tx=0x7f19b402a3f0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 -- 172.21.15.28:0/1653468571 shutdown_connections 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 --2- 172.21.15.28:0/1653468571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19c0106f10 0x7f19c010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 --2- 172.21.15.28:0/1653468571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c01067e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 --2- 172.21.15.28:0/1653468571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c0101de0 0x7f19c01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 -- 172.21.15.28:0/1653468571 >> 172.21.15.28:0/1653468571 conn(0x7f19c00fd8c0 msgr2=0x7f19c00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 -- 172.21.15.28:0/1653468571 shutdown_connections 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.589+0000 7f19c6777640 1 -- 172.21.15.28:0/1653468571 wait complete. 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 Processor -- start 2024-09-15T22:04:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 -- start start 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c0101de0 0x7f19c019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19c0106f10 0x7f19c010db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0111f70 con 0x7f19c0102790 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f19c0111df0 con 0x7f19c0106f10 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19c6777640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19c01120f0 con 0x7f19c0101de0 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.590+0000 7f19bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c019f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45456/0 (socket says 172.21.15.28:45456) 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 -- 172.21.15.28:0/3905425650 learned_addr learned my addr 172.21.15.28:0/3905425650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bffff640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c0101de0 0x7f19c019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19c4d14640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19c0106f10 0x7f19c010db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 -- 172.21.15.28:0/3905425650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c0101de0 msgr2=0x7f19c019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c0101de0 0x7f19c019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 -- 172.21.15.28:0/3905425650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19c0106f10 msgr2=0x7f19c010db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19c0106f10 0x7f19c010db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 -- 172.21.15.28:0/3905425650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c010e470 con 0x7f19c0102790 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19c4d14640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19c0106f10 0x7f19c010db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bf7fe640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c019f560 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f19b4006010 tx=0x7f19b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bd7fa640 1 -- 172.21.15.28:0/3905425650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b403f030 con 0x7f19c0102790 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bd7fa640 1 -- 172.21.15.28:0/3905425650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19b4003ce0 con 0x7f19c0102790 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19c010e6a0 con 0x7f19c0102790 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19bd7fa640 1 -- 172.21.15.28:0/3905425650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4035070 con 0x7f19c0102790 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.591+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19c019ac40 con 0x7f19c0102790 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.592+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19c01032c0 con 0x7f19c0102790 2024-09-15T22:04:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.593+0000 7f19bd7fa640 1 -- 172.21.15.28:0/3905425650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f19b4042030 con 0x7f19c0102790 2024-09-15T22:04:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.594+0000 7f19bd7fa640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19a80a33b0 0x7f19a80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.594+0000 7f19bffff640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19a80a33b0 0x7f19a80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.595+0000 7f19bd7fa640 1 -- 172.21.15.28:0/3905425650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f19b40f0a40 con 0x7f19c0102790 2024-09-15T22:04:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.595+0000 7f19bffff640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19a80a33b0 0x7f19a80a5870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f19b0005fd0 tx=0x7f19b000b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.598+0000 7f19bd7fa640 1 -- 172.21.15.28:0/3905425650 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f19b40bd120 con 0x7f19c0102790 2024-09-15T22:04:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:04.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.828+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19c019b660 con 0x7f19c0102790 2024-09-15T22:04:04.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.829+0000 7f19bd7fa640 1 -- 172.21.15.28:0/3905425650 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11986 (secure 0 0 0) 0x7f19b40bca20 con 0x7f19c0102790 2024-09-15T22:04:04.831 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:04:04.832 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"btime":"2024-09-15T22:01:08:357214+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:01:08.288094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:04:04.832 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-09-15T22:04:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19a80a33b0 msgr2=0x7f19a80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19a80a33b0 0x7f19a80a5870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f19b0005fd0 tx=0x7f19b000b390 comp rx=0 tx=0).stop 2024-09-15T22:04:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 msgr2=0x7f19c019f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c019f560 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f19b4006010 tx=0x7f19b40042f0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 shutdown_connections 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f19a80a33b0 0x7f19a80a5870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19c0106f10 0x7f19c010db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19c0102790 0x7f19c019f560 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 --2- 172.21.15.28:0/3905425650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19c0101de0 0x7f19c019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 >> 172.21.15.28:0/3905425650 conn(0x7f19c00fd8c0 msgr2=0x7f19c00fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 shutdown_connections 2024-09-15T22:04:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:04.831+0000 7f19c6777640 1 -- 172.21.15.28:0/3905425650 wait complete. 2024-09-15T22:04:05.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:04 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3905425650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:05.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:04 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3905425650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:05.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:04 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3905425650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:05.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:04:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 -- 172.21.15.28:0/2371255422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 msgr2=0x7f161c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 --2- 172.21.15.28:0/2371255422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c1021c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f160c009900 tx=0x7f160c02a410 comp rx=0 tx=0).stop 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 -- 172.21.15.28:0/2371255422 shutdown_connections 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 --2- 172.21.15.28:0/2371255422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c106f10 0x7f161c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 --2- 172.21.15.28:0/2371255422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f161c102790 0x7f161c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 --2- 172.21.15.28:0/2371255422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c1021c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 -- 172.21.15.28:0/2371255422 >> 172.21.15.28:0/2371255422 conn(0x7f161c0fd880 msgr2=0x7f161c0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 -- 172.21.15.28:0/2371255422 shutdown_connections 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.720+0000 7f1621197640 1 -- 172.21.15.28:0/2371255422 wait complete. 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 Processor -- start 2024-09-15T22:04:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 -- start start 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f161c102790 0x7f161c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c106f10 0x7f161c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f161c111f70 con 0x7f161c101de0 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f161c111df0 con 0x7f161c106f10 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.721+0000 7f1621197640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1120f0 con 0x7f161c102790 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45482/0 (socket says 172.21.15.28:45482) 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 -- 172.21.15.28:0/107149005 learned_addr learned my addr 172.21.15.28:0/107149005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161a59c640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f161c102790 0x7f161c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161b59e640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c106f10 0x7f161c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 -- 172.21.15.28:0/107149005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f161c102790 msgr2=0x7f161c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f161c102790 0x7f161c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 -- 172.21.15.28:0/107149005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c106f10 msgr2=0x7f161c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c106f10 0x7f161c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 -- 172.21.15.28:0/107149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c19d3c0 con 0x7f161c101de0 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161a59c640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f161c102790 0x7f161c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f161ad9d640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c198410 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f160c006010 tx=0x7f160c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f1603fff640 1 -- 172.21.15.28:0/107149005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c03f030 con 0x7f161c101de0 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f1603fff640 1 -- 172.21.15.28:0/107149005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f160c003ce0 con 0x7f161c101de0 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161c19d650 con 0x7f161c101de0 2024-09-15T22:04:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.722+0000 7f1603fff640 1 -- 172.21.15.28:0/107149005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c035070 con 0x7f161c101de0 2024-09-15T22:04:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.723+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161c1a4f90 con 0x7f161c101de0 2024-09-15T22:04:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.725+0000 7f1603fff640 1 -- 172.21.15.28:0/107149005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f160c042030 con 0x7f161c101de0 2024-09-15T22:04:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1603fff640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f16040a33b0 0x7f16040a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f161a59c640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f16040a33b0 0x7f16040a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1603fff640 1 -- 172.21.15.28:0/107149005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f160c0f0740 con 0x7f161c101de0 2024-09-15T22:04:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 Processor -- start 2024-09-15T22:04:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 -- start start 2024-09-15T22:04:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f161a59c640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f16040a33b0 0x7f16040a5870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f16100058c0 tx=0x7f1610005850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c94d00 0x55e1d3c950e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3c9f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1d3bf6c10 con 0x55e1d3c9fee0 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e1d3b9e520 con 0x55e1d3c95620 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.726+0000 7f1623570740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e1d3bf2bd0 con 0x55e1d3c94d00 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3c9f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45492/0 (socket says 172.21.15.28:45492) 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 -- 172.21.15.28:0/984432651 learned_addr learned my addr 172.21.15.28:0/984432651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161ad9d640 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c94d00 0x55e1d3c950e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 -- 172.21.15.28:0/984432651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c94d00 msgr2=0x55e1d3c950e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c94d00 0x55e1d3c950e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 -- 172.21.15.28:0/984432651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 msgr2=0x55e1d3c9f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3c9f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 -- 172.21.15.28:0/984432651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d39cb490 con 0x55e1d3c9fee0 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161a59c640 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3c9f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.727+0000 7f161b59e640 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca22d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f161400ae10 tx=0x7f161400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51fc9897f555b68 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.728+0000 7f16017fa640 1 -- 172.21.15.28:0/984432651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614003b70 con 0x55e1d3c9fee0 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.728+0000 7f16017fa640 1 -- 172.21.15.28:0/984432651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1614003d10 con 0x55e1d3c9fee0 2024-09-15T22:04:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.728+0000 7f16017fa640 1 -- 172.21.15.28:0/984432651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614010b40 con 0x55e1d3c9fee0 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 -- 172.21.15.28:0/984432651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c9fee0 msgr2=0x55e1d3ca22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca22d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f161400ae10 tx=0x7f161400f790 comp rx=0 tx=0).stop 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 -- 172.21.15.28:0/984432651 shutdown_connections 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca22d0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3c9f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 --2- 172.21.15.28:0/984432651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c94d00 0x55e1d3c950e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 -- 172.21.15.28:0/984432651 >> 172.21.15.28:0/984432651 conn(0x55e1d3c91290 msgr2=0x55e1d3c91670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 -- 172.21.15.28:0/984432651 shutdown_connections 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 -- 172.21.15.28:0/984432651 wait complete. 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 -- start start 2024-09-15T22:04:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.729+0000 7f1623570740 1 Processor -- start 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f1623570740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3c9cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f1623570740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c95620 0x55e1d3c9d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3c9cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f1623570740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3c9cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45498/0 (socket says 172.21.15.28:45498) 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 -- 172.21.15.28:0/3468822843 learned_addr learned my addr 172.21.15.28:0/3468822843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e1d3ca2b50 con 0x55e1d3c94d00 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e1d3b7f700 con 0x55e1d3c9fee0 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e1d3ca2cd0 con 0x55e1d3c95620 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f1623570740 10 client.0 _ll_register_callbacks cb 0x55e1d3ba0ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161a59c640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c95620 0x55e1d3c9d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161b59e640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 -- 172.21.15.28:0/3468822843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c95620 msgr2=0x55e1d3c9d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c95620 0x55e1d3c9d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 -- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c9fee0 msgr2=0x55e1d3c9d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161ad9d640 1 -- 172.21.15.28:0/3468822843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d3cac040 con 0x55e1d3c94d00 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.730+0000 7f161b59e640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1614003d10 con 0x55e1d3c94d00 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f161ad9d640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3c9cdd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f160c000c00 tx=0x7f160c0bc550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfa1585b94edea5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f15feff5640 10 client.16500 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f15feff5640 1 -- 172.21.15.28:0/3468822843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c0d9020 con 0x55e1d3c94d00 2024-09-15T22:04:05.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f15feff5640 1 -- 172.21.15.28:0/3468822843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f160c0f0390 con 0x55e1d3c94d00 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f15feff5640 1 -- 172.21.15.28:0/3468822843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f160c049050 con 0x55e1d3c94d00 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f1623570740 10 client.16500 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55e1d3caec00 con 0x55e1d3c94d00 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f15feff5640 1 -- 172.21.15.28:0/3468822843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f160c0f0070 con 0x55e1d3c94d00 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f1623570740 10 client.16500 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.731+0000 7f1623570740 10 client.16500 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.732+0000 7f1623570740 20 client.16500 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.732+0000 7f1623570740 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3cb0010 0x55e1d3cb24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.732+0000 7f1623570740 4 client.16500 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.732+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1d3c04340 con 0x55e1d3cb0010 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.732+0000 7f161a59c640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3cb0010 0x55e1d3cb24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.732+0000 7f161a59c640 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3cb0010 0x55e1d3cb24a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8287bfadb2558216 server_cookie=d4bc0e60b9bb35be in_seq=0 out_seq=0 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.732+0000 7f15feff5640 10 client.16500 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.733+0000 7f15feff5640 1 -- 172.21.15.28:0/3468822843 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e1d3c04340 con 0x55e1d3cb0010 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.733+0000 7f15feff5640 10 client.16500 handle_command_reply: tid=0 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.733+0000 7f1623570740 1 client.16500 shutdown 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.733+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 msgr2=0x55e1d3c9cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.733+0000 7f1623570740 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3c9cdd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f160c000c00 tx=0x7f160c0bc550 comp rx=0 tx=0).stop 2024-09-15T22:04:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.733+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 shutdown_connections 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.733+0000 7f1623570740 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f1623570740 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c95620 0x55e1d3c9d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f1623570740 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3cb0010 0x55e1d3cb24a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f15feff5640 0 client.16500 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f1623570740 1 --2- 172.21.15.28:0/3468822843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3c9cdd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 >> 172.21.15.28:0/3468822843 conn(0x55e1d3c91290 msgr2=0x55e1d3c8bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 shutdown_connections 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f1623570740 1 -- 172.21.15.28:0/3468822843 wait complete. 2024-09-15T22:04:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.734+0000 7f1623570740 20 client.16500 trim_cache size 0 max 16384 2024-09-15T22:04:05.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 Processor -- start 2024-09-15T22:04:05.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 -- start start 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3cb7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3cb82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1acd30 con 0x55e1d3c94d00 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1ae870 con 0x55e1d3c95620 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f1623570740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1ad400 con 0x55e1d3c9fee0 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.751+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3cb7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3cb7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45510/0 (socket says 172.21.15.28:45510) 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 -- 172.21.15.28:0/883250407 learned_addr learned my addr 172.21.15.28:0/883250407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161b59e640 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161a59c640 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3cb82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 -- 172.21.15.28:0/883250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 msgr2=0x55e1d3c9a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 -- 172.21.15.28:0/883250407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 msgr2=0x55e1d3cb82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3cb82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 -- 172.21.15.28:0/883250407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d3cb2b20 con 0x55e1d3c94d00 2024-09-15T22:04:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f161ad9d640 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3cb7db0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f160c0bc480 tx=0x7f160c0c0000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdbe08d71777a49d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f1600ff9640 1 -- 172.21.15.28:0/883250407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c0d9070 con 0x55e1d3c94d00 2024-09-15T22:04:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f1600ff9640 1 -- 172.21.15.28:0/883250407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1614003d10 con 0x55e1d3c94d00 2024-09-15T22:04:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.752+0000 7f1600ff9640 1 -- 172.21.15.28:0/883250407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c0c53f0 con 0x55e1d3c94d00 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 -- 172.21.15.28:0/883250407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 msgr2=0x55e1d3cb7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3cb7db0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f160c0bc480 tx=0x7f160c0c0000 comp rx=0 tx=0).stop 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 -- 172.21.15.28:0/883250407 shutdown_connections 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3c9a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3cb82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 --2- 172.21.15.28:0/883250407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3cb7db0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 -- 172.21.15.28:0/883250407 >> 172.21.15.28:0/883250407 conn(0x55e1d3c91290 msgr2=0x55e1d3c9ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.753+0000 7f1623570740 1 -- 172.21.15.28:0/883250407 shutdown_connections 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 -- 172.21.15.28:0/883250407 wait complete. 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 -- start start 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 Processor -- start 2024-09-15T22:04:05.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3ca8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3ca4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1ae1a0 con 0x55e1d3c94d00 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1adad0 con 0x55e1d3c95620 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1aad10 con 0x55e1d3c9fee0 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.754+0000 7f1623570740 10 client.0 _ll_register_callbacks cb 0x7f160c005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3ca8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3ca8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45522/0 (socket says 172.21.15.28:45522) 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 -- 172.21.15.28:0/1678590732 learned_addr learned my addr 172.21.15.28:0/1678590732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161a59c640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3ca4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161b59e640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 -- 172.21.15.28:0/1678590732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 msgr2=0x55e1d3ca5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 -- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 msgr2=0x55e1d3ca4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3ca4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 -- 172.21.15.28:0/1678590732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d3c92420 con 0x55e1d3c94d00 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161a59c640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3ca4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f161ad9d640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3ca8510 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f160c005ab0 tx=0x7f160c0edbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b291a174257907 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f15feff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f15feff5640 1 -- 172.21.15.28:0/1678590732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c0d9070 con 0x55e1d3c94d00 2024-09-15T22:04:05.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1614003d10 con 0x55e1d3c94d00 2024-09-15T22:04:05.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.755+0000 7f15feff5640 1 -- 172.21.15.28:0/1678590732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f160c0c5160 con 0x55e1d3c94d00 2024-09-15T22:04:05.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f15feff5640 1 -- 172.21.15.28:0/1678590732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f160c03b580 con 0x55e1d3c94d00 2024-09-15T22:04:05.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 10 client.16506 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:05.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e1d3c91f20 con 0x55e1d3c94d00 2024-09-15T22:04:05.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f15feff5640 1 -- 172.21.15.28:0/1678590732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1614003d10 con 0x55e1d3c94d00 2024-09-15T22:04:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f15feff5640 1 -- 172.21.15.28:0/1678590732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f160c0aa030 con 0x55e1d3c94d00 2024-09-15T22:04:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 10 client.16506 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 10 client.16506 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 20 client.16506 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3d1bb50 0x55e1d3d1df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 4 client.16506 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.h.json"}] 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.h.json"}) -- 0x55e1d3bfe690 con 0x55e1d3d1bb50 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.756+0000 7f161a59c640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3d1bb50 0x55e1d3d1df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.757+0000 7f15feff5640 10 client.16506 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.757+0000 7f161a59c640 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3d1bb50 0x55e1d3d1df40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5966dd03fad65175 server_cookie=98bd0b60e37a5541 in_seq=0 out_seq=0 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f15feff5640 1 -- 172.21.15.28:0/1678590732 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x55e1d3bfe690 con 0x55e1d3d1bb50 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f15feff5640 10 client.16506 handle_command_reply: tid=0 2024-09-15T22:04:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 client.16506 shutdown 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 msgr2=0x55e1d3ca8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3ca8510 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f160c005ab0 tx=0x7f160c0edbe0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 shutdown_connections 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e1d3c9fee0 0x55e1d3ca5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e1d3c95620 0x55e1d3ca4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55e1d3d1bb50 0x55e1d3d1df40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 --2- 172.21.15.28:0/1678590732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e1d3c94d00 0x55e1d3ca8510 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f15feff5640 0 client.16506 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 >> 172.21.15.28:0/1678590732 conn(0x55e1d3c91290 msgr2=0x55e1d3c9cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 shutdown_connections 2024-09-15T22:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 1 -- 172.21.15.28:0/1678590732 wait complete. 2024-09-15T22:04:05.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.759+0000 7f1623570740 20 client.16506 trim_cache size 0 max 16384 2024-09-15T22:04:05.766 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/var/run/ceph/ops-ebf3369a-1-mds.h.json" 2024-09-15T22:04:05.766 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f16040a33b0 msgr2=0x7f16040a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f16040a33b0 0x7f16040a5870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f16100058c0 tx=0x7f1610005850 comp rx=0 tx=0).stop 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 msgr2=0x7f161c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c198410 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f160c006010 tx=0x7f160c0042f0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 shutdown_connections 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f16040a33b0 0x7f16040a5870 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c106f10 0x7f161c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f161c102790 0x7f161c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 --2- 172.21.15.28:0/107149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f161c101de0 0x7f161c198410 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 >> 172.21.15.28:0/107149005 conn(0x7f161c0fd880 msgr2=0x7f161c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 shutdown_connections 2024-09-15T22:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:05.760+0000 7f1621197640 1 -- 172.21.15.28:0/107149005 wait complete. 2024-09-15T22:04:06.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:05 smithi028 ceph-mon[30969]: pgmap v205: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 28 op/s 2024-09-15T22:04:06.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:05 smithi203 ceph-mon[35546]: pgmap v205: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 28 op/s 2024-09-15T22:04:06.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:05 smithi174 ceph-mon[35244]: pgmap v205: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 54 KiB/s wr, 28 op/s 2024-09-15T22:04:06.427 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 2 (c) to a remote file /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.c.json 2024-09-15T22:04:06.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 ops --flags=locks --path=/var/run/ceph/ops-ebf3369a-1-mds.c.json 2024-09-15T22:04:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 -- 172.21.15.28:0/1373661575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f886810c0b0 msgr2=0x7f886810e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 --2- 172.21.15.28:0/1373661575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f886810c0b0 0x7f886810e550 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8860009ca0 tx=0x7f886002a950 comp rx=0 tx=0).stop 2024-09-15T22:04:06.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 -- 172.21.15.28:0/1373661575 shutdown_connections 2024-09-15T22:04:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 --2- 172.21.15.28:0/1373661575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f886810c0b0 0x7f886810e550 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 --2- 172.21.15.28:0/1373661575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8868103600 0x7f886810bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 --2- 172.21.15.28:0/1373661575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8868102ce0 0x7f88681030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 -- 172.21.15.28:0/1373661575 >> 172.21.15.28:0/1373661575 conn(0x7f8868077990 msgr2=0x7f88680fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 -- 172.21.15.28:0/1373661575 shutdown_connections 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.791+0000 7f886d713640 1 -- 172.21.15.28:0/1373661575 wait complete. 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 Processor -- start 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 -- start start 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8868102ce0 0x7f88681982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868103600 0x7f88681987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886810c0b0 0x7f886819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8868111e70 con 0x7f8868103600 2024-09-15T22:04:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8868111cf0 con 0x7f886810c0b0 2024-09-15T22:04:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f886d713640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8868111ff0 con 0x7f8868102ce0 2024-09-15T22:04:06.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868103600 0x7f88681987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868103600 0x7f88681987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45544/0 (socket says 172.21.15.28:45544) 2024-09-15T22:04:06.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f88667fc640 1 -- 172.21.15.28:0/3417848411 learned_addr learned my addr 172.21.15.28:0/3417848411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.792+0000 7f8866ffd640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8868102ce0 0x7f88681982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88677fe640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886810c0b0 0x7f886819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88667fc640 1 -- 172.21.15.28:0/3417848411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8868102ce0 msgr2=0x7f88681982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88667fc640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8868102ce0 0x7f88681982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88667fc640 1 -- 172.21.15.28:0/3417848411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886810c0b0 msgr2=0x7f886819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88667fc640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886810c0b0 0x7f886819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88667fc640 1 -- 172.21.15.28:0/3417848411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886819d290 con 0x7f8868103600 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88677fe640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886810c0b0 0x7f886819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f88667fc640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868103600 0x7f88681987e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f885c00d950 tx=0x7f885c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f884ffff640 1 -- 172.21.15.28:0/3417848411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c017070 con 0x7f8868103600 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f884ffff640 1 -- 172.21.15.28:0/3417848411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f885c009c30 con 0x7f8868103600 2024-09-15T22:04:06.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886819d580 con 0x7f8868103600 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f884ffff640 1 -- 172.21.15.28:0/3417848411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c00f6b0 con 0x7f8868103600 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.793+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88681a4e10 con 0x7f8868103600 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f884ffff640 1 -- 172.21.15.28:0/3417848411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f885c0109d0 con 0x7f8868103600 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 Processor -- start 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 -- start start 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8edaf0 0x5564cd8eded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8ee410 0x5564cd8f8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564cd8522f0 con 0x5564cd8edaf0 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5564cd7fa270 con 0x5564cd8f8cb0 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88704a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564cd84e2b0 con 0x5564cd8ee410 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8edaf0 0x5564cd8eded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88677fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8edaf0 0x5564cd8eded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45556/0 (socket says 172.21.15.28:45556) 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88677fe640 1 -- 172.21.15.28:0/3805547765 learned_addr learned my addr 172.21.15.28:0/3805547765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f88667fc640 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f8866ffd640 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8ee410 0x5564cd8f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.796+0000 7f884ffff640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88500a34e0 0x7f88500a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 -- 172.21.15.28:0/3805547765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8ee410 msgr2=0x5564cd8f8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8ee410 0x5564cd8f8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 -- 172.21.15.28:0/3805547765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 msgr2=0x5564cd8fb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 -- 172.21.15.28:0/3805547765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564cd7f9340 con 0x5564cd8edaf0 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f8866ffd640 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8ee410 0x5564cd8f8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f884ffff640 1 -- 172.21.15.28:0/3417848411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f885c0cfba0 con 0x7f8868103600 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88500a34e0 0x7f88500a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8edaf0 0x5564cd8eded0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f886002fe50 tx=0x7f886002fe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9135f1eb14658f6c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f884d7fa640 1 -- 172.21.15.28:0/3805547765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860040040 con 0x5564cd8edaf0 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f884d7fa640 1 -- 172.21.15.28:0/3805547765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8860003cb0 con 0x5564cd8edaf0 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f88677fe640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88500a34e0 0x7f88500a59a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f886004b000 tx=0x7f886004c000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.797+0000 7f884d7fa640 1 -- 172.21.15.28:0/3805547765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860034070 con 0x5564cd8edaf0 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 -- 172.21.15.28:0/3805547765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8edaf0 msgr2=0x5564cd8eded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8edaf0 0x5564cd8eded0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f886002fe50 tx=0x7f886002fe80 comp rx=0 tx=0).stop 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 -- 172.21.15.28:0/3805547765 shutdown_connections 2024-09-15T22:04:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8ee410 0x5564cd8f8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 --2- 172.21.15.28:0/3805547765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8edaf0 0x5564cd8eded0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 -- 172.21.15.28:0/3805547765 >> 172.21.15.28:0/3805547765 conn(0x5564cd8ea060 msgr2=0x5564cd8ea440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 -- 172.21.15.28:0/3805547765 shutdown_connections 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 -- 172.21.15.28:0/3805547765 wait complete. 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 -- start start 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.798+0000 7f88704a4740 1 Processor -- start 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8edaf0 0x5564cd8f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8f6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564cd8fb880 con 0x5564cd8ee410 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5564cd7db290 con 0x5564cd8f8cb0 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564cd8fba00 con 0x5564cd8edaf0 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 10 client.0 _ll_register_callbacks cb 0x5564cd80c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45566/0 (socket says 172.21.15.28:45566) 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 -- 172.21.15.28:0/290575852 learned_addr learned my addr 172.21.15.28:0/290575852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f8866ffd640 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8edaf0 0x5564cd8f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88677fe640 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8f6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 -- 172.21.15.28:0/290575852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8edaf0 msgr2=0x5564cd8f5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8edaf0 0x5564cd8f5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 -- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 msgr2=0x5564cd8f6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8f6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 -- 172.21.15.28:0/290575852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564cd565210 con 0x5564cd8ee410 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88667fc640 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f61c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f885c09b820 tx=0x7f885c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2f0bc744a1f1af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f884aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f884aff5640 1 -- 172.21.15.28:0/290575852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c020070 con 0x5564cd8ee410 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f884aff5640 1 -- 172.21.15.28:0/290575852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f885c0a4300 con 0x5564cd8ee410 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.799+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8860003cb0 con 0x5564cd8ee410 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f884aff5640 1 -- 172.21.15.28:0/290575852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f885c009de0 con 0x5564cd8ee410 2024-09-15T22:04:06.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 10 client.16518 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5564cd631620 con 0x5564cd8ee410 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f884aff5640 1 -- 172.21.15.28:0/290575852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8860003cb0 con 0x5564cd8ee410 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f884aff5640 1 -- 172.21.15.28:0/290575852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f885c09af70 con 0x5564cd8ee410 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 10 client.16518 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 10 client.16518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 20 client.16518 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd90b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 4 client.16518 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564cd85fa20 con 0x5564cd908d30 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.800+0000 7f8866ffd640 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd90b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.801+0000 7f8866ffd640 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd90b120 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45413c12e69a227b server_cookie=86eb8910873b6635 in_seq=0 out_seq=0 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.801+0000 7f884aff5640 10 client.16518 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:06.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.801+0000 7f884aff5640 1 -- 172.21.15.28:0/290575852 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5564cd85fa20 con 0x5564cd908d30 2024-09-15T22:04:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.801+0000 7f884aff5640 10 client.16518 handle_command_reply: tid=0 2024-09-15T22:04:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 client.16518 shutdown 2024-09-15T22:04:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 msgr2=0x5564cd8f61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f61c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f885c09b820 tx=0x7f885c005880 comp rx=0 tx=0).stop 2024-09-15T22:04:06.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 shutdown_connections 2024-09-15T22:04:06.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd90b120 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8f8cb0 0x5564cd8f6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f884aff5640 0 client.16518 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f61c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 --2- 172.21.15.28:0/290575852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8edaf0 0x5564cd8f5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 >> 172.21.15.28:0/290575852 conn(0x5564cd8ea060 msgr2=0x5564cd8e91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 shutdown_connections 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 1 -- 172.21.15.28:0/290575852 wait complete. 2024-09-15T22:04:06.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.802+0000 7f88704a4740 20 client.16518 trim_cache size 0 max 16384 2024-09-15T22:04:06.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 Processor -- start 2024-09-15T22:04:06.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 -- start start 2024-09-15T22:04:06.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd8f6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd9109c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88681aedf0 con 0x5564cd8ee410 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88681b0930 con 0x5564cd8edaf0 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.816+0000 7f88704a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f88681af4c0 con 0x5564cd8f8cb0 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45574/0 (socket says 172.21.15.28:45574) 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 -- 172.21.15.28:0/3257627412 learned_addr learned my addr 172.21.15.28:0/3257627412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88677fe640 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd9109c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 -- 172.21.15.28:0/3257627412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 msgr2=0x5564cd9109c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd9109c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 -- 172.21.15.28:0/3257627412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 msgr2=0x5564cd8f6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f8866ffd640 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd8f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd8f6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 -- 172.21.15.28:0/3257627412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564cd9056a0 con 0x5564cd8ee410 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f8866ffd640 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd8f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f88667fc640 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f7bf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f885c0a4260 tx=0x7f885c0d32c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc3c6af9f44f50a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f884cff9640 1 -- 172.21.15.28:0/3257627412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c020070 con 0x5564cd8ee410 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f884cff9640 1 -- 172.21.15.28:0/3257627412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f885c09ce50 con 0x5564cd8ee410 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.817+0000 7f884cff9640 1 -- 172.21.15.28:0/3257627412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c0a4560 con 0x5564cd8ee410 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- 172.21.15.28:0/3257627412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 msgr2=0x5564cd8f7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f7bf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f885c0a4260 tx=0x7f885c0d32c0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- 172.21.15.28:0/3257627412 shutdown_connections 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd9109c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd8f7bf0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 --2- 172.21.15.28:0/3257627412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd8f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- 172.21.15.28:0/3257627412 >> 172.21.15.28:0/3257627412 conn(0x5564cd8ea060 msgr2=0x5564cd8f74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- 172.21.15.28:0/3257627412 shutdown_connections 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- 172.21.15.28:0/3257627412 wait complete. 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- start start 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 Processor -- start 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd900f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd9014b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88681b0260 con 0x5564cd8ee410 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88681afb90 con 0x5564cd8edaf0 2024-09-15T22:04:06.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f88681acdd0 con 0x5564cd8f8cb0 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.818+0000 7f88704a4740 10 client.0 _ll_register_callbacks cb 0x7f885c00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd9014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd9014b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45584/0 (socket says 172.21.15.28:45584) 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 -- 172.21.15.28:0/1936069085 learned_addr learned my addr 172.21.15.28:0/1936069085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f8866ffd640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd900f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 -- 172.21.15.28:0/1936069085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 msgr2=0x5564cd8fc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88677fe640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 -- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 msgr2=0x5564cd900f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd900f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 -- 172.21.15.28:0/1936069085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564cd8fcb20 con 0x5564cd8ee410 2024-09-15T22:04:06.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f8866ffd640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd900f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88677fe640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88667fc640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd9014b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f885c097f80 tx=0x7f885c09b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4b8ed435f17c76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f884aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f884aff5640 1 -- 172.21.15.28:0/1936069085 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c020070 con 0x5564cd8ee410 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f884aff5640 1 -- 172.21.15.28:0/1936069085 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f885c097870 con 0x5564cd8ee410 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f885c09ce50 con 0x5564cd8ee410 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f884aff5640 1 -- 172.21.15.28:0/1936069085 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f885c09fed0 con 0x5564cd8ee410 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88704a4740 10 client.16524 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5564cd973490 con 0x5564cd8ee410 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.819+0000 7f884aff5640 1 -- 172.21.15.28:0/1936069085 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f885c09ce50 con 0x5564cd8ee410 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f884aff5640 1 -- 172.21.15.28:0/1936069085 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f885c09abe0 con 0x5564cd8ee410 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f88704a4740 10 client.16524 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:06.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f88704a4740 10 client.16524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:06.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f88704a4740 20 client.16524 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:06.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f88704a4740 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd9759d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f88704a4740 4 client.16524 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:04:06.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5564cd859d70 con 0x5564cd908d30 2024-09-15T22:04:06.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f8866ffd640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd9759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f8866ffd640 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd9759d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fa5813856f7a5f2 server_cookie=fee3b2b8326bc5 in_seq=0 out_seq=0 2024-09-15T22:04:06.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.820+0000 7f884aff5640 10 client.16524 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:06.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f884aff5640 1 -- 172.21.15.28:0/1936069085 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5564cd859d70 con 0x5564cd908d30 2024-09-15T22:04:06.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f884aff5640 10 client.16524 handle_command_reply: tid=0 2024-09-15T22:04:06.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 client.16524 shutdown 2024-09-15T22:04:06.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 msgr2=0x5564cd9014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd9014b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f885c097f80 tx=0x7f885c09b2e0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 shutdown_connections 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5564cd908d30 0x5564cd9759d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564cd8f8cb0 0x5564cd8fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564cd8ee410 0x5564cd9014b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 --2- 172.21.15.28:0/1936069085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564cd8edaf0 0x5564cd900f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f884aff5640 0 client.16524 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 >> 172.21.15.28:0/1936069085 conn(0x5564cd8ea060 msgr2=0x5564cd8f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 shutdown_connections 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 1 -- 172.21.15.28:0/1936069085 wait complete. 2024-09-15T22:04:06.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.831+0000 7f88704a4740 20 client.16524 trim_cache size 0 max 16384 2024-09-15T22:04:06.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:04:06.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:04:06.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:04:06.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88500a34e0 msgr2=0x7f88500a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88500a34e0 0x7f88500a59a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f886004b000 tx=0x7f886004c000 comp rx=0 tx=0).stop 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868103600 msgr2=0x7f88681987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868103600 0x7f88681987e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f885c00d950 tx=0x7f885c00de10 comp rx=0 tx=0).stop 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 shutdown_connections 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88500a34e0 0x7f88500a59a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886810c0b0 0x7f886819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868103600 0x7f88681987e0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 --2- 172.21.15.28:0/3417848411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8868102ce0 0x7f88681982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 >> 172.21.15.28:0/3417848411 conn(0x7f8868077990 msgr2=0x7f88680fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 shutdown_connections 2024-09-15T22:04:06.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:06.832+0000 7f886d713640 1 -- 172.21.15.28:0/3417848411 wait complete. 2024-09-15T22:04:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.968+0000 7f3bc626e640 1 -- 172.21.15.28:0/4063566988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0106f30 msgr2=0x7f3bc010eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.968+0000 7f3bc626e640 1 --2- 172.21.15.28:0/4063566988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0106f30 0x7f3bc010eab0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009ca0 tx=0x7f3bb802a8c0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 -- 172.21.15.28:0/4063566988 shutdown_connections 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 --2- 172.21.15.28:0/4063566988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0106f30 0x7f3bc010eab0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 --2- 172.21.15.28:0/4063566988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bc0100cc0 0x7f3bc0106800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 --2- 172.21.15.28:0/4063566988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc0100310 0x7f3bc01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 -- 172.21.15.28:0/4063566988 >> 172.21.15.28:0/4063566988 conn(0x7f3bc0077210 msgr2=0x7f3bc00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 -- 172.21.15.28:0/4063566988 shutdown_connections 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 -- 172.21.15.28:0/4063566988 wait complete. 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 Processor -- start 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.969+0000 7f3bc626e640 1 -- start start 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc626e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bc0100310 0x7f3bc01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc626e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0100cc0 0x7f3bc0198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc626e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc0106f30 0x7f3bc019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc626e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc0111f90 con 0x7f3bc0100cc0 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc626e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc0111e10 con 0x7f3bc0100310 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc626e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc0112110 con 0x7f3bc0106f30 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0100cc0 0x7f3bc0198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0100cc0 0x7f3bc0198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45610/0 (socket says 172.21.15.28:45610) 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/225443802 learned_addr learned my addr 172.21.15.28:0/225443802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc480b640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc0106f30 0x7f3bc019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbffff640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bc0100310 0x7f3bc01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/225443802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc0106f30 msgr2=0x7f3bc019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc0106f30 0x7f3bc019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/225443802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bc0100310 msgr2=0x7f3bc01983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bc0100310 0x7f3bc01983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/225443802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc019d3a0 con 0x7f3bc0100cc0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.970+0000 7f3bc480b640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc0106f30 0x7f3bc019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.971+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0100cc0 0x7f3bc0198930 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400d950 tx=0x7f3bb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.971+0000 7f3bbd7fa640 1 -- 172.21.15.28:0/225443802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4017030 con 0x7f3bc0100cc0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.971+0000 7f3bbd7fa640 1 -- 172.21.15.28:0/225443802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb4004020 con 0x7f3bc0100cc0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.971+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc019d690 con 0x7f3bc0100cc0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.971+0000 7f3bbd7fa640 1 -- 172.21.15.28:0/225443802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40047a0 con 0x7f3bc0100cc0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.971+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc01a4f70 con 0x7f3bc0100cc0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.973+0000 7f3bbd7fa640 1 -- 172.21.15.28:0/225443802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f3bb40041d0 con 0x7f3bc0100cc0 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.973+0000 7f3bc8647740 1 Processor -- start 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.973+0000 7f3bc8647740 1 -- start start 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb56d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc8647740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dceb4d32f0 con 0x55dceb56eb40 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc8647740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dceb47acf0 con 0x55dceb578370 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc8647740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dceb4cf2b0 con 0x55dceb56d5d0 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb56d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bbd7fa640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ba80a3430 0x7f3ba80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb56d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45622/0 (socket says 172.21.15.28:45622) 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 -- 172.21.15.28:0/2581360549 learned_addr learned my addr 172.21.15.28:0/2581360549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bbffff640 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bbd7fa640 1 -- 172.21.15.28:0/225443802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f3bb40cfa30 con 0x7f3bc0100cc0 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ba80a3430 0x7f3ba80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 -- 172.21.15.28:0/2581360549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 msgr2=0x55dceb56da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 -- 172.21.15.28:0/2581360549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 msgr2=0x55dceb57a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 -- 172.21.15.28:0/2581360549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dceb45f3a0 con 0x55dceb56eb40 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.974+0000 7f3bc480b640 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb56d090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8004af0 tx=0x7f3bb8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f2cae12a0063e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3ba67fc640 1 -- 172.21.15.28:0/2581360549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8031820 con 0x55dceb56eb40 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3ba67fc640 1 -- 172.21.15.28:0/2581360549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb804d070 con 0x55dceb56eb40 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3ba67fc640 1 -- 172.21.15.28:0/2581360549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8004740 con 0x55dceb56eb40 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc480b640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ba80a3430 0x7f3ba80a58f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3bb804c000 tx=0x7f3bb804a000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc8647740 1 -- 172.21.15.28:0/2581360549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 msgr2=0x55dceb56d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc8647740 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb56d090 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8004af0 tx=0x7f3bb8003970 comp rx=0 tx=0).stop 2024-09-15T22:04:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc8647740 1 -- 172.21.15.28:0/2581360549 shutdown_connections 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc8647740 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc8647740 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc8647740 1 --2- 172.21.15.28:0/2581360549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb56d090 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.975+0000 7f3bc8647740 1 -- 172.21.15.28:0/2581360549 >> 172.21.15.28:0/2581360549 conn(0x55dceb536310 msgr2=0x55dceb569c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 -- 172.21.15.28:0/2581360549 shutdown_connections 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 -- 172.21.15.28:0/2581360549 wait complete. 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 -- start start 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 Processor -- start 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb577d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb56eb40 0x55dceb575030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb578370 0x55dceb575640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dceb57b070 con 0x55dceb56d5d0 2024-09-15T22:04:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dceb48e060 con 0x55dceb56eb40 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dceb57b1f0 con 0x55dceb578370 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.976+0000 7f3bc8647740 10 client.0 _ll_register_callbacks cb 0x55dceb1e18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb577d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb577d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45634/0 (socket says 172.21.15.28:45634) 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 -- 172.21.15.28:0/4283398853 learned_addr learned my addr 172.21.15.28:0/4283398853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bc480b640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb578370 0x55dceb575640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb56eb40 0x55dceb575030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 -- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb578370 msgr2=0x55dceb575640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb578370 0x55dceb575640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 -- 172.21.15.28:0/4283398853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb56eb40 msgr2=0x55dceb575030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb56eb40 0x55dceb575030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 -- 172.21.15.28:0/4283398853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dceb2443c0 con 0x55dceb56d5d0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb56eb40 0x55dceb575030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bbffff640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb577d80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0011780 tx=0x7f3bb0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99a6098e611a6ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3b9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/4283398853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb001e060 con 0x55dceb56d5d0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/4283398853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f3bb000fdb0 con 0x55dceb56d5d0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb804d070 con 0x55dceb56d5d0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/4283398853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3bb0025050 con 0x55dceb56d5d0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bc8647740 10 client.16536 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.977+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dceb586ba0 con 0x55dceb56d5d0 2024-09-15T22:04:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/4283398853 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb804d070 con 0x55dceb56d5d0 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/4283398853 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f3bb0018030 con 0x55dceb56d5d0 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3bc8647740 10 client.16536 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3bc8647740 10 client.16536 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3bc8647740 20 client.16536 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3bc8647740 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb58a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3bc8647740 4 client.16536 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dceb4e0860 con 0x55dceb588060 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.978+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb58a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.979+0000 7f3b9f7fe640 10 client.16536 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.979+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb58a490 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e2b165f04e75020a server_cookie=b46ff6e973baea9 in_seq=0 out_seq=0 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.979+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/4283398853 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dceb4e0860 con 0x55dceb588060 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.979+0000 7f3b9f7fe640 10 client.16536 handle_command_reply: tid=0 2024-09-15T22:04:06.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.979+0000 7f3bc8647740 1 client.16536 shutdown 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.979+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 msgr2=0x55dceb577d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.979+0000 7f3bc8647740 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb577d80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0011780 tx=0x7f3bb0011c40 comp rx=0 tx=0).stop 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 shutdown_connections 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb58a490 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb578370 0x55dceb575640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3b9f7fe640 0 client.16536 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb56eb40 0x55dceb575030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 --2- 172.21.15.28:0/4283398853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb577d80 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 >> 172.21.15.28:0/4283398853 conn(0x55dceb536310 msgr2=0x55dceb56f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 shutdown_connections 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 1 -- 172.21.15.28:0/4283398853 wait complete. 2024-09-15T22:04:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.980+0000 7f3bc8647740 20 client.16536 trim_cache size 0 max 16384 2024-09-15T22:04:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 Processor -- start 2024-09-15T22:04:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 -- start start 2024-09-15T22:04:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb584310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb569700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc01aefd0 con 0x55dceb56eb40 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc01b0b10 con 0x55dceb578370 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc8647740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc01af6a0 con 0x55dceb56d5d0 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb584310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb584310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/309419756 learned_addr learned my addr 172.21.15.28:0/309419756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bbffff640 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.994+0000 7f3bc480b640 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb569700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/309419756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 msgr2=0x55dceb56e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/309419756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 msgr2=0x55dceb569700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb569700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/309419756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dceb56d2d0 con 0x55dceb56eb40 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3bc480b640 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb569700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb584310 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4097e10 tx=0x7f3bb406e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9acea8de80291627 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3ba5ffb640 1 -- 172.21.15.28:0/309419756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4066800 con 0x55dceb56eb40 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3ba5ffb640 1 -- 172.21.15.28:0/309419756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb409c0b0 con 0x55dceb56eb40 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.995+0000 7f3ba5ffb640 1 -- 172.21.15.28:0/309419756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4098020 con 0x55dceb56eb40 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 -- 172.21.15.28:0/309419756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 msgr2=0x55dceb584310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb584310 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4097e10 tx=0x7f3bb406e7e0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 -- 172.21.15.28:0/309419756 shutdown_connections 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb569700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56eb40 0x55dceb584310 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 --2- 172.21.15.28:0/309419756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56d5d0 0x55dceb56e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 -- 172.21.15.28:0/309419756 >> 172.21.15.28:0/309419756 conn(0x55dceb536310 msgr2=0x55dceb564a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 -- 172.21.15.28:0/309419756 shutdown_connections 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 -- 172.21.15.28:0/309419756 wait complete. 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.996+0000 7f3bc8647740 1 -- start start 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 1 Processor -- start 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb580b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56eb40 0x55dceb57be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc01b0440 con 0x55dceb56d5d0 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc01afd70 con 0x55dceb578370 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc01acfb0 con 0x55dceb56eb40 2024-09-15T22:04:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc8647740 10 client.0 _ll_register_callbacks cb 0x7f3bb000c690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb580b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb580b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45646/0 (socket says 172.21.15.28:45646) 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 -- 172.21.15.28:0/3384671978 learned_addr learned my addr 172.21.15.28:0/3384671978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bc480b640 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56eb40 0x55dceb57be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 -- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56eb40 msgr2=0x55dceb57be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56eb40 0x55dceb57be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 -- 172.21.15.28:0/3384671978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 msgr2=0x55dceb57c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.997+0000 7f3bbffff640 1 -- 172.21.15.28:0/3384671978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dceb570120 con 0x55dceb56d5d0 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3bbffff640 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb580b10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000a5a0 tx=0x7f3bb0005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323b73bcf169e275 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3b9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/3384671978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0011240 con 0x55dceb56d5d0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/3384671978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7f3bb000fbb0 con 0x55dceb56d5d0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bb409c0b0 con 0x55dceb56d5d0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/3384671978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3bb001e020 con 0x55dceb56d5d0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3bc8647740 10 client.16542 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dceb56a960 con 0x55dceb56d5d0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.998+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/3384671978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb409c0b0 con 0x55dceb56d5d0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/3384671978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f3bb0018040 con 0x55dceb56d5d0 2024-09-15T22:04:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3bc8647740 10 client.16542 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3bc8647740 10 client.16542 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3bc8647740 20 client.16542 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3bc8647740 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb5f5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3bc8647740 4 client.16542 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dceb4dad70 con 0x55dceb588060 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:06.999+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb5f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.000+0000 7f3b9f7fe640 10 client.16542 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.000+0000 7f3bbf7fe640 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb5f5070 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3c75e3c9cda4f7b5 server_cookie=c3ab875fd5de594d in_seq=0 out_seq=0 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.000+0000 7f3b9f7fe640 1 -- 172.21.15.28:0/3384671978 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dceb4dad70 con 0x55dceb588060 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.000+0000 7f3b9f7fe640 10 client.16542 handle_command_reply: tid=0 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.000+0000 7f3bc8647740 1 client.16542 shutdown 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.000+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 msgr2=0x55dceb580b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.000+0000 7f3bc8647740 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb580b10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000a5a0 tx=0x7f3bb0005780 comp rx=0 tx=0).stop 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 shutdown_connections 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55dceb588060 0x55dceb5f5070 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dceb578370 0x55dceb57c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dceb56eb40 0x55dceb57be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3b9f7fe640 0 client.16542 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 --2- 172.21.15.28:0/3384671978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dceb56d5d0 0x55dceb580b10 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 >> 172.21.15.28:0/3384671978 conn(0x55dceb536310 msgr2=0x55dceb56e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 shutdown_connections 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 1 -- 172.21.15.28:0/3384671978 wait complete. 2024-09-15T22:04:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.001+0000 7f3bc8647740 20 client.16542 trim_cache size 0 max 16384 2024-09-15T22:04:07.010 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ba80a3430 msgr2=0x7f3ba80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ba80a3430 0x7f3ba80a58f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3bb804c000 tx=0x7f3bb804a000 comp rx=0 tx=0).stop 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0100cc0 msgr2=0x7f3bc0198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0100cc0 0x7f3bc0198930 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400d950 tx=0x7f3bb400de10 comp rx=0 tx=0).stop 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 shutdown_connections 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ba80a3430 0x7f3ba80a58f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bc0106f30 0x7f3bc019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc0100cc0 0x7f3bc0198930 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 --2- 172.21.15.28:0/225443802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bc0100310 0x7f3bc01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 >> 172.21.15.28:0/225443802 conn(0x7f3bc0077210 msgr2=0x7f3bc01056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:07.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 shutdown_connections 2024-09-15T22:04:07.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:07.002+0000 7f3bc626e640 1 -- 172.21.15.28:0/225443802 wait complete. 2024-09-15T22:04:07.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:04:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:08.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:07 smithi028 ceph-mon[30969]: pgmap v206: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-09-15T22:04:08.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:07 smithi203 ceph-mon[35546]: pgmap v206: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-09-15T22:04:08.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:07 smithi174 ceph-mon[35244]: pgmap v206: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 53 KiB/s wr, 28 op/s 2024-09-15T22:04:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 -- 172.21.15.28:0/859610040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 msgr2=0x7fc138101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 --2- 172.21.15.28:0/859610040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc138101200 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc128009900 tx=0x7fc12802a480 comp rx=0 tx=0).stop 2024-09-15T22:04:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 -- 172.21.15.28:0/859610040 shutdown_connections 2024-09-15T22:04:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 --2- 172.21.15.28:0/859610040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc138102ef0 0x7fc13810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 --2- 172.21.15.28:0/859610040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc138102570 0x7fc1381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 --2- 172.21.15.28:0/859610040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc138101200 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 -- 172.21.15.28:0/859610040 >> 172.21.15.28:0/859610040 conn(0x7fc1380fc7d0 msgr2=0x7fc1380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 -- 172.21.15.28:0/859610040 shutdown_connections 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 -- 172.21.15.28:0/859610040 wait complete. 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 Processor -- start 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.329+0000 7fc13df4c640 1 -- start start 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc13df4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc1381984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc13df4c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc138102570 0x7fc138198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc13df4c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc138102ef0 0x7fc13819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc13df4c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1381120a0 con 0x7fc138100e20 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc13df4c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc138111f20 con 0x7fc138102ef0 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc13df4c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc138112220 con 0x7fc138102570 2024-09-15T22:04:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc1381984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc1381984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45666/0 (socket says 172.21.15.28:45666) 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 -- 172.21.15.28:0/1557086195 learned_addr learned my addr 172.21.15.28:0/1557086195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc12effd640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc138102570 0x7fc138198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc137fff640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc138102ef0 0x7fc13819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 -- 172.21.15.28:0/1557086195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc138102570 msgr2=0x7fc138198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc138102570 0x7fc138198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 -- 172.21.15.28:0/1557086195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc138102ef0 msgr2=0x7fc13819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc138102ef0 0x7fc13819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 -- 172.21.15.28:0/1557086195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13819d480 con 0x7fc138100e20 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc12effd640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc138102570 0x7fc138198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc137fff640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc138102ef0 0x7fc13819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.330+0000 7fc1377fe640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc1381984d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc128006010 tx=0x7fc128003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.331+0000 7fc1357fa640 1 -- 172.21.15.28:0/1557086195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12803f070 con 0x7fc138100e20 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.331+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13819d710 con 0x7fc138100e20 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.331+0000 7fc1357fa640 1 -- 172.21.15.28:0/1557086195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc128031c30 con 0x7fc138100e20 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.331+0000 7fc1357fa640 1 -- 172.21.15.28:0/1557086195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc128035030 con 0x7fc138100e20 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.331+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1381a5050 con 0x7fc138100e20 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.333+0000 7fc1357fa640 1 -- 172.21.15.28:0/1557086195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fc128042070 con 0x7fc138100e20 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.333+0000 7fc140325740 1 Processor -- start 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc140325740 1 -- start start 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc140325740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75727d0 0x55d6a7572bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc140325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a75819a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc140325740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a75842d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc140325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6a74d8c10 con 0x55d6a75730f0 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc140325740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d6a7480520 con 0x55d6a7581ee0 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc140325740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d6a74d4bd0 con 0x55d6a75727d0 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a75819a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc137fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75727d0 0x55d6a7572bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a75819a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45668/0 (socket says 172.21.15.28:45668) 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc137fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75727d0 0x55d6a7572bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:51032/0 (socket says 172.21.15.28:51032) 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 -- 172.21.15.28:0/2846281788 learned_addr learned my addr 172.21.15.28:0/2846281788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc12effd640 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a75842d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1357fa640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc1240a33b0 0x7fc1240a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 -- 172.21.15.28:0/2846281788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75727d0 msgr2=0x55d6a7572bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75727d0 0x55d6a7572bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 -- 172.21.15.28:0/2846281788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 msgr2=0x55d6a75842d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a75842d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc1377fe640 1 -- 172.21.15.28:0/2846281788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6a72ad490 con 0x55d6a75730f0 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc137fff640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc1240a33b0 0x7fc1240a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.334+0000 7fc12effd640 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a75842d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.335+0000 7fc1357fa640 1 -- 172.21.15.28:0/1557086195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc1280f09f0 con 0x7fc138100e20 2024-09-15T22:04:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.335+0000 7fc1377fe640 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a75819a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc1280edcf0 tx=0x7fc128087310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a0a4f114d7aa8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.335+0000 7fc137fff640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc1240a33b0 0x7fc1240a5870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc130005fd0 tx=0x7fc130004a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.335+0000 7fc12dffb640 1 -- 172.21.15.28:0/2846281788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1280bc270 con 0x55d6a75730f0 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.335+0000 7fc12dffb640 1 -- 172.21.15.28:0/2846281788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc128087820 con 0x55d6a75730f0 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.335+0000 7fc12dffb640 1 -- 172.21.15.28:0/2846281788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12809c8c0 con 0x55d6a75730f0 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- 172.21.15.28:0/2846281788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 msgr2=0x55d6a75819a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a75819a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc1280edcf0 tx=0x7fc128087310 comp rx=0 tx=0).stop 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- 172.21.15.28:0/2846281788 shutdown_connections 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a75842d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a75819a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 --2- 172.21.15.28:0/2846281788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75727d0 0x55d6a7572bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- 172.21.15.28:0/2846281788 >> 172.21.15.28:0/2846281788 conn(0x55d6a756ee80 msgr2=0x55d6a756f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- 172.21.15.28:0/2846281788 shutdown_connections 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- 172.21.15.28:0/2846281788 wait complete. 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- start start 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 Processor -- start 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75727d0 0x55d6a757ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a757f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a757f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6a7584870 con 0x55d6a75727d0 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d6a7461700 con 0x55d6a7581ee0 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.336+0000 7fc140325740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d6a75849f0 con 0x55d6a75730f0 2024-09-15T22:04:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc140325740 10 client.0 _ll_register_callbacks cb 0x55d6a7482ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75727d0 0x55d6a757ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75727d0 0x55d6a757ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45682/0 (socket says 172.21.15.28:45682) 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 -- 172.21.15.28:0/360641350 learned_addr learned my addr 172.21.15.28:0/360641350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc137fff640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a757f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc12effd640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a757f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 -- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 msgr2=0x55d6a757f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a757f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 -- 172.21.15.28:0/360641350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 msgr2=0x55d6a757f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a757f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 -- 172.21.15.28:0/360641350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6a758dce0 con 0x55d6a75727d0 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc12effd640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a757f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc137fff640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a757f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc1377fe640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75727d0 0x55d6a757ee50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc1280b92e0 tx=0x7fc1280f0c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe018f84390c075 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc116ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc116ffd640 1 -- 172.21.15.28:0/360641350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc128049070 con 0x55d6a75727d0 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc116ffd640 1 -- 172.21.15.28:0/360641350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc12808fd00 con 0x55d6a75727d0 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc128087820 con 0x55d6a75727d0 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc116ffd640 1 -- 172.21.15.28:0/360641350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc1280bc270 con 0x55d6a75727d0 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc140325740 10 client.16554 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.337+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d6a7584870 con 0x55d6a75727d0 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc116ffd640 1 -- 172.21.15.28:0/360641350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc128087820 con 0x55d6a75727d0 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc116ffd640 1 -- 172.21.15.28:0/360641350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fc1280b8f80 con 0x55d6a75727d0 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc140325740 10 client.16554 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc140325740 10 client.16554 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc140325740 20 client.16554 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc140325740 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a7591c10 0x55d6a7594090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc140325740 4 client.16554 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6a74e6340 con 0x55d6a7591c10 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.338+0000 7fc12effd640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a7591c10 0x55d6a7594090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc12effd640 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a7591c10 0x55d6a7594090 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a6ae62a2eae4a448 server_cookie=9c38bf70b2609574 in_seq=0 out_seq=0 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc116ffd640 10 client.16554 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc116ffd640 1 -- 172.21.15.28:0/360641350 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d6a74e6340 con 0x55d6a7591c10 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc116ffd640 10 client.16554 handle_command_reply: tid=0 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc140325740 1 client.16554 shutdown 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75727d0 msgr2=0x55d6a757ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc140325740 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75727d0 0x55d6a757ee50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc1280b92e0 tx=0x7fc1280f0c70 comp rx=0 tx=0).stop 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 shutdown_connections 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.339+0000 7fc140325740 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a7591c10 0x55d6a7594090 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc140325740 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a7581ee0 0x55d6a757f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc140325740 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a757f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc140325740 1 --2- 172.21.15.28:0/360641350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75727d0 0x55d6a757ee50 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 >> 172.21.15.28:0/360641350 conn(0x55d6a756ee80 msgr2=0x55d6a7574be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc116ffd640 0 client.16554 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 shutdown_connections 2024-09-15T22:04:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc140325740 1 -- 172.21.15.28:0/360641350 wait complete. 2024-09-15T22:04:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.340+0000 7fc140325740 20 client.16554 trim_cache size 0 max 16384 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 Processor -- start 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 -- start start 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a756ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a75999b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a7581ee0 0x55d6a7599ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1381acd30 con 0x55d6a7581ee0 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1381ae870 con 0x55d6a75727d0 2024-09-15T22:04:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.359+0000 7fc140325740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1381ad400 con 0x55d6a75730f0 2024-09-15T22:04:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a75999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc137fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a7581ee0 0x55d6a7599ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a75999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:51054/0 (socket says 172.21.15.28:51054) 2024-09-15T22:04:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 -- 172.21.15.28:0/4192727980 learned_addr learned my addr 172.21.15.28:0/4192727980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc1377fe640 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a756ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 -- 172.21.15.28:0/4192727980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 msgr2=0x55d6a756ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a756ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 -- 172.21.15.28:0/4192727980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a7581ee0 msgr2=0x55d6a7599ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a7581ee0 0x55d6a7599ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 -- 172.21.15.28:0/4192727980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6a757d3a0 con 0x55d6a75730f0 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc1377fe640 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a756ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc137fff640 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a7581ee0 0x55d6a7599ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12effd640 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a75999b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc120005f20 tx=0x7fc1200093d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0a85b19b74b2444 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12d7fa640 1 -- 172.21.15.28:0/4192727980 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120005220 con 0x55d6a75730f0 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12d7fa640 1 -- 172.21.15.28:0/4192727980 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc120004c80 con 0x55d6a75730f0 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.360+0000 7fc12d7fa640 1 -- 172.21.15.28:0/4192727980 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12001b8b0 con 0x55d6a75730f0 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 -- 172.21.15.28:0/4192727980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 msgr2=0x55d6a75999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a75999b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc120005f20 tx=0x7fc1200093d0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 -- 172.21.15.28:0/4192727980 shutdown_connections 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a7581ee0 0x55d6a7599ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a75730f0 0x55d6a75999b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 --2- 172.21.15.28:0/4192727980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a756ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 -- 172.21.15.28:0/4192727980 >> 172.21.15.28:0/4192727980 conn(0x55d6a756ee80 msgr2=0x55d6a756d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 -- 172.21.15.28:0/4192727980 shutdown_connections 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.361+0000 7fc140325740 1 -- 172.21.15.28:0/4192727980 wait complete. 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 -- start start 2024-09-15T22:04:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 Processor -- start 2024-09-15T22:04:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a758a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a7585730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a7581ee0 0x55d6a7585cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1381ae1a0 con 0x55d6a75730f0 2024-09-15T22:04:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1381adad0 con 0x55d6a75727d0 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1381aad10 con 0x55d6a7581ee0 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc140325740 10 client.0 _ll_register_callbacks cb 0x7fc12808ff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.362+0000 7fc12effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a7585730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a7585730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45700/0 (socket says 172.21.15.28:45700) 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 -- 172.21.15.28:0/245555465 learned_addr learned my addr 172.21.15.28:0/245555465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc1377fe640 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a758a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 -- 172.21.15.28:0/245555465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a7581ee0 msgr2=0x55d6a7585cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a7581ee0 0x55d6a7585cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 -- 172.21.15.28:0/245555465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 msgr2=0x55d6a758a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a758a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 -- 172.21.15.28:0/245555465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6a75864e0 con 0x55d6a75730f0 2024-09-15T22:04:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc12effd640 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a7585730 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc120020000 tx=0x7fc12001b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48f28f4335a5eb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc116ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc116ffd640 1 -- 172.21.15.28:0/245555465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120004930 con 0x55d6a75730f0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc116ffd640 1 -- 172.21.15.28:0/245555465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9379+0+0 (secure 0 0 0) 0x7fc120004020 con 0x55d6a75730f0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.363+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc120004c80 con 0x55d6a75730f0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc116ffd640 1 -- 172.21.15.28:0/245555465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc120004ad0 con 0x55d6a75730f0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 10 client.16560 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d6a75fca00 con 0x55d6a75730f0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc116ffd640 1 -- 172.21.15.28:0/245555465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc120004c80 con 0x55d6a75730f0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc116ffd640 1 -- 172.21.15.28:0/245555465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fc12000f040 con 0x55d6a75730f0 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 10 client.16560 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 10 client.16560 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 20 client.16560 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a75fd0c0 0x55d6a75ff4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 4 client.16560 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.c.json"}] 2024-09-15T22:04:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.c.json"}) -- 0x55d6a74e0690 con 0x55d6a75fd0c0 2024-09-15T22:04:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.364+0000 7fc1377fe640 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a75fd0c0 0x55d6a75ff4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.365+0000 7fc116ffd640 10 client.16560 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.365+0000 7fc1377fe640 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a75fd0c0 0x55d6a75ff4e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c2d6c2b637df2e server_cookie=4f0ecb6cec60ec33 in_seq=0 out_seq=0 2024-09-15T22:04:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.366+0000 7fc116ffd640 1 -- 172.21.15.28:0/245555465 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x55d6a74e0690 con 0x55d6a75fd0c0 2024-09-15T22:04:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.366+0000 7fc116ffd640 10 client.16560 handle_command_reply: tid=0 2024-09-15T22:04:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.366+0000 7fc140325740 1 client.16560 shutdown 2024-09-15T22:04:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.366+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 msgr2=0x55d6a7585730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.366+0000 7fc140325740 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a7585730 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc120020000 tx=0x7fc12001b440 comp rx=0 tx=0).stop 2024-09-15T22:04:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 shutdown_connections 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55d6a75fd0c0 0x55d6a75ff4e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d6a7581ee0 0x55d6a7585cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6a75730f0 0x55d6a7585730 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 --2- 172.21.15.28:0/245555465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6a75727d0 0x55d6a758a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 >> 172.21.15.28:0/245555465 conn(0x55d6a756ee80 msgr2=0x55d6a757f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc116ffd640 0 client.16560 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 shutdown_connections 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 1 -- 172.21.15.28:0/245555465 wait complete. 2024-09-15T22:04:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc140325740 20 client.16560 trim_cache size 0 max 16384 2024-09-15T22:04:08.374 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:08.375 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/var/run/ceph/ops-ebf3369a-1-mds.c.json" 2024-09-15T22:04:08.375 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc1240a33b0 msgr2=0x7fc1240a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc13df4c640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc1240a33b0 0x7fc1240a5870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc130005fd0 tx=0x7fc130004a40 comp rx=0 tx=0).stop 2024-09-15T22:04:08.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 msgr2=0x7fc1381984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.367+0000 7fc13df4c640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc1381984d0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc128006010 tx=0x7fc128003c10 comp rx=0 tx=0).stop 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 shutdown_connections 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc1240a33b0 0x7fc1240a5870 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc138102ef0 0x7fc13819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc138102570 0x7fc138198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 --2- 172.21.15.28:0/1557086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc138100e20 0x7fc1381984d0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 >> 172.21.15.28:0/1557086195 conn(0x7fc1380fc7d0 msgr2=0x7fc1380fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 shutdown_connections 2024-09-15T22:04:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:08.368+0000 7fc13df4c640 1 -- 172.21.15.28:0/1557086195 wait complete. 2024-09-15T22:04:08.617 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:09.126 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 1 (i) to a remote file /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.i.json 2024-09-15T22:04:09.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 ops --flags=locks --path=/var/run/ceph/ops-ebf3369a-1-mds.i.json 2024-09-15T22:04:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 -- 172.21.15.28:0/768293196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff8102de0 msgr2=0x7fbff81031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 --2- 172.21.15.28:0/768293196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff8102de0 0x7fbff81031c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009900 tx=0x7fbff402a480 comp rx=0 tx=0).stop 2024-09-15T22:04:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 -- 172.21.15.28:0/768293196 shutdown_connections 2024-09-15T22:04:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 --2- 172.21.15.28:0/768293196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff810c180 0x7fbff810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 --2- 172.21.15.28:0/768293196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbff8103700 0x7fbff810bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 --2- 172.21.15.28:0/768293196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff8102de0 0x7fbff81031c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 -- 172.21.15.28:0/768293196 >> 172.21.15.28:0/768293196 conn(0x7fbff80fc7a0 msgr2=0x7fbff80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 -- 172.21.15.28:0/768293196 shutdown_connections 2024-09-15T22:04:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.549+0000 7fbffed5b640 1 -- 172.21.15.28:0/768293196 wait complete. 2024-09-15T22:04:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 Processor -- start 2024-09-15T22:04:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 -- start start 2024-09-15T22:04:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff8102de0 0x7fbff8100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbff8103700 0x7fbff8100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff810c180 0x7fbff81014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8111fb0 con 0x7fbff810c180 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8111e30 con 0x7fbff8103700 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffed5b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbff8112130 con 0x7fbff8102de0 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff810c180 0x7fbff81014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff810c180 0x7fbff81014c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45714/0 (socket says 172.21.15.28:45714) 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 -- 172.21.15.28:0/2990631190 learned_addr learned my addr 172.21.15.28:0/2990631190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff8102de0 0x7fbff8100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbfeffff640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbff8103700 0x7fbff8100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 -- 172.21.15.28:0/2990631190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff8102de0 msgr2=0x7fbff8100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff8102de0 0x7fbff8100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 -- 172.21.15.28:0/2990631190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbff8103700 msgr2=0x7fbff8100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbff8103700 0x7fbff8100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.550+0000 7fbffd2f8640 1 -- 172.21.15.28:0/2990631190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff81a6970 con 0x7fbff810c180 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.551+0000 7fbffd2f8640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff810c180 0x7fbff81014c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fbff000b970 tx=0x7fbff000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.551+0000 7fbfedffb640 1 -- 172.21.15.28:0/2990631190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0019070 con 0x7fbff810c180 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.551+0000 7fbfedffb640 1 -- 172.21.15.28:0/2990631190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff000cb40 con 0x7fbff810c180 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.551+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff81a6c60 con 0x7fbff810c180 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.551+0000 7fbfedffb640 1 -- 172.21.15.28:0/2990631190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0007650 con 0x7fbff810c180 2024-09-15T22:04:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.551+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff81a7150 con 0x7fbff810c180 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.554+0000 7fbfedffb640 1 -- 172.21.15.28:0/2990631190 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fbff001c070 con 0x7fbff810c180 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.554+0000 7fc001134740 1 Processor -- start 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.554+0000 7fc001134740 1 -- start start 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fc001134740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bf6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fc001134740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf65d0 0x556b63bf6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fc001134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63c03760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fc001134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b63b5c2f0 con 0x556b63c01370 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fc001134740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556b63b03cf0 con 0x556b63bf7b40 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfedffb640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbfe00a33b0 0x7fbfe00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fc001134740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b63b582b0 con 0x556b63bf65d0 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63c03760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63c03760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45720/0 (socket says 172.21.15.28:45720) 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfedffb640 1 -- 172.21.15.28:0/2990631190 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7fbff00cf660 con 0x7fbff810c180 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 -- 172.21.15.28:0/3715856914 learned_addr learned my addr 172.21.15.28:0/3715856914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbffd2f8640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbfe00a33b0 0x7fbfe00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bf6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf65d0 0x556b63bf6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 -- 172.21.15.28:0/3715856914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf65d0 msgr2=0x556b63bf6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf65d0 0x556b63bf6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 -- 172.21.15.28:0/3715856914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 msgr2=0x556b63bf6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bf6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 -- 172.21.15.28:0/3715856914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b63ae83a0 con 0x556b63c01370 2024-09-15T22:04:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63c03760 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8011730 tx=0x7fbfe800a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928035b3f48d03d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.555+0000 7fbffd2f8640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbfe00a33b0 0x7fbfe00a5870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbff000b940 tx=0x7fbff0004a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fbfdeffd640 1 -- 172.21.15.28:0/3715856914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8004020 con 0x556b63c01370 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fbfdeffd640 1 -- 172.21.15.28:0/3715856914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe8005500 con 0x556b63c01370 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fbfdeffd640 1 -- 172.21.15.28:0/3715856914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8004c60 con 0x556b63c01370 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 -- 172.21.15.28:0/3715856914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 msgr2=0x556b63c03760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63c03760 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8011730 tx=0x7fbfe800a6a0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 -- 172.21.15.28:0/3715856914 shutdown_connections 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63c03760 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf65d0 0x556b63bf6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 --2- 172.21.15.28:0/3715856914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bf6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 -- 172.21.15.28:0/3715856914 >> 172.21.15.28:0/3715856914 conn(0x556b63bbf310 msgr2=0x556b63bf2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 -- 172.21.15.28:0/3715856914 shutdown_connections 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.556+0000 7fc001134740 1 -- 172.21.15.28:0/3715856914 wait complete. 2024-09-15T22:04:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 -- start start 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 Processor -- start 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c00d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63c01370 0x556b63bfe5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b63c04070 con 0x556b63bf65d0 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556b63b17060 con 0x556b63c01370 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b63c041f0 con 0x556b63bf7b40 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fc001134740 10 client.0 _ll_register_callbacks cb 0x556b6386a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c00d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c00d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45728/0 (socket says 172.21.15.28:45728) 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.557+0000 7fbfeffff640 1 -- 172.21.15.28:0/3626604451 learned_addr learned my addr 172.21.15.28:0/3626604451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63c01370 0x556b63bfe5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbffcaf7640 1 -- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63c01370 msgr2=0x556b63bfe5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63c01370 0x556b63bfe5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbffcaf7640 1 -- 172.21.15.28:0/3626604451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 msgr2=0x556b63c00d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c00d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbffcaf7640 1 -- 172.21.15.28:0/3626604451 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b638cd3c0 con 0x556b63bf7b40 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c00d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfdff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbff4005940 tx=0x7fbff4031a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77acaf1ff589c240 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbfd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3626604451 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4035040 con 0x556b63bf7b40 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3626604451 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7fbff40042c0 con 0x556b63bf7b40 2024-09-15T22:04:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.558+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe8005500 con 0x556b63bf7b40 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3626604451 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbff404f050 con 0x556b63bf7b40 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 10 client.26074 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556b63c0fae0 con 0x556b63bf7b40 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3626604451 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe8005500 con 0x556b63bf7b40 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3626604451 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fbff40467b0 con 0x556b63bf7b40 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 10 client.26074 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 10 client.26074 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 20 client.26074 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c10fa0 0x556b63c133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 4 client.26074 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b63b69860 con 0x556b63c10fa0 2024-09-15T22:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.559+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c10fa0 0x556b63c133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.560+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c10fa0 0x556b63c133b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=96491e815bc17ba2 server_cookie=426a34378b701ec in_seq=0 out_seq=0 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.560+0000 7fbfd7fff640 10 client.26074 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.560+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3626604451 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556b63b69860 con 0x556b63c10fa0 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.560+0000 7fbfd7fff640 10 client.26074 handle_command_reply: tid=0 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.560+0000 7fc001134740 1 client.26074 shutdown 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 msgr2=0x556b63bfdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfdff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbff4005940 tx=0x7fbff4031a60 comp rx=0 tx=0).stop 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 shutdown_connections 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63c01370 0x556b63bfe5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfdff0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c10fa0 0x556b63c133b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 --2- 172.21.15.28:0/3626604451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c00d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fbfd7fff640 0 client.26074 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 >> 172.21.15.28:0/3626604451 conn(0x556b63bbf310 msgr2=0x556b63bf8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 shutdown_connections 2024-09-15T22:04:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 1 -- 172.21.15.28:0/3626604451 wait complete. 2024-09-15T22:04:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.561+0000 7fc001134740 20 client.26074 trim_cache size 0 max 16384 2024-09-15T22:04:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 Processor -- start 2024-09-15T22:04:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 -- start start 2024-09-15T22:04:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf65d0 0x556b63bf2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63bf9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b1390 con 0x556b63c01370 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b2ed0 con 0x556b63bf65d0 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fc001134740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b1a60 con 0x556b63bf7b40 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63bf9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63bf9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45744/0 (socket says 172.21.15.28:45744) 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fbfeffff640 1 -- 172.21.15.28:0/1394522849 learned_addr learned my addr 172.21.15.28:0/1394522849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fbfeffff640 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf65d0 0x556b63bf2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.574+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfeffff640 1 -- 172.21.15.28:0/1394522849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 msgr2=0x556b63bfff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfeffff640 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfeffff640 1 -- 172.21.15.28:0/1394522849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf65d0 msgr2=0x556b63bf2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfeffff640 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf65d0 0x556b63bf2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfeffff640 1 -- 172.21.15.28:0/1394522849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b63bf24f0 con 0x556b63c01370 2024-09-15T22:04:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfeffff640 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63bf9120 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800a670 tx=0x7fbfe8004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e130e9bd1fd63c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfde7fc640 1 -- 172.21.15.28:0/1394522849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8011250 con 0x556b63c01370 2024-09-15T22:04:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfde7fc640 1 -- 172.21.15.28:0/1394522849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe8004970 con 0x556b63c01370 2024-09-15T22:04:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.575+0000 7fbfde7fc640 1 -- 172.21.15.28:0/1394522849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe800f9f0 con 0x556b63c01370 2024-09-15T22:04:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 -- 172.21.15.28:0/1394522849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 msgr2=0x556b63bf9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63bf9120 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800a670 tx=0x7fbfe8004840 comp rx=0 tx=0).stop 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 -- 172.21.15.28:0/1394522849 shutdown_connections 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63c01370 0x556b63bf9120 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63bf7b40 0x556b63bfff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 --2- 172.21.15.28:0/1394522849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf65d0 0x556b63bf2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 -- 172.21.15.28:0/1394522849 >> 172.21.15.28:0/1394522849 conn(0x556b63bbf310 msgr2=0x556b63bff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 -- 172.21.15.28:0/1394522849 shutdown_connections 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 -- 172.21.15.28:0/1394522849 wait complete. 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 -- start start 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.576+0000 7fc001134740 1 Processor -- start 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fc001134740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c09bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fc001134740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bfad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fc001134740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63c01370 0x556b63bfb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fc001134740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b2800 con 0x556b63bf65d0 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fc001134740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81b2130 con 0x556b63bf7b40 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fc001134740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81af390 con 0x556b63c01370 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fc001134740 10 client.0 _ll_register_callbacks cb 0x7fbff4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c09bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c09bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45746/0 (socket says 172.21.15.28:45746) 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 -- 172.21.15.28:0/3970598612 learned_addr learned my addr 172.21.15.28:0/3970598612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bfad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 -- 172.21.15.28:0/3970598612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63c01370 msgr2=0x556b63bfb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63c01370 0x556b63bfb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 -- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 msgr2=0x556b63bfad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bfad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.577+0000 7fbfeffff640 1 -- 172.21.15.28:0/3970598612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b63bfbab0 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fbfeffff640 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c09bd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8011730 tx=0x7fbfe8005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6027a0249268a2ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fbfd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3970598612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8004d20 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3970598612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7fbfe8011040 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe8004970 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3970598612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbfe8043050 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fc001134740 10 client.16578 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556b63c7bce0 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.578+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3970598612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe8004970 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3970598612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fbfe8034070 con 0x556b63bf65d0 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fc001134740 10 client.16578 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fc001134740 10 client.16578 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:04:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fc001134740 20 client.16578 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fc001134740 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c7c1e0 0x556b63c7e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fc001134740 4 client.16578 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:04:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b63b63d70 con 0x556b63c7c1e0 2024-09-15T22:04:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c7c1e0 0x556b63c7e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fbffcaf7640 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c7c1e0 0x556b63c7e600 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ac640f3288b5fa1e server_cookie=b33103e9e12ea4a3 in_seq=0 out_seq=0 2024-09-15T22:04:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.579+0000 7fbfd7fff640 10 client.16578 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fbfd7fff640 1 -- 172.21.15.28:0/3970598612 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b63b63d70 con 0x556b63c7c1e0 2024-09-15T22:04:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fbfd7fff640 10 client.16578 handle_command_reply: tid=0 2024-09-15T22:04:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 client.16578 shutdown 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 msgr2=0x556b63c09bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c09bd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8011730 tx=0x7fbfe8005570 comp rx=0 tx=0).stop 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 shutdown_connections 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b63c01370 0x556b63bfb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b63bf7b40 0x556b63bfad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x556b63c7c1e0 0x556b63c7e600 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fbfd7fff640 0 client.16578 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 --2- 172.21.15.28:0/3970598612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b63bf65d0 0x556b63c09bd0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 >> 172.21.15.28:0/3970598612 conn(0x556b63bbf310 msgr2=0x556b63c00070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 shutdown_connections 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.580+0000 7fc001134740 1 -- 172.21.15.28:0/3970598612 wait complete. 2024-09-15T22:04:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fc001134740 20 client.16578 trim_cache size 0 max 16384 2024-09-15T22:04:09.587 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:04:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbfe00a33b0 msgr2=0x7fbfe00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbfe00a33b0 0x7fbfe00a5870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbff000b940 tx=0x7fbff0004a40 comp rx=0 tx=0).stop 2024-09-15T22:04:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff810c180 msgr2=0x7fbff81014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff810c180 0x7fbff81014c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fbff000b970 tx=0x7fbff000be30 comp rx=0 tx=0).stop 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 shutdown_connections 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbfe00a33b0 0x7fbfe00a5870 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff810c180 0x7fbff81014c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbff8103700 0x7fbff8100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 --2- 172.21.15.28:0/2990631190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbff8102de0 0x7fbff8100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 >> 172.21.15.28:0/2990631190 conn(0x7fbff80fc7a0 msgr2=0x7fbff80fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 shutdown_connections 2024-09-15T22:04:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:09.581+0000 7fbffed5b640 1 -- 172.21.15.28:0/2990631190 wait complete. 2024-09-15T22:04:10.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:09 smithi203 ceph-mon[35546]: pgmap v207: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:04:10.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:09 smithi203 ceph-mon[35546]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T22:04:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:09 smithi174 ceph-mon[35244]: pgmap v207: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:04:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:09 smithi174 ceph-mon[35244]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T22:04:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:09 smithi028 ceph-mon[30969]: pgmap v207: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:04:10.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:09 smithi028 ceph-mon[30969]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T22:04:10.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:04:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.902+0000 7fb19ca34640 1 -- 172.21.15.28:0/3147632846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198106f20 msgr2=0x7fb19810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.902+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3147632846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198106f20 0x7fb19810eaa0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb190009900 tx=0x7fb19002a3f0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.903+0000 7fb19ca34640 1 -- 172.21.15.28:0/3147632846 shutdown_connections 2024-09-15T22:04:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.903+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3147632846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198106f20 0x7fb19810eaa0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.903+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3147632846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1981027a0 0x7fb1981067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.903+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3147632846 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb198101df0 0x7fb1981021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.903+0000 7fb19ca34640 1 -- 172.21.15.28:0/3147632846 >> 172.21.15.28:0/3147632846 conn(0x7fb1980fd8f0 msgr2=0x7fb1980ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.903+0000 7fb19ca34640 1 -- 172.21.15.28:0/3147632846 shutdown_connections 2024-09-15T22:04:10.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.903+0000 7fb19ca34640 1 -- 172.21.15.28:0/3147632846 wait complete. 2024-09-15T22:04:10.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 Processor -- start 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 -- start start 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101df0 0x7fb198198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1981027a0 0x7fb198198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb198106f20 0x7fb19819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb198111f80 con 0x7fb198101df0 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb198111e00 con 0x7fb198106f20 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19ca34640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb198112100 con 0x7fb1981027a0 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101df0 0x7fb198198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101df0 0x7fb198198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55450/0 (socket says 172.21.15.28:55450) 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb19659c640 1 -- 172.21.15.28:0/3048180320 learned_addr learned my addr 172.21.15.28:0/3048180320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:10.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.904+0000 7fb196d9d640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb198106f20 0x7fb19819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb195d9b640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1981027a0 0x7fb198198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19659c640 1 -- 172.21.15.28:0/3048180320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1981027a0 msgr2=0x7fb198198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19659c640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1981027a0 0x7fb198198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19659c640 1 -- 172.21.15.28:0/3048180320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb198106f20 msgr2=0x7fb19819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19659c640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb198106f20 0x7fb19819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19659c640 1 -- 172.21.15.28:0/3048180320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19819d300 con 0x7fb198101df0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19659c640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101df0 0x7fb198198350 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb18800eab0 tx=0x7fb18800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb18f7fe640 1 -- 172.21.15.28:0/3048180320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18800ce40 con 0x7fb198101df0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19819d5f0 con 0x7fb198101df0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb18f7fe640 1 -- 172.21.15.28:0/3048180320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb188011950 con 0x7fb198101df0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb18f7fe640 1 -- 172.21.15.28:0/3048180320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188010ae0 con 0x7fb198101df0 2024-09-15T22:04:10.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.905+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1981a4ed0 con 0x7fb198101df0 2024-09-15T22:04:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.908+0000 7fb18f7fe640 1 -- 172.21.15.28:0/3048180320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fb188011b00 con 0x7fb198101df0 2024-09-15T22:04:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.908+0000 7fb19ee0d740 1 Processor -- start 2024-09-15T22:04:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.908+0000 7fb19ee0d740 1 -- start start 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d5c790 0x5625e2d5cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d6b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d6bee0 0x5625e2d6e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625e2cc2c10 con 0x5625e2d5d0b0 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625e2c6a520 con 0x5625e2d6bee0 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5625e2cbebd0 con 0x5625e2d5c790 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb18f7fe640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb1840a33b0 0x7fb1840a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d6b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb196d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d5c790 0x5625e2d5cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb196d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d5c790 0x5625e2d5cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:38594/0 (socket says 172.21.15.28:38594) 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb196d9d640 1 -- 172.21.15.28:0/4180462900 learned_addr learned my addr 172.21.15.28:0/4180462900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb196d9d640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb1840a33b0 0x7fb1840a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb18f7fe640 1 -- 172.21.15.28:0/3048180320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7fb1880d0800 con 0x7fb198101df0 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19659c640 1 -- 172.21.15.28:0/4180462900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d5c790 msgr2=0x5625e2d5cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19659c640 1 --2- 172.21.15.28:0/4180462900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d5c790 0x5625e2d5cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19659c640 1 -- 172.21.15.28:0/4180462900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d6bee0 msgr2=0x5625e2d6e2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19659c640 1 --2- 172.21.15.28:0/4180462900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d6bee0 0x5625e2d6e2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.909+0000 7fb19659c640 1 -- 172.21.15.28:0/4180462900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e2a97490 con 0x5625e2d5d0b0 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.910+0000 7fb196d9d640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb1840a33b0 0x7fb1840a5870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1900096c0 tx=0x7fb190005be0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.910+0000 7fb19659c640 1 --2- 172.21.15.28:0/4180462900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d6b9a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb1880cdb40 tx=0x7fb18806c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06c9178add60c33 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.910+0000 7fb18cff9640 1 -- 172.21.15.28:0/4180462900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18809cdd0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.910+0000 7fb18cff9640 1 -- 172.21.15.28:0/4180462900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb18809c6d0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.910+0000 7fb18cff9640 1 -- 172.21.15.28:0/4180462900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1880ca490 con 0x5625e2d5d0b0 2024-09-15T22:04:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 -- 172.21.15.28:0/4180462900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 msgr2=0x5625e2d6b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/4180462900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d6b9a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb1880cdb40 tx=0x7fb18806c2f0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 -- 172.21.15.28:0/4180462900 shutdown_connections 2024-09-15T22:04:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/4180462900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d6bee0 0x5625e2d6e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/4180462900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d6b9a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/4180462900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d5c790 0x5625e2d5cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 -- 172.21.15.28:0/4180462900 >> 172.21.15.28:0/4180462900 conn(0x5625e2d58e80 msgr2=0x5625e2d59260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 -- 172.21.15.28:0/4180462900 shutdown_connections 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 -- 172.21.15.28:0/4180462900 wait complete. 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 -- start start 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.911+0000 7fb19ee0d740 1 Processor -- start 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d68e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d693c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d69900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625e2d6eab0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625e2c4b700 con 0x5625e2d5c790 2024-09-15T22:04:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5625e2d6ec30 con 0x5625e2d6bee0 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19ee0d740 10 client.0 _ll_register_callbacks cb 0x5625e2c6cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d68e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d693c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55478/0 (socket says 172.21.15.28:55478) 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 -- 172.21.15.28:0/359111722 learned_addr learned my addr 172.21.15.28:0/359111722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb196d9d640 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d69900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 -- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 msgr2=0x5625e2d69900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d69900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 -- 172.21.15.28:0/359111722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 msgr2=0x5625e2d68e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d68e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb195d9b640 1 -- 172.21.15.28:0/359111722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e2d780c0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.912+0000 7fb19659c640 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d68e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb195d9b640 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d693c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb18000e650 tx=0x7fb18000eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93bcb91a4785ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb17dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb17dffb640 1 -- 172.21.15.28:0/359111722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18000aa90 con 0x5625e2d5d0b0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb17dffb640 1 -- 172.21.15.28:0/359111722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7fb18000ac30 con 0x5625e2d5d0b0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb18809c6d0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb17dffb640 1 -- 172.21.15.28:0/359111722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb180023050 con 0x5625e2d5d0b0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb19ee0d740 10 client.16590 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5625e2d7ae00 con 0x5625e2d5d0b0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb17dffb640 1 -- 172.21.15.28:0/359111722 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb18000ae30 con 0x5625e2d5d0b0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb17dffb640 1 -- 172.21.15.28:0/359111722 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fb18001a440 con 0x5625e2d5d0b0 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb19ee0d740 10 client.16590 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb19ee0d740 10 client.16590 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.913+0000 7fb19ee0d740 20 client.16590 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.914+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2d7c310 0x5625e2d7e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.914+0000 7fb19ee0d740 4 client.16590 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.914+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb18809c6d0 con 0x5625e2d7c310 2024-09-15T22:04:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.914+0000 7fb19659c640 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2d7c310 0x5625e2d7e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.914+0000 7fb19659c640 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2d7c310 0x5625e2d7e750 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d5c996d5747845ff server_cookie=d03d544813de9ae7 in_seq=0 out_seq=0 2024-09-15T22:04:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.914+0000 7fb17dffb640 10 client.16590 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb17dffb640 1 -- 172.21.15.28:0/359111722 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fb18809c6d0 con 0x5625e2d7c310 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb17dffb640 10 client.16590 handle_command_reply: tid=0 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 client.16590 shutdown 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 msgr2=0x5625e2d693c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d693c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb18000e650 tx=0x7fb18000eb10 comp rx=0 tx=0).stop 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 shutdown_connections 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2d7c310 0x5625e2d7e750 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d69900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d693c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/359111722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d68e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 >> 172.21.15.28:0/359111722 conn(0x5625e2d58e80 msgr2=0x5625e2d5ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.915+0000 7fb17dffb640 0 client.16590 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.916+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 shutdown_connections 2024-09-15T22:04:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.916+0000 7fb19ee0d740 1 -- 172.21.15.28:0/359111722 wait complete. 2024-09-15T22:04:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.916+0000 7fb19ee0d740 20 client.16590 trim_cache size 0 max 16384 2024-09-15T22:04:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 Processor -- start 2024-09-15T22:04:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 -- start start 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d64390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d57ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d84020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981acf40 con 0x5625e2d5d0b0 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981aea80 con 0x5625e2d5c790 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.939+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981ad610 con 0x5625e2d6bee0 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d57ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d57ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55494/0 (socket says 172.21.15.28:55494) 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 -- 172.21.15.28:0/1396332266 learned_addr learned my addr 172.21.15.28:0/1396332266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb19659c640 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d64390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb196d9d640 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d84020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 -- 172.21.15.28:0/1396332266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 msgr2=0x5625e2d84020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d84020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 -- 172.21.15.28:0/1396332266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 msgr2=0x5625e2d64390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d64390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 -- 172.21.15.28:0/1396332266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e2d78a60 con 0x5625e2d5d0b0 2024-09-15T22:04:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.940+0000 7fb195d9b640 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d57ed0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb18000d010 tx=0x7fb18001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1033f90a4776aecf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.941+0000 7fb17ffff640 1 -- 172.21.15.28:0/1396332266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180020040 con 0x5625e2d5d0b0 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.941+0000 7fb17ffff640 1 -- 172.21.15.28:0/1396332266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb18001ac60 con 0x5625e2d5d0b0 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.941+0000 7fb17ffff640 1 -- 172.21.15.28:0/1396332266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180005400 con 0x5625e2d5d0b0 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 -- 172.21.15.28:0/1396332266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 msgr2=0x5625e2d57ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d57ed0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb18000d010 tx=0x7fb18001bf80 comp rx=0 tx=0).stop 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 -- 172.21.15.28:0/1396332266 shutdown_connections 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d84020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d57ed0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/1396332266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d64390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 -- 172.21.15.28:0/1396332266 >> 172.21.15.28:0/1396332266 conn(0x5625e2d58e80 msgr2=0x5625e2d577c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 -- 172.21.15.28:0/1396332266 shutdown_connections 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 -- 172.21.15.28:0/1396332266 wait complete. 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 -- start start 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.942+0000 7fb19ee0d740 1 Processor -- start 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d65de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d66320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19ee0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d66860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981ae3b0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981adce0 con 0x5625e2d5c790 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19ee0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981aaf40 con 0x5625e2d6bee0 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19ee0d740 10 client.0 _ll_register_callbacks cb 0x7fb18000c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d66320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d66320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55504/0 (socket says 172.21.15.28:55504) 2024-09-15T22:04:10.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 -- 172.21.15.28:0/496155052 learned_addr learned my addr 172.21.15.28:0/496155052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:10.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19659c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d65de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb196d9d640 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d66860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 -- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 msgr2=0x5625e2d66860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d66860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 -- 172.21.15.28:0/496155052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 msgr2=0x5625e2d65de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d65de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb195d9b640 1 -- 172.21.15.28:0/496155052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e2de5ff0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.943+0000 7fb19659c640 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d65de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb195d9b640 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d66320 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb18000e650 tx=0x7fb18000eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387c2b325e001156 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb17dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb17dffb640 1 -- 172.21.15.28:0/496155052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180037020 con 0x5625e2d5d0b0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb17dffb640 1 -- 172.21.15.28:0/496155052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7fb18000aa90 con 0x5625e2d5d0b0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5625e2cd0340 con 0x5625e2d5d0b0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb17dffb640 1 -- 172.21.15.28:0/496155052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb180042050 con 0x5625e2d5d0b0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb19ee0d740 10 client.16596 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5625e2de71f0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb17dffb640 1 -- 172.21.15.28:0/496155052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5625e2cd0340 con 0x5625e2d5d0b0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb17dffb640 1 -- 172.21.15.28:0/496155052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fb1800035f0 con 0x5625e2d5d0b0 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb19ee0d740 10 client.16596 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb19ee0d740 10 client.16596 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:04:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb19ee0d740 20 client.16596 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.944+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2de7940 0x5625e2de9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.945+0000 7fb19ee0d740 4 client.16596 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.i.json"}] 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.945+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.i.json"}) -- 0x5625e2cca690 con 0x5625e2de7940 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.945+0000 7fb19659c640 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2de7940 0x5625e2de9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.945+0000 7fb19659c640 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2de7940 0x5625e2de9d60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a015b5aae23f742e server_cookie=7904015e87dd8531 in_seq=0 out_seq=0 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.945+0000 7fb17dffb640 10 client.16596 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb17dffb640 1 -- 172.21.15.28:0/496155052 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x5625e2cca690 con 0x5625e2de7940 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb17dffb640 10 client.16596 handle_command_reply: tid=0 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 client.16596 shutdown 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 msgr2=0x5625e2d66320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d66320 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb18000e650 tx=0x7fb18000eb10 comp rx=0 tx=0).stop 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 shutdown_connections 2024-09-15T22:04:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x5625e2de7940 0x5625e2de9d60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625e2d6bee0 0x5625e2d66860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625e2d5d0b0 0x5625e2d66320 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 --2- 172.21.15.28:0/496155052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625e2d5c790 0x5625e2d65de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.946+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 >> 172.21.15.28:0/496155052 conn(0x5625e2d58e80 msgr2=0x5625e2d77560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb17dffb640 0 client.16596 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 shutdown_connections 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ee0d740 1 -- 172.21.15.28:0/496155052 wait complete. 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ee0d740 20 client.16596 trim_cache size 0 max 16384 2024-09-15T22:04:10.952 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:10.953 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/var/run/ceph/ops-ebf3369a-1-mds.i.json" 2024-09-15T22:04:10.953 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb1840a33b0 msgr2=0x7fb1840a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb1840a33b0 0x7fb1840a5870 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1900096c0 tx=0x7fb190005be0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101df0 msgr2=0x7fb198198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101df0 0x7fb198198350 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb18800eab0 tx=0x7fb18800ef70 comp rx=0 tx=0).stop 2024-09-15T22:04:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 shutdown_connections 2024-09-15T22:04:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fb1840a33b0 0x7fb1840a5870 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb198106f20 0x7fb19819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1981027a0 0x7fb198198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 --2- 172.21.15.28:0/3048180320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101df0 0x7fb198198350 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.947+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 >> 172.21.15.28:0/3048180320 conn(0x7fb1980fd8f0 msgr2=0x7fb1980febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.948+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 shutdown_connections 2024-09-15T22:04:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:10.948+0000 7fb19ca34640 1 -- 172.21.15.28:0/3048180320 wait complete. 2024-09-15T22:04:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/2324157070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54106f10 msgr2=0x7f9e5410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/2324157070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54106f10 0x7f9e5410ea90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009ca0 tx=0x7f9e4c02a8c0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/2324157070 shutdown_connections 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/2324157070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54106f10 0x7f9e5410ea90 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/2324157070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54102790 0x7f9e541067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/2324157070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e54101de0 0x7f9e541021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/2324157070 >> 172.21.15.28:0/2324157070 conn(0x7f9e540fd8c0 msgr2=0x7f9e540ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/2324157070 shutdown_connections 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.656+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/2324157070 wait complete. 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.657+0000 7f9e5b4f4640 1 Processor -- start 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.657+0000 7f9e5b4f4640 1 -- start start 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54101de0 0x7f9e541983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54102790 0x7f9e541988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e54106f10 0x7f9e5419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e54111f70 con 0x7f9e54101de0 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e54111df0 con 0x7f9e54102790 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9e541120f0 con 0x7f9e54106f10 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54101de0 0x7f9e541983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54101de0 0x7f9e541983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55518/0 (socket says 172.21.15.28:55518) 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 -- 172.21.15.28:0/361183617 learned_addr learned my addr 172.21.15.28:0/361183617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59a91640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e54106f10 0x7f9e5419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54102790 0x7f9e541988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 -- 172.21.15.28:0/361183617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e54106f10 msgr2=0x7f9e5419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e54106f10 0x7f9e5419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 -- 172.21.15.28:0/361183617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54102790 msgr2=0x7f9e541988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54102790 0x7f9e541988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 -- 172.21.15.28:0/361183617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5419d360 con 0x7f9e54101de0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54102790 0x7f9e541988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e59290640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54101de0 0x7f9e541983b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000d950 tx=0x7f9e5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e4a7fc640 1 -- 172.21.15.28:0/361183617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50017030 con 0x7f9e54101de0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e4a7fc640 1 -- 172.21.15.28:0/361183617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e50004020 con 0x7f9e54101de0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5419d650 con 0x7f9e54101de0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e4a7fc640 1 -- 172.21.15.28:0/361183617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e500047a0 con 0x7f9e54101de0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.658+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e541a4f30 con 0x7f9e54101de0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e4a7fc640 1 -- 172.21.15.28:0/361183617 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f9e500041d0 con 0x7f9e54101de0 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 Processor -- start 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 -- start start 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb25b40 0x556d1eb24090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb245d0 0x556d1eb24a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb31760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d1ea8a2f0 con 0x556d1eb2f370 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d1ea31cf0 con 0x556d1eb245d0 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.661+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556d1ea862b0 con 0x556d1eb25b40 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e4a7fc640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9e400a33b0 0x7f9e400a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb31760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb31760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55526/0 (socket says 172.21.15.28:55526) 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 -- 172.21.15.28:0/853914709 learned_addr learned my addr 172.21.15.28:0/853914709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e59a91640 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb25b40 0x556d1eb24090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e59a91640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9e400a33b0 0x7f9e400a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e59290640 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb245d0 0x556d1eb24a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e4a7fc640 1 -- 172.21.15.28:0/361183617 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f9e500cfbd0 con 0x7f9e54101de0 2024-09-15T22:04:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 -- 172.21.15.28:0/853914709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb25b40 msgr2=0x556d1eb24090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb25b40 0x556d1eb24090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 -- 172.21.15.28:0/853914709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb245d0 msgr2=0x556d1eb24a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb245d0 0x556d1eb24a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 -- 172.21.15.28:0/853914709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d1ea163a0 con 0x556d1eb2f370 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb31760 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009ba0 tx=0x7f9e4400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c046b1474129e93d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e59a91640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9e400a33b0 0x7f9e400a5870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c005e30 tx=0x7f9e4c0095c0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e3f7fe640 1 -- 172.21.15.28:0/853914709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44016070 con 0x556d1eb2f370 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.662+0000 7f9e3f7fe640 1 -- 172.21.15.28:0/853914709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e44011950 con 0x556d1eb2f370 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e3f7fe640 1 -- 172.21.15.28:0/853914709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44010820 con 0x556d1eb2f370 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/853914709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 msgr2=0x556d1eb31760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb31760 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9e44009ba0 tx=0x7f9e4400e6c0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/853914709 shutdown_connections 2024-09-15T22:04:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb31760 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb245d0 0x556d1eb24a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/853914709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb25b40 0x556d1eb24090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/853914709 >> 172.21.15.28:0/853914709 conn(0x556d1eaed310 msgr2=0x556d1eb20c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/853914709 shutdown_connections 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.663+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/853914709 wait complete. 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 -- start start 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 Processor -- start 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb2ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb2c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d1eb32070 con 0x556d1eb25b40 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d1ea45060 con 0x556d1eb2f370 2024-09-15T22:04:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556d1eb321f0 con 0x556d1eb245d0 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e5d8cd740 10 client.0 _ll_register_callbacks cb 0x556d1e7988c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e58a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e58a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55528/0 (socket says 172.21.15.28:55528) 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e58a8f640 1 -- 172.21.15.28:0/1142001282 learned_addr learned my addr 172.21.15.28:0/1142001282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e59a91640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb2c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e59290640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb2ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.664+0000 7f9e58a8f640 1 -- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 msgr2=0x556d1eb2ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb2ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e58a8f640 1 -- 172.21.15.28:0/1142001282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 msgr2=0x556d1eb2c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb2c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e58a8f640 1 -- 172.21.15.28:0/1142001282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d1e7fb3c0 con 0x556d1eb25b40 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e59a91640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb2c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2bf90 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9e44010520 tx=0x7f9e44011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc16fe17c2aedb20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e3cff9640 1 -- 172.21.15.28:0/1142001282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44016050 con 0x556d1eb25b40 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e44011950 con 0x556d1eb25b40 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e3cff9640 1 -- 172.21.15.28:0/1142001282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f9e44005040 con 0x556d1eb25b40 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e3cff9640 1 -- 172.21.15.28:0/1142001282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e44030050 con 0x556d1eb25b40 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e5d8cd740 10 client.16608 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d1eb3da80 con 0x556d1eb25b40 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e3cff9640 1 -- 172.21.15.28:0/1142001282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e44011950 con 0x556d1eb25b40 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.665+0000 7f9e3cff9640 1 -- 172.21.15.28:0/1142001282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9e4402b030 con 0x556d1eb25b40 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e5d8cd740 10 client.16608 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e5d8cd740 10 client.16608 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:04:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e5d8cd740 20 client.16608 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1eb3ee30 0x556d1eb41220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e5d8cd740 4 client.16608 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d1ea97860 con 0x556d1eb3ee30 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e59290640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1eb3ee30 0x556d1eb41220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e3cff9640 10 client.16608 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.666+0000 7f9e59290640 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1eb3ee30 0x556d1eb41220 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57279d3676e7b540 server_cookie=7c4e8f8fa22df761 in_seq=0 out_seq=0 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e3cff9640 1 -- 172.21.15.28:0/1142001282 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d1ea97860 con 0x556d1eb3ee30 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e3cff9640 10 client.16608 handle_command_reply: tid=0 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 client.16608 shutdown 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 msgr2=0x556d1eb2bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2bf90 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9e44010520 tx=0x7f9e44011680 comp rx=0 tx=0).stop 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 shutdown_connections 2024-09-15T22:04:11.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1eb3ee30 0x556d1eb41220 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb2c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2bf90 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e3cff9640 0 client.16608 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/1142001282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb2ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 >> 172.21.15.28:0/1142001282 conn(0x556d1eaed310 msgr2=0x556d1eb26960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 shutdown_connections 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.667+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/1142001282 wait complete. 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.668+0000 7f9e5d8cd740 20 client.16608 trim_cache size 0 max 16384 2024-09-15T22:04:11.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 Processor -- start 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 -- start start 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb3b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb468d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e541af1e0 con 0x556d1eb25b40 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e541b0d20 con 0x556d1eb2f370 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9e541af8b0 con 0x556d1eb245d0 2024-09-15T22:04:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55536/0 (socket says 172.21.15.28:55536) 2024-09-15T22:04:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 -- 172.21.15.28:0/737690153 learned_addr learned my addr 172.21.15.28:0/737690153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e59a91640 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 -- 172.21.15.28:0/737690153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 msgr2=0x556d1eb3b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb3b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 -- 172.21.15.28:0/737690153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 msgr2=0x556d1eb468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 -- 172.21.15.28:0/737690153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d1eb2a4e0 con 0x556d1eb25b40 2024-09-15T22:04:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.682+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2af70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9e44000c00 tx=0x7f9e4400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baca448eff8fcb20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e3effd640 1 -- 172.21.15.28:0/737690153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44038070 con 0x556d1eb25b40 2024-09-15T22:04:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e3effd640 1 -- 172.21.15.28:0/737690153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e440116f0 con 0x556d1eb25b40 2024-09-15T22:04:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e3effd640 1 -- 172.21.15.28:0/737690153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44016050 con 0x556d1eb25b40 2024-09-15T22:04:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/737690153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 msgr2=0x556d1eb2af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2af70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9e44000c00 tx=0x7f9e4400eef0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/737690153 shutdown_connections 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb2f370 0x556d1eb468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb25b40 0x556d1eb2af70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/737690153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb3b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/737690153 >> 172.21.15.28:0/737690153 conn(0x556d1eaed310 msgr2=0x556d1eb2a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/737690153 shutdown_connections 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.683+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/737690153 wait complete. 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 -- start start 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 Processor -- start 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb37b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb25b40 0x556d1eb28cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb29210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e541b0650 con 0x556d1eb2f370 2024-09-15T22:04:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e541aff80 con 0x556d1eb25b40 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9e541ad1e0 con 0x556d1eb245d0 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e5d8cd740 10 client.0 _ll_register_callbacks cb 0x7f9e500046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e59a91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb29210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e59a91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb29210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55550/0 (socket says 172.21.15.28:55550) 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.684+0000 7f9e59a91640 1 -- 172.21.15.28:0/4208298611 learned_addr learned my addr 172.21.15.28:0/4208298611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e59290640 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb37b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e58a8f640 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb25b40 0x556d1eb28cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e59a91640 1 -- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 msgr2=0x556d1eb37b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e59a91640 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb37b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e59a91640 1 -- 172.21.15.28:0/4208298611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb25b40 msgr2=0x556d1eb28cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e59a91640 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb25b40 0x556d1eb28cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e59a91640 1 -- 172.21.15.28:0/4208298611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d1eb29a40 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e59a91640 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb29210 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009ca0 tx=0x7f9e4c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc15bba0bd8a91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e3cff9640 1 -- 172.21.15.28:0/4208298611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c059060 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e440116f0 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e3cff9640 1 -- 172.21.15.28:0/4208298611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f9e4c031830 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e3cff9640 1 -- 172.21.15.28:0/4208298611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e4c004590 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e5d8cd740 10 client.16614 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.685+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d1eba98e0 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e3cff9640 1 -- 172.21.15.28:0/4208298611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e440116f0 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e3cff9640 1 -- 172.21.15.28:0/4208298611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f9e4c053070 con 0x556d1eb2f370 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e5d8cd740 10 client.16614 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e5d8cd740 10 client.16614 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:04:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e5d8cd740 20 client.16614 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1ebaa040 0x556d1ebac460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e5d8cd740 4 client.16614 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d1ea91d70 con 0x556d1ebaa040 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e59290640 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1ebaa040 0x556d1ebac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e3cff9640 10 client.16614 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.686+0000 7f9e59290640 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1ebaa040 0x556d1ebac460 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=833d46574ab72c4f server_cookie=56b4ed07a598fbda in_seq=0 out_seq=0 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e3cff9640 1 -- 172.21.15.28:0/4208298611 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d1ea91d70 con 0x556d1ebaa040 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e3cff9640 10 client.16614 handle_command_reply: tid=0 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 client.16614 shutdown 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 msgr2=0x556d1eb29210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb29210 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009ca0 tx=0x7f9e4c005d80 comp rx=0 tx=0).stop 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 shutdown_connections 2024-09-15T22:04:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556d1ebaa040 0x556d1ebac460 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d1eb2f370 0x556d1eb29210 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d1eb25b40 0x556d1eb28cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e3cff9640 0 client.16614 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 --2- 172.21.15.28:0/4208298611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556d1eb245d0 0x556d1eb37b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 >> 172.21.15.28:0/4208298611 conn(0x556d1eaed310 msgr2=0x556d1eb2d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 shutdown_connections 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 1 -- 172.21.15.28:0/4208298611 wait complete. 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.687+0000 7f9e5d8cd740 20 client.16614 trim_cache size 0 max 16384 2024-09-15T22:04:11.705 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:04:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9e400a33b0 msgr2=0x7f9e400a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9e400a33b0 0x7f9e400a5870 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c005e30 tx=0x7f9e4c0095c0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54101de0 msgr2=0x7f9e541983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54101de0 0x7f9e541983b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000d950 tx=0x7f9e5000de10 comp rx=0 tx=0).stop 2024-09-15T22:04:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 shutdown_connections 2024-09-15T22:04:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9e400a33b0 0x7f9e400a5870 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e54106f10 0x7f9e5419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e54102790 0x7f9e541988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 --2- 172.21.15.28:0/361183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e54101de0 0x7f9e541983b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 >> 172.21.15.28:0/361183617 conn(0x7f9e540fd8c0 msgr2=0x7f9e540fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 shutdown_connections 2024-09-15T22:04:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:11.688+0000 7f9e5b4f4640 1 -- 172.21.15.28:0/361183617 wait complete. 2024-09-15T22:04:12.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:11 smithi203 ceph-mon[35546]: pgmap v209: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 19 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-09-15T22:04:12.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:11 smithi174 ceph-mon[35244]: pgmap v209: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 19 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-09-15T22:04:12.346 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:11 smithi028 ceph-mon[30969]: pgmap v209: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 19 KiB/s rd, 56 KiB/s wr, 31 op/s 2024-09-15T22:04:12.375 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 0 (b) to a remote file /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.b.json 2024-09-15T22:04:12.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 ops --flags=locks --path=/var/run/ceph/ops-ebf3369a-1-mds.b.json 2024-09-15T22:04:12.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:04:12.837 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.374+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/1714567730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 msgr2=0x7f4cf0106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.374+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/1714567730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0106860 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009930 tx=0x7f4ce402a420 comp rx=0 tx=0).stop 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.374+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/1714567730 shutdown_connections 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.374+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/1714567730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf0106fc0 0x7f4cf010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.375+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/1714567730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0106860 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.375+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/1714567730 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cf0100370 0x7f4cf0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.375+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/1714567730 >> 172.21.15.28:0/1714567730 conn(0x7f4cf0077210 msgr2=0x7f4cf00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.375+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/1714567730 shutdown_connections 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.375+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/1714567730 wait complete. 2024-09-15T22:04:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.375+0000 7f4cf5ec2640 1 Processor -- start 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4cf5ec2640 1 -- start start 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4cf5ec2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf0100370 0x7f4cf0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4cf5ec2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4cf5ec2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cf0106fc0 0x7f4cf019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4cf5ec2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0111ff0 con 0x7f4cf0100d20 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4cf5ec2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0111e70 con 0x7f4cf0100370 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4cf5ec2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0112170 con 0x7f4cf0106fc0 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55558/0 (socket says 172.21.15.28:55558) 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4ceeffd640 1 -- 172.21.15.28:0/16566646 learned_addr learned my addr 172.21.15.28:0/16566646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4ceffff640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cf0106fc0 0x7f4cf019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4ceeffd640 1 -- 172.21.15.28:0/16566646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cf0106fc0 msgr2=0x7f4cf019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cf0106fc0 0x7f4cf019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.376+0000 7f4ceeffd640 1 -- 172.21.15.28:0/16566646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf0100370 msgr2=0x7f4cf0198430 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf0100370 0x7f4cf0198430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4ceeffd640 1 -- 172.21.15.28:0/16566646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf019d3e0 con 0x7f4cf0100d20 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4ceffff640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cf0106fc0 0x7f4cf019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0198970 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4ce402a940 tx=0x7f4ce4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4cecff9640 1 -- 172.21.15.28:0/16566646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce403f070 con 0x7f4cf0100d20 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf019d670 con 0x7f4cf0100d20 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4cecff9640 1 -- 172.21.15.28:0/16566646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce4031c90 con 0x7f4cf0100d20 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4cecff9640 1 -- 172.21.15.28:0/16566646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4035030 con 0x7f4cf0100d20 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.377+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf01a4fb0 con 0x7f4cf0100d20 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 Processor -- start 2024-09-15T22:04:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 -- start start 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cecff9640 1 -- 172.21.15.28:0/16566646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f4ce4042070 con 0x7f4cf0100d20 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617d9ee0 0x5634617dc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563461730c10 con 0x5634617cb0f0 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5634616d8520 con 0x5634617ca7d0 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cf829b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56346172cbd0 con 0x5634617d9ee0 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55568/0 (socket says 172.21.15.28:55568) 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 -- 172.21.15.28:0/639774154 learned_addr learned my addr 172.21.15.28:0/639774154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 -- 172.21.15.28:0/639774154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617d9ee0 msgr2=0x5634617dc2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:04:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/639774154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617d9ee0 0x5634617dc2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 -- 172.21.15.28:0/639774154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 msgr2=0x5634617cabb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/639774154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617cabb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.380+0000 7f4cef7fe640 1 -- 172.21.15.28:0/639774154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563461505490 con 0x5634617cb0f0 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.381+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/639774154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d99a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009bd0 tx=0x7f4ce000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6697b0acc74308d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.381+0000 7f4cd5ffb640 1 -- 172.21.15.28:0/639774154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0005020 con 0x5634617cb0f0 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.381+0000 7f4cd5ffb640 1 -- 172.21.15.28:0/639774154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce00047a0 con 0x5634617cb0f0 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.381+0000 7f4cd5ffb640 1 -- 172.21.15.28:0/639774154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0010690 con 0x5634617cb0f0 2024-09-15T22:04:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.381+0000 7f4cecff9640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4cd80a33b0 0x7f4cd80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.381+0000 7f4ceffff640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4cd80a33b0 0x7f4cd80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.381+0000 7f4cecff9640 1 -- 172.21.15.28:0/16566646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f4ce40f0940 con 0x7f4cf0100d20 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 -- 172.21.15.28:0/639774154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 msgr2=0x5634617d99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 --2- 172.21.15.28:0/639774154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d99a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009bd0 tx=0x7f4ce000e720 comp rx=0 tx=0).stop 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4ceffff640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4cd80a33b0 0x7f4cd80a5870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005fd0 tx=0x7f4ce80094a0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 -- 172.21.15.28:0/639774154 shutdown_connections 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 --2- 172.21.15.28:0/639774154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617d9ee0 0x5634617dc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 --2- 172.21.15.28:0/639774154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d99a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 --2- 172.21.15.28:0/639774154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 -- 172.21.15.28:0/639774154 >> 172.21.15.28:0/639774154 conn(0x5634617c6e80 msgr2=0x5634617c7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 -- 172.21.15.28:0/639774154 shutdown_connections 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 -- 172.21.15.28:0/639774154 wait complete. 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.382+0000 7f4cf829b740 1 -- start start 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 1 Processor -- start 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617ca7d0 0x5634617d6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617d9ee0 0x5634617d7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634617dcab0 con 0x5634617cb0f0 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5634616b9700 con 0x5634617d9ee0 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5634617dcc30 con 0x5634617ca7d0 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cf829b740 10 client.0 _ll_register_callbacks cb 0x5634616daae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55570/0 (socket says 172.21.15.28:55570) 2024-09-15T22:04:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4ceeffd640 1 -- 172.21.15.28:0/2644825562 learned_addr learned my addr 172.21.15.28:0/2644825562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617ca7d0 0x5634617d6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.383+0000 7f4ceffff640 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617d9ee0 0x5634617d7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4ceeffd640 1 -- 172.21.15.28:0/2644825562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617ca7d0 msgr2=0x5634617d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617ca7d0 0x5634617d6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4ceeffd640 1 -- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617d9ee0 msgr2=0x5634617d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617d9ee0 0x5634617d7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4ceeffd640 1 -- 172.21.15.28:0/2644825562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634617e6100 con 0x5634617cb0f0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d7430 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009900 tx=0x7f4ce4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fb09569059b7c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cceffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cceffd640 1 -- 172.21.15.28:0/2644825562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4049050 con 0x5634617cb0f0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cceffd640 1 -- 172.21.15.28:0/2644825562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f4ce40b8e80 con 0x5634617cb0f0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ce00047a0 con 0x5634617cb0f0 2024-09-15T22:04:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cceffd640 1 -- 172.21.15.28:0/2644825562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4ce40bc170 con 0x5634617cb0f0 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cf829b740 10 client.16626 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5634617e8e20 con 0x5634617cb0f0 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.384+0000 7f4cceffd640 1 -- 172.21.15.28:0/2644825562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce00047a0 con 0x5634617cb0f0 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cceffd640 1 -- 172.21.15.28:0/2644825562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f4ce40c4ff0 con 0x5634617cb0f0 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cf829b740 10 client.16626 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cf829b740 10 client.16626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cf829b740 20 client.16626 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cf829b740 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634617ea1c0 0x5634617ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cf829b740 4 client.16626 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56346173e340 con 0x5634617ea1c0 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.385+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634617ea1c0 0x5634617ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.386+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634617ea1c0 0x5634617ec5b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9be8543e7446c108 server_cookie=d7ff61d71ee6bb32 in_seq=0 out_seq=0 2024-09-15T22:04:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.386+0000 7f4cceffd640 10 client.16626 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cceffd640 1 -- 172.21.15.28:0/2644825562 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56346173e340 con 0x5634617ea1c0 2024-09-15T22:04:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cceffd640 10 client.16626 handle_command_reply: tid=0 2024-09-15T22:04:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 client.16626 shutdown 2024-09-15T22:04:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 msgr2=0x5634617d7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d7430 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009900 tx=0x7f4ce4005dc0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 shutdown_connections 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5634617ea1c0 0x5634617ec5b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617d9ee0 0x5634617d7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617cb0f0 0x5634617d7430 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cceffd640 0 client.16626 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 --2- 172.21.15.28:0/2644825562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617ca7d0 0x5634617d6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 >> 172.21.15.28:0/2644825562 conn(0x5634617c6e80 msgr2=0x5634617ccbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 shutdown_connections 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 1 -- 172.21.15.28:0/2644825562 wait complete. 2024-09-15T22:04:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.387+0000 7f4cf829b740 20 client.16626 trim_cache size 0 max 16384 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 Processor -- start 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 -- start start 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617d77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x5634617f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01ace60 con 0x5634617d9ee0 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01ae9a0 con 0x5634617ca7d0 2024-09-15T22:04:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cf829b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01ad530 con 0x5634617cb0f0 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x5634617f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x5634617f1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55586/0 (socket says 172.21.15.28:55586) 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4ceffff640 1 -- 172.21.15.28:0/1889414644 learned_addr learned my addr 172.21.15.28:0/1889414644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.404+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617d77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4ceffff640 1 -- 172.21.15.28:0/1889414644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 msgr2=0x5634617c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4ceffff640 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617c5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4ceffff640 1 -- 172.21.15.28:0/1889414644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 msgr2=0x5634617d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4ceffff640 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617d77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4ceffff640 1 -- 172.21.15.28:0/1889414644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634617ec920 con 0x5634617d9ee0 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617d77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4ceffff640 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x5634617f1e40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80124a0 tx=0x7f4ce8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0afc2b1039a22 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4cd57fa640 1 -- 172.21.15.28:0/1889414644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce801f620 con 0x5634617d9ee0 2024-09-15T22:04:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4cd57fa640 1 -- 172.21.15.28:0/1889414644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce801fca0 con 0x5634617d9ee0 2024-09-15T22:04:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.405+0000 7f4cd57fa640 1 -- 172.21.15.28:0/1889414644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8011e30 con 0x5634617d9ee0 2024-09-15T22:04:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 -- 172.21.15.28:0/1889414644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 msgr2=0x5634617f1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x5634617f1e40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80124a0 tx=0x7f4ce8013000 comp rx=0 tx=0).stop 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 -- 172.21.15.28:0/1889414644 shutdown_connections 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x5634617f1e40 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 --2- 172.21.15.28:0/1889414644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617d77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 -- 172.21.15.28:0/1889414644 >> 172.21.15.28:0/1889414644 conn(0x5634617c6e80 msgr2=0x5634617c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 -- 172.21.15.28:0/1889414644 shutdown_connections 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.406+0000 7f4cf829b740 1 -- 172.21.15.28:0/1889414644 wait complete. 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 -- start start 2024-09-15T22:04:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 Processor -- start 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617de520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617dea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x56346184de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01ae2d0 con 0x5634617d9ee0 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01adc00 con 0x5634617ca7d0 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01aae60 con 0x5634617cb0f0 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x56346184de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4cf829b740 10 client.0 _ll_register_callbacks cb 0x7f4ce4009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x56346184de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55588/0 (socket says 172.21.15.28:55588) 2024-09-15T22:04:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4ceffff640 1 -- 172.21.15.28:0/608985094 learned_addr learned my addr 172.21.15.28:0/608985094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617de520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.407+0000 7f4ceeffd640 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617dea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4ceffff640 1 -- 172.21.15.28:0/608985094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 msgr2=0x5634617dea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4ceffff640 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617dea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4ceffff640 1 -- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 msgr2=0x5634617de520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4ceffff640 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617de520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4ceffff640 1 -- 172.21.15.28:0/608985094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634617d1cb0 con 0x5634617d9ee0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4ceffff640 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x56346184de50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005c40 tx=0x7f4ce8011610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82de29a7838b12b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4cceffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4cceffd640 1 -- 172.21.15.28:0/608985094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8011cc0 con 0x5634617d9ee0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4cceffd640 1 -- 172.21.15.28:0/608985094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f4ce8004a40 con 0x5634617d9ee0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.408+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ce801fca0 con 0x5634617d9ee0 2024-09-15T22:04:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cceffd640 1 -- 172.21.15.28:0/608985094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4ce803b050 con 0x5634617d9ee0 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 10 client.16632 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563461855600 con 0x5634617d9ee0 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cceffd640 1 -- 172.21.15.28:0/608985094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce801fca0 con 0x5634617d9ee0 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cceffd640 1 -- 172.21.15.28:0/608985094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f4ce800f040 con 0x5634617d9ee0 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 10 client.16632 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 10 client.16632 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 20 client.16632 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563461855c80 0x5634618580a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 4 client.16632 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.b.json"}] 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.409+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-ebf3369a-1-mds.b.json"}) -- 0x563461738690 con 0x563461855c80 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.410+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563461855c80 0x5634618580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:13.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.410+0000 7f4cef7fe640 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563461855c80 0x5634618580a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=905afde6ca3cf964 server_cookie=84b33fa915663583 in_seq=0 out_seq=0 2024-09-15T22:04:13.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.410+0000 7f4cceffd640 10 client.16632 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.440+0000 7f4cceffd640 1 -- 172.21.15.28:0/608985094 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x563461738690 con 0x563461855c80 2024-09-15T22:04:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.440+0000 7f4cceffd640 10 client.16632 handle_command_reply: tid=0 2024-09-15T22:04:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.440+0000 7f4cf829b740 1 client.16632 shutdown 2024-09-15T22:04:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 msgr2=0x56346184de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x56346184de50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005c40 tx=0x7f4ce8011610 comp rx=0 tx=0).stop 2024-09-15T22:04:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 shutdown_connections 2024-09-15T22:04:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563461855c80 0x5634618580a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634617d9ee0 0x56346184de50 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cceffd640 0 client.16632 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5634617cb0f0 0x5634617dea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 --2- 172.21.15.28:0/608985094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5634617ca7d0 0x5634617de520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 >> 172.21.15.28:0/608985094 conn(0x5634617c6e80 msgr2=0x5634617ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 shutdown_connections 2024-09-15T22:04:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 1 -- 172.21.15.28:0/608985094 wait complete. 2024-09-15T22:04:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.441+0000 7f4cf829b740 20 client.16632 trim_cache size 0 max 16384 2024-09-15T22:04:13.445 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:13.445 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/var/run/ceph/ops-ebf3369a-1-mds.b.json" 2024-09-15T22:04:13.445 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4cd80a33b0 msgr2=0x7f4cd80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4cd80a33b0 0x7f4cd80a5870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005fd0 tx=0x7f4ce80094a0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 msgr2=0x7f4cf0198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0198970 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4ce402a940 tx=0x7f4ce4003b90 comp rx=0 tx=0).stop 2024-09-15T22:04:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 shutdown_connections 2024-09-15T22:04:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4cd80a33b0 0x7f4cd80a5870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cf0106fc0 0x7f4cf019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf0100d20 0x7f4cf0198970 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 --2- 172.21.15.28:0/16566646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf0100370 0x7f4cf0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:13.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 >> 172.21.15.28:0/16566646 conn(0x7f4cf0077210 msgr2=0x7f4cf0105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:13.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.442+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 shutdown_connections 2024-09-15T22:04:13.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:13.443+0000 7f4cf5ec2640 1 -- 172.21.15.28:0/16566646 wait complete. 2024-09-15T22:04:13.447 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:14.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:13 smithi028 ceph-mon[30969]: pgmap v210: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-09-15T22:04:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:13 smithi203 ceph-mon[35546]: pgmap v210: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-09-15T22:04:14.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:13 smithi174 ceph-mon[35244]: pgmap v210: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-09-15T22:04:14.484 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-15T22:04:14.522 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.KkVqFptazH 2024-09-15T22:04:14.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.f.json /tmp/tmp.KkVqFptazH 2024-09-15T22:04:14.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.KkVqFptazH 2024-09-15T22:04:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 -- 172.21.15.28:0/2416860684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 msgr2=0x7f3cf410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 --2- 172.21.15.28:0/2416860684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf410ea90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009ca0 tx=0x7f3cec02a8c0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 -- 172.21.15.28:0/2416860684 shutdown_connections 2024-09-15T22:04:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 --2- 172.21.15.28:0/2416860684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf410ea90 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 --2- 172.21.15.28:0/2416860684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cf4102790 0x7f3cf41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 --2- 172.21.15.28:0/2416860684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cf4101de0 0x7f3cf41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 -- 172.21.15.28:0/2416860684 >> 172.21.15.28:0/2416860684 conn(0x7f3cf40fd8c0 msgr2=0x7f3cf40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 -- 172.21.15.28:0/2416860684 shutdown_connections 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.717+0000 7f3cf970d640 1 -- 172.21.15.28:0/2416860684 wait complete. 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 Processor -- start 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 -- start start 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cf4101de0 0x7f3cf40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cf4102790 0x7f3cf4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4111fc0 con 0x7f3cf4106f10 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4111e40 con 0x7f3cf4101de0 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf970d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4112140 con 0x7f3cf4102790 2024-09-15T22:04:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf4074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55618/0 (socket says 172.21.15.28:55618) 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cf37fe640 1 -- 172.21.15.28:0/1122424946 learned_addr learned my addr 172.21.15.28:0/1122424946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.718+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cf4102790 0x7f3cf4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cf4101de0 0x7f3cf40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf37fe640 1 -- 172.21.15.28:0/1122424946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cf4102790 msgr2=0x7f3cf4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf37fe640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cf4102790 0x7f3cf4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf37fe640 1 -- 172.21.15.28:0/1122424946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cf4101de0 msgr2=0x7f3cf40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf37fe640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cf4101de0 0x7f3cf40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf37fe640 1 -- 172.21.15.28:0/1122424946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf41a68f0 con 0x7f3cf4106f10 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cf4101de0 0x7f3cf40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf37fe640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf4074f70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3cec003f10 tx=0x7f3cec003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf0ff9640 1 -- 172.21.15.28:0/1122424946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec031710 con 0x7f3cf4106f10 2024-09-15T22:04:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf0ff9640 1 -- 172.21.15.28:0/1122424946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cec0318b0 con 0x7f3cf4106f10 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf41a6b80 con 0x7f3cf4106f10 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf0ff9640 1 -- 172.21.15.28:0/1122424946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec02fb90 con 0x7f3cf4106f10 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.719+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf41a70c0 con 0x7f3cf4106f10 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 Processor -- start 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cf0ff9640 1 -- 172.21.15.28:0/1122424946 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f3cec02fd30 con 0x7f3cf4106f10 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 -- start start 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9d8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd93e2f0 con 0x55d0fd9d9b40 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd8e5cf0 con 0x55d0fd9d85d0 2024-09-15T22:04:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd93a2b0 con 0x55d0fd9e3370 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9d8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55622/0 (socket says 172.21.15.28:55622) 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.722+0000 7f3cea7fc640 1 -- 172.21.15.28:0/959345943 learned_addr learned my addr 172.21.15.28:0/959345943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 -- 172.21.15.28:0/959345943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9e3370 msgr2=0x55d0fd9e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 -- 172.21.15.28:0/959345943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 msgr2=0x55d0fd9d8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cf0ff9640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ce00a33b0 0x7f3ce00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 -- 172.21.15.28:0/959345943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fd8ca3a0 con 0x55d0fd9d9b40 2024-09-15T22:04:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cf0ff9640 1 -- 172.21.15.28:0/1122424946 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f3cec0f1500 con 0x7f3cf4106f10 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9d8090 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0058a0 tx=0x7f3cdc00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcbfda633c4fee7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ce00a33b0 0x7f3ce00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3ce97fa640 1 -- 172.21.15.28:0/959345943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00f480 con 0x55d0fd9d9b40 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3ce97fa640 1 -- 172.21.15.28:0/959345943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cdc00fc80 con 0x55d0fd9d9b40 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.723+0000 7f3ce97fa640 1 -- 172.21.15.28:0/959345943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc01e490 con 0x55d0fd9d9b40 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 -- 172.21.15.28:0/959345943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 msgr2=0x55d0fd9d8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ce00a33b0 0x7f3ce00a5870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc01b000 tx=0x7f3cdc01ebb0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9d8090 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0058a0 tx=0x7f3cdc00b7c0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 -- 172.21.15.28:0/959345943 shutdown_connections 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/959345943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9d8090 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 -- 172.21.15.28:0/959345943 >> 172.21.15.28:0/959345943 conn(0x55d0fd9a1310 msgr2=0x55d0fd9d4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 -- 172.21.15.28:0/959345943 shutdown_connections 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 -- 172.21.15.28:0/959345943 wait complete. 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.724+0000 7f3cfbae6740 1 -- start start 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 1 Processor -- start 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9e2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9dff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd9e6070 con 0x55d0fd9d85d0 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd8f9060 con 0x55d0fd9e3370 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd9e61f0 con 0x55d0fd9d9b40 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cfbae6740 10 client.0 _ll_register_callbacks cb 0x55d0fd64c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9e2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9e2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55632/0 (socket says 172.21.15.28:55632) 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 -- 172.21.15.28:0/3294548723 learned_addr learned my addr 172.21.15.28:0/3294548723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9dff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 -- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 msgr2=0x55d0fd9dff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9dff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 -- 172.21.15.28:0/3294548723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 msgr2=0x55d0fd9e0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cf2ffd640 1 -- 172.21.15.28:0/3294548723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fd6af3c0 con 0x55d0fd9d85d0 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.725+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9e2c00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400bf80 tx=0x7f3ce4011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd4aa3fa69a7667 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cda7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cda7fc640 1 -- 172.21.15.28:0/3294548723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4012710 con 0x55d0fd9d85d0 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cda7fc640 1 -- 172.21.15.28:0/3294548723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f3ce40128b0 con 0x55d0fd9d85d0 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cdc00fc80 con 0x55d0fd9d85d0 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cda7fc640 1 -- 172.21.15.28:0/3294548723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3ce4012ab0 con 0x55d0fd9d85d0 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 10 client.16644 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d0fd9f1a90 con 0x55d0fd9d85d0 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cda7fc640 1 -- 172.21.15.28:0/3294548723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cdc00fc80 con 0x55d0fd9d85d0 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cda7fc640 1 -- 172.21.15.28:0/3294548723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f3ce401a3d0 con 0x55d0fd9d85d0 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 10 client.16644 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 10 client.16644 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 20 client.16644 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fd9f5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 4 client.16644 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.726+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0fd94b860 con 0x55d0fd9f2f40 2024-09-15T22:04:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.727+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fd9f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.727+0000 7f3cda7fc640 10 client.16644 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.727+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fd9f5400 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a4424ac9678949c server_cookie=bd19ff61773585d8 in_seq=0 out_seq=0 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cda7fc640 1 -- 172.21.15.28:0/3294548723 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d0fd94b860 con 0x55d0fd9f2f40 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cda7fc640 10 client.16644 handle_command_reply: tid=0 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 client.16644 shutdown 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d85d0 msgr2=0x55d0fd9e2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9e2c00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400bf80 tx=0x7f3ce4011870 comp rx=0 tx=0).stop 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 shutdown_connections 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fd9f5400 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cda7fc640 0 client.16644 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/3294548723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9e2c00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 >> 172.21.15.28:0/3294548723 conn(0x55d0fd9a1310 msgr2=0x55d0fd9da930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.728+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 shutdown_connections 2024-09-15T22:04:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.729+0000 7f3cfbae6740 1 -- 172.21.15.28:0/3294548723 wait complete. 2024-09-15T22:04:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.729+0000 7f3cfbae6740 20 client.16644 trim_cache size 0 max 16384 2024-09-15T22:04:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 Processor -- start 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 -- start start 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9ef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9e3370 0x55d0fd9facd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf41b1340 con 0x55d0fd9e3370 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf41b2e80 con 0x55d0fd9d85d0 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf41b1a10 con 0x55d0fd9d9b40 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9e3370 0x55d0fd9facd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9e3370 0x55d0fd9facd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55634/0 (socket says 172.21.15.28:55634) 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cf37fe640 1 -- 172.21.15.28:0/515321177 learned_addr learned my addr 172.21.15.28:0/515321177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.742+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3cf37fe640 1 -- 172.21.15.28:0/515321177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 msgr2=0x55d0fd9ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3cf37fe640 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3cf37fe640 1 -- 172.21.15.28:0/515321177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 msgr2=0x55d0fd9d4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3cf37fe640 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3cf37fe640 1 -- 172.21.15.28:0/515321177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fd9db120 con 0x55d0fd9e3370 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3cf37fe640 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9e3370 0x55d0fd9facd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0bd510 tx=0x7f3cec0bd410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5fbfb6e8c97439 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3ce8ff9640 1 -- 172.21.15.28:0/515321177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec0bd580 con 0x55d0fd9e3370 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3ce8ff9640 1 -- 172.21.15.28:0/515321177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cec036040 con 0x55d0fd9e3370 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.743+0000 7f3ce8ff9640 1 -- 172.21.15.28:0/515321177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec0f1170 con 0x55d0fd9e3370 2024-09-15T22:04:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 -- 172.21.15.28:0/515321177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9e3370 msgr2=0x55d0fd9facd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9e3370 0x55d0fd9facd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0bd510 tx=0x7f3cec0bd410 comp rx=0 tx=0).stop 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 -- 172.21.15.28:0/515321177 shutdown_connections 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9e3370 0x55d0fd9facd0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/515321177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9d4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 -- 172.21.15.28:0/515321177 >> 172.21.15.28:0/515321177 conn(0x55d0fd9a1310 msgr2=0x55d0fd9d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 -- 172.21.15.28:0/515321177 shutdown_connections 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 -- 172.21.15.28:0/515321177 wait complete. 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.744+0000 7f3cfbae6740 1 -- start start 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 1 Processor -- start 2024-09-15T22:04:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9ebc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf41b27b0 con 0x55d0fd9d9b40 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf41b20e0 con 0x55d0fd9e3370 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf41af340 con 0x55d0fd9d85d0 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cfbae6740 10 client.0 _ll_register_callbacks cb 0x7f3ce400b4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9e6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55648/0 (socket says 172.21.15.28:55648) 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cf2ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9ebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cea7fc640 1 -- 172.21.15.28:0/1640546941 learned_addr learned my addr 172.21.15.28:0/1640546941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cf37fe640 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cf2ffd640 1 -- 172.21.15.28:0/1640546941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 msgr2=0x55d0fd9e74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cf2ffd640 1 -- 172.21.15.28:0/1640546941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 msgr2=0x55d0fd9e6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9e6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cf2ffd640 1 -- 172.21.15.28:0/1640546941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fd9e7cf0 con 0x55d0fd9d85d0 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.745+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cf2ffd640 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9ebc30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400bf50 tx=0x7f3ce400b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=926e653351b4fd22 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cda7fc640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cda7fc640 1 -- 172.21.15.28:0/1640546941 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4012710 con 0x55d0fd9d85d0 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cda7fc640 1 -- 172.21.15.28:0/1640546941 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9555+0+0 (secure 0 0 0) 0x7f3ce40128b0 con 0x55d0fd9d85d0 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cec036040 con 0x55d0fd9d85d0 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cda7fc640 1 -- 172.21.15.28:0/1640546941 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3ce401aa10 con 0x55d0fd9d85d0 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cfbae6740 10 client.26137 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d0fda5dd30 con 0x55d0fd9d85d0 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.746+0000 7f3cda7fc640 1 -- 172.21.15.28:0/1640546941 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cec036040 con 0x55d0fd9d85d0 2024-09-15T22:04:14.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cda7fc640 1 -- 172.21.15.28:0/1640546941 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f3ce4004680 con 0x55d0fd9d85d0 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cfbae6740 10 client.26137 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cfbae6740 10 client.26137 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cfbae6740 20 client.26137 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fda60490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cfbae6740 4 client.26137 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0fd945d70 con 0x55d0fd9f2f40 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fda60490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cda7fc640 10 client.26137 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.747+0000 7f3cea7fc640 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fda60490 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d088857c23cd0df server_cookie=7313091a012aadcc in_seq=0 out_seq=0 2024-09-15T22:04:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cda7fc640 1 -- 172.21.15.28:0/1640546941 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0fd945d70 con 0x55d0fd9f2f40 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cda7fc640 10 client.26137 handle_command_reply: tid=0 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 client.26137 shutdown 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d85d0 msgr2=0x55d0fd9ebc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9ebc30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400bf50 tx=0x7f3ce400b7b0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 shutdown_connections 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55d0fd9f2f40 0x55d0fda60490 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d0fd9e3370 0x55d0fd9e74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fd9d9b40 0x55d0fd9e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cda7fc640 0 client.26137 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:04:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 --2- 172.21.15.28:0/1640546941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d0fd9d85d0 0x55d0fd9ebc30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 >> 172.21.15.28:0/1640546941 conn(0x55d0fd9a1310 msgr2=0x55d0fd9da110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 shutdown_connections 2024-09-15T22:04:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 1 -- 172.21.15.28:0/1640546941 wait complete. 2024-09-15T22:04:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cfbae6740 20 client.26137 trim_cache size 0 max 16384 2024-09-15T22:04:14.762 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:04:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ce00a33b0 msgr2=0x7f3ce00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cf970d640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ce00a33b0 0x7f3ce00a5870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc01b000 tx=0x7f3cdc01ebb0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 msgr2=0x7f3cf4074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:14.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.748+0000 7f3cf970d640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf4074f70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3cec003f10 tx=0x7f3cec003f40 comp rx=0 tx=0).stop 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 shutdown_connections 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3ce00a33b0 0x7f3ce00a5870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf4106f10 0x7f3cf4074f70 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3cf4102790 0x7f3cf4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 --2- 172.21.15.28:0/1122424946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cf4101de0 0x7f3cf40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 >> 172.21.15.28:0/1122424946 conn(0x7f3cf40fd8c0 msgr2=0x7f3cf40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 shutdown_connections 2024-09-15T22:04:14.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:14.749+0000 7f3cf970d640 1 -- 172.21.15.28:0/1122424946 wait complete. 2024-09-15T22:04:14.772 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.KkVqFptazH is 21KB 2024-09-15T22:04:14.822 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.KkVqFptazH 2024-09-15T22:04:14.839 DEBUG:tasks.quiescer.fs.[cephfs]:read 21490B of ops from '/var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.f.json' on mds.4 (f) 2024-09-15T22:04:14.843 INFO:tasks.quiescer.fs.[cephfs]:Pulled 9 ops from rank 4 (f) into home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/ops-ebf3369a-1-mds.f.json 2024-09-15T22:04:14.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -- /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.f.json 2024-09-15T22:04:14.905 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-15T22:04:14.934 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.9wPk8B0B5D 2024-09-15T22:04:14.935 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.h.json /tmp/tmp.9wPk8B0B5D 2024-09-15T22:04:15.072 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.9wPk8B0B5D 2024-09-15T22:04:15.192 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.9wPk8B0B5D is 45KB 2024-09-15T22:04:15.215 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.9wPk8B0B5D 2024-09-15T22:04:15.232 DEBUG:tasks.quiescer.fs.[cephfs]:read 45970B of ops from '/var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.h.json' on mds.3 (h) 2024-09-15T22:04:15.238 INFO:tasks.quiescer.fs.[cephfs]:Pulled 18 ops from rank 3 (h) into home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/ops-ebf3369a-1-mds.h.json 2024-09-15T22:04:15.238 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -- /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.h.json 2024-09-15T22:04:15.299 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-15T22:04:15.318 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.c43iCZAndv 2024-09-15T22:04:15.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.c.json /tmp/tmp.c43iCZAndv 2024-09-15T22:04:15.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.c43iCZAndv 2024-09-15T22:04:15.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:04:15.516 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.c43iCZAndv is 31KB 2024-09-15T22:04:15.566 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.c43iCZAndv 2024-09-15T22:04:15.582 DEBUG:tasks.quiescer.fs.[cephfs]:read 31893B of ops from '/var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.c.json' on mds.2 (c) 2024-09-15T22:04:15.586 INFO:tasks.quiescer.fs.[cephfs]:Pulled 13 ops from rank 2 (c) into home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/ops-ebf3369a-1-mds.c.json 2024-09-15T22:04:15.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -- /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.c.json 2024-09-15T22:04:15.647 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-15T22:04:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:15.704 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.cgxECyW6hV 2024-09-15T22:04:15.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.i.json /tmp/tmp.cgxECyW6hV 2024-09-15T22:04:15.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.cgxECyW6hV 2024-09-15T22:04:15.893 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.cgxECyW6hV is 21KB 2024-09-15T22:04:15.942 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.cgxECyW6hV 2024-09-15T22:04:15.961 DEBUG:tasks.quiescer.fs.[cephfs]:read 21606B of ops from '/var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.i.json' on mds.1 (i) 2024-09-15T22:04:15.964 INFO:tasks.quiescer.fs.[cephfs]:Pulled 9 ops from rank 1 (i) into home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/ops-ebf3369a-1-mds.i.json 2024-09-15T22:04:15.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -- /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.i.json 2024-09-15T22:04:16.025 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-15T22:04:16.043 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.fvUh21n2td 2024-09-15T22:04:16.043 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.b.json /tmp/tmp.fvUh21n2td 2024-09-15T22:04:16.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:15 smithi028 ceph-mon[30969]: pgmap v211: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-09-15T22:04:16.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:15 smithi028 ceph-mon[30969]: osdmap e95: 12 total, 12 up, 12 in 2024-09-15T22:04:16.113 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.fvUh21n2td 2024-09-15T22:04:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.173+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2413745743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c08102d40 msgr2=0x7f5c08103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.173+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2413745743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c08102d40 0x7f5c08103120 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009ca0 tx=0x7f5bf802a950 comp rx=0 tx=0).stop 2024-09-15T22:04:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.173+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2413745743 shutdown_connections 2024-09-15T22:04:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.173+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2413745743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c0810c170 0x7f5c0810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.173+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2413745743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c08103660 0x7f5c0810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.173+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2413745743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c08102d40 0x7f5c08103120 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.173+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2413745743 >> 172.21.15.28:0/2413745743 conn(0x7f5c080fc7a0 msgr2=0x7f5c080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2413745743 shutdown_connections 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2413745743 wait complete. 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 Processor -- start 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 -- start start 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c08102d40 0x7f5c08198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c08103660 0x7f5c08198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c0810c170 0x7f5c0819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c08111f70 con 0x7f5c0810c170 2024-09-15T22:04:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c08111df0 con 0x7f5c08103660 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0fe7f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081120f0 con 0x7f5c08102d40 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.174+0000 7f5c0e41c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c0810c170 0x7f5c0819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c0810c170 0x7f5c0819cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55670/0 (socket says 172.21.15.28:55670) 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 -- 172.21.15.28:0/2838941673 learned_addr learned my addr 172.21.15.28:0/2838941673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c08102d40 0x7f5c08198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c08103660 0x7f5c08198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 -- 172.21.15.28:0/2838941673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c08102d40 msgr2=0x7f5c08198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c08102d40 0x7f5c08198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 -- 172.21.15.28:0/2838941673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c08103660 msgr2=0x7f5c08198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c08103660 0x7f5c08198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 -- 172.21.15.28:0/2838941673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0819d400 con 0x7f5c0810c170 2024-09-15T22:04:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c08103660 0x7f5c08198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c0810c170 0x7f5c0819cd20 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000d750 tx=0x7f5c0000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c06ffd640 1 -- 172.21.15.28:0/2838941673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00017070 con 0x7f5c0810c170 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c06ffd640 1 -- 172.21.15.28:0/2838941673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c00005500 con 0x7f5c0810c170 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0819d6f0 con 0x7f5c0810c170 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c06ffd640 1 -- 172.21.15.28:0/2838941673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00004d90 con 0x7f5c0810c170 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.175+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c081a4fd0 con 0x7f5c0810c170 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.178+0000 7f5c06ffd640 1 -- 172.21.15.28:0/2838941673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f5c00004f30 con 0x7f5c0810c170 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.178+0000 7f5c12258740 1 Processor -- start 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.178+0000 7f5c12258740 1 -- start start 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c12258740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf07b40 0x55b5adf06090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c12258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf06a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c12258740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf11370 0x55b5adf13760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c12258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5ade6c2f0 con 0x55b5adf065d0 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c12258740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5ade13cf0 con 0x55b5adf11370 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c12258740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b5ade682b0 con 0x55b5adf07b40 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf06a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf06a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55672/0 (socket says 172.21.15.28:55672) 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/772626134 learned_addr learned my addr 172.21.15.28:0/772626134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf07b40 0x55b5adf06090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf11370 0x55b5adf13760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/772626134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf07b40 msgr2=0x55b5adf06090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf07b40 0x55b5adf06090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/772626134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf11370 msgr2=0x55b5adf13760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf11370 0x55b5adf13760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/772626134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5addf83a0 con 0x55b5adf065d0 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf11370 0x55b5adf13760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c06ffd640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bf00a33b0 0x7f5bf00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bf00a33b0 0x7f5bf00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf06a10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8004ad0 tx=0x7f5bf80098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ea04a38a0758b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.179+0000 7f5c06ffd640 1 -- 172.21.15.28:0/2838941673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f5c000cfa80 con 0x7f5c0810c170 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.180+0000 7f5beffff640 1 -- 172.21.15.28:0/772626134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8048070 con 0x55b5adf065d0 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.180+0000 7f5beffff640 1 -- 172.21.15.28:0/772626134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf802f940 con 0x55b5adf065d0 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.180+0000 7f5beffff640 1 -- 172.21.15.28:0/772626134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8034040 con 0x55b5adf065d0 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.180+0000 7f5c12258740 1 -- 172.21.15.28:0/772626134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 msgr2=0x55b5adf06a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.180+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bf00a33b0 0x7f5bf00a5870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55b5adf07070 tx=0x7f5bfc005850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.180+0000 7f5c12258740 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf06a10 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8004ad0 tx=0x7f5bf80098d0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- 172.21.15.28:0/772626134 shutdown_connections 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf11370 0x55b5adf13760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf06a10 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 --2- 172.21.15.28:0/772626134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf07b40 0x55b5adf06090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- 172.21.15.28:0/772626134 >> 172.21.15.28:0/772626134 conn(0x55b5adecf310 msgr2=0x55b5adf02c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- 172.21.15.28:0/772626134 shutdown_connections 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- 172.21.15.28:0/772626134 wait complete. 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- start start 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 Processor -- start 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf10db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf11370 0x55b5adf0e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5adf14010 con 0x55b5adf065d0 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5ade27060 con 0x55b5adf07b40 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.181+0000 7f5c12258740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b5adf14190 con 0x55b5adf11370 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c12258740 10 client.0 _ll_register_callbacks cb 0x55b5adb7a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf10db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf10db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55682/0 (socket says 172.21.15.28:55682) 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/347393266 learned_addr learned my addr 172.21.15.28:0/347393266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf11370 0x55b5adf0e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/347393266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf11370 msgr2=0x55b5adf0e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf11370 0x55b5adf0e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 msgr2=0x55b5adf0e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/347393266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5adbdd3c0 con 0x55b5adf065d0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf11370 0x55b5adf0e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf10db0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8003d60 tx=0x7f5bf802fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747650c0358798da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5bed7fa640 1 -- 172.21.15.28:0/347393266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8048030 con 0x55b5adf065d0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5bed7fa640 1 -- 172.21.15.28:0/347393266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f5bf8040050 con 0x55b5adf065d0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf802f940 con 0x55b5adf065d0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5bed7fa640 1 -- 172.21.15.28:0/347393266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5bf8034040 con 0x55b5adf065d0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c12258740 10 client.16662 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.182+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b5adf1fa70 con 0x55b5adf065d0 2024-09-15T22:04:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5bed7fa640 1 -- 172.21.15.28:0/347393266 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf802f940 con 0x55b5adf065d0 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5bed7fa640 1 -- 172.21.15.28:0/347393266 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f5bf804e4f0 con 0x55b5adf065d0 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5c12258740 10 client.16662 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5c12258740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5c12258740 20 client.16662 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5c12258740 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf20dc0 0x55b5adf231b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5c12258740 4 client.16662 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5ade79860 con 0x55b5adf20dc0 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.183+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf20dc0 0x55b5adf231b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5bed7fa640 10 client.16662 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf20dc0 0x55b5adf231b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c0f83acccdf5e82 server_cookie=834d3a6cf93afcbc in_seq=0 out_seq=0 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5bed7fa640 1 -- 172.21.15.28:0/347393266 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5ade79860 con 0x55b5adf20dc0 2024-09-15T22:04:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5bed7fa640 10 client.16662 handle_command_reply: tid=0 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5c12258740 1 client.16662 shutdown 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 msgr2=0x55b5adf10db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5c12258740 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf10db0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8003d60 tx=0x7f5bf802fd20 comp rx=0 tx=0).stop 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 shutdown_connections 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.184+0000 7f5c12258740 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf20dc0 0x55b5adf231b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5c12258740 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf11370 0x55b5adf0e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5c12258740 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5c12258740 1 --2- 172.21.15.28:0/347393266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf065d0 0x55b5adf10db0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5bed7fa640 0 client.16662 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 >> 172.21.15.28:0/347393266 conn(0x55b5adecf310 msgr2=0x55b5adf08960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 shutdown_connections 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5c12258740 1 -- 172.21.15.28:0/347393266 wait complete. 2024-09-15T22:04:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.185+0000 7f5c12258740 20 client.16662 trim_cache size 0 max 16384 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.197+0000 7f5c12258740 1 Processor -- start 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c12258740 1 -- start start 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c12258740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf0cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c12258740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf28750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c12258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf28c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c12258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081af000 con 0x55b5adf11370 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c12258740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081b0b40 con 0x55b5adf07b40 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c12258740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081af6d0 con 0x55b5adf065d0 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf28c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf28c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55688/0 (socket says 172.21.15.28:55688) 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 -- 172.21.15.28:0/896572095 learned_addr learned my addr 172.21.15.28:0/896572095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf0cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf28750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 -- 172.21.15.28:0/896572095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 msgr2=0x55b5adf0cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf0cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 -- 172.21.15.28:0/896572095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 msgr2=0x55b5adf28750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf28750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 -- 172.21.15.28:0/896572095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5adf03960 con 0x55b5adf11370 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.198+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf28c90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009610 tx=0x7f5c00000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7796d538c2f225 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5bef7fe640 1 -- 172.21.15.28:0/896572095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00020040 con 0x55b5adf11370 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5bef7fe640 1 -- 172.21.15.28:0/896572095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c0009b950 con 0x55b5adf11370 2024-09-15T22:04:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5bef7fe640 1 -- 172.21.15.28:0/896572095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00097fc0 con 0x55b5adf11370 2024-09-15T22:04:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 -- 172.21.15.28:0/896572095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 msgr2=0x55b5adf28c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf28c90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009610 tx=0x7f5c00000bc0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 -- 172.21.15.28:0/896572095 shutdown_connections 2024-09-15T22:04:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf28c90 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf28750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 --2- 172.21.15.28:0/896572095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf0cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 -- 172.21.15.28:0/896572095 >> 172.21.15.28:0/896572095 conn(0x55b5adecf310 msgr2=0x55b5adf0c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 -- 172.21.15.28:0/896572095 shutdown_connections 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 -- 172.21.15.28:0/896572095 wait complete. 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.199+0000 7f5c12258740 1 -- start start 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 1 Processor -- start 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf198a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf0b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081b0470 con 0x55b5adf11370 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081afda0 con 0x55b5adf07b40 2024-09-15T22:04:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c081ad000 con 0x55b5adf065d0 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c12258740 10 client.0 _ll_register_callbacks cb 0x7f5bf8046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0e41c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf0b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0dc1b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0dc1b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf198a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:38858/0 (socket says 172.21.15.28:38858) 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0e41c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf0b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55698/0 (socket says 172.21.15.28:55698) 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0dc1b640 1 -- 172.21.15.28:0/434522488 learned_addr learned my addr 172.21.15.28:0/434522488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0d41a640 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0e41c640 1 -- 172.21.15.28:0/434522488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 msgr2=0x55b5adf198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf198a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0e41c640 1 -- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 msgr2=0x55b5adf0ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.200+0000 7f5c0e41c640 1 -- 172.21.15.28:0/434522488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5adf0b5e0 con 0x55b5adf11370 2024-09-15T22:04:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c0e41c640 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf0b070 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x55b5adf1f9f0 tx=0x7f5c00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad9e2c562085370 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5bed7fa640 1 -- 172.21.15.28:0/434522488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c000b8070 con 0x55b5adf11370 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5bed7fa640 1 -- 172.21.15.28:0/434522488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f5c000ba2c0 con 0x55b5adf11370 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c0009b950 con 0x55b5adf11370 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5bed7fa640 1 -- 172.21.15.28:0/434522488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5c00020070 con 0x55b5adf11370 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 10 client.16668 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b5adf8b480 con 0x55b5adf11370 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5bed7fa640 1 -- 172.21.15.28:0/434522488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c0009b950 con 0x55b5adf11370 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5bed7fa640 1 -- 172.21.15.28:0/434522488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f5c0009ada0 con 0x55b5adf11370 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 10 client.16668 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 10 client.16668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 20 client.16668 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf8bc00 0x55b5adf8e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 4 client.16668 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.201+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5ade73d70 con 0x55b5adf8bc00 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.202+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf8bc00 0x55b5adf8e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.202+0000 7f5c0dc1b640 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf8bc00 0x55b5adf8e020 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45b9df7f4519e5d4 server_cookie=495bf4e742e8519e in_seq=0 out_seq=0 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.202+0000 7f5bed7fa640 10 client.16668 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.202+0000 7f5bed7fa640 1 -- 172.21.15.28:0/434522488 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5ade73d70 con 0x55b5adf8bc00 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.202+0000 7f5bed7fa640 10 client.16668 handle_command_reply: tid=0 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.202+0000 7f5c12258740 1 client.16668 shutdown 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 msgr2=0x55b5adf0b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf0b070 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x55b5adf1f9f0 tx=0x7f5c00009bd0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 shutdown_connections 2024-09-15T22:04:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55b5adf8bc00 0x55b5adf8e020 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5adf11370 0x55b5adf0b070 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5adf07b40 0x55b5adf0ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5bed7fa640 0 client.16668 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 --2- 172.21.15.28:0/434522488 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b5adf065d0 0x55b5adf198a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 >> 172.21.15.28:0/434522488 conn(0x55b5adecf310 msgr2=0x55b5adf08930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 shutdown_connections 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 1 -- 172.21.15.28:0/434522488 wait complete. 2024-09-15T22:04:16.229 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:04:16.230 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:15 smithi203 ceph-mon[35546]: pgmap v211: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-09-15T22:04:16.230 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:15 smithi203 ceph-mon[35546]: osdmap e95: 12 total, 12 up, 12 in 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c12258740 20 client.16668 trim_cache size 0 max 16384 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bf00a33b0 msgr2=0x7f5bf00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bf00a33b0 0x7f5bf00a5870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55b5adf07070 tx=0x7f5bfc005850 comp rx=0 tx=0).stop 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c0810c170 msgr2=0x7f5c0819cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c0810c170 0x7f5c0819cd20 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000d750 tx=0x7f5c0000dc10 comp rx=0 tx=0).stop 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 shutdown_connections 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bf00a33b0 0x7f5bf00a5870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c0810c170 0x7f5c0819cd20 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c08103660 0x7f5c08198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 --2- 172.21.15.28:0/2838941673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c08102d40 0x7f5c08198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.203+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 >> 172.21.15.28:0/2838941673 conn(0x7f5c080fc7a0 msgr2=0x7f5c080fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.204+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 shutdown_connections 2024-09-15T22:04:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:16.204+0000 7f5c0fe7f640 1 -- 172.21.15.28:0/2838941673 wait complete. 2024-09-15T22:04:16.249 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.fvUh21n2td is 1MB 2024-09-15T22:04:16.259 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:15 smithi174 ceph-mon[35244]: pgmap v211: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-09-15T22:04:16.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:15 smithi174 ceph-mon[35244]: osdmap e95: 12 total, 12 up, 12 in 2024-09-15T22:04:16.328 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.fvUh21n2td 2024-09-15T22:04:16.346 DEBUG:tasks.quiescer.fs.[cephfs]:read 1103291B of ops from '/var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.b.json' on mds.0 (b) 2024-09-15T22:04:16.465 INFO:tasks.quiescer.fs.[cephfs]:Pulled 411 ops from rank 0 (b) into home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/ops-ebf3369a-1-mds.b.json 2024-09-15T22:04:16.466 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -- /var/run/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ops-ebf3369a-1-mds.b.json 2024-09-15T22:04:16.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:04:16.930 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 -- 172.21.15.28:0/3158110872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100e30 msgr2=0x7f0024104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 --2- 172.21.15.28:0/3158110872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100e30 0x7f0024104e80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0018009900 tx=0x7f001802a410 comp rx=0 tx=0).stop 2024-09-15T22:04:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 -- 172.21.15.28:0/3158110872 shutdown_connections 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 --2- 172.21.15.28:0/3158110872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00241055b0 0x7f002410d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 --2- 172.21.15.28:0/3158110872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100e30 0x7f0024104e80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 --2- 172.21.15.28:0/3158110872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0024100480 0x7f0024100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 -- 172.21.15.28:0/3158110872 >> 172.21.15.28:0/3158110872 conn(0x7f0024077210 msgr2=0x7f00240fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 -- 172.21.15.28:0/3158110872 shutdown_connections 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.422+0000 7f0029574640 1 -- 172.21.15.28:0/3158110872 wait complete. 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 Processor -- start 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 -- start start 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100480 0x7f0024198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0024100e30 0x7f0024198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00241055b0 0x7f002419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0024111f90 con 0x7f0024100480 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0024111e10 con 0x7f0024100e30 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0029574640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0024112110 con 0x7f00241055b0 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0022ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100480 0x7f0024198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0022ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100480 0x7f0024198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55726/0 (socket says 172.21.15.28:55726) 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.423+0000 7f0022ffd640 1 -- 172.21.15.28:0/3074263314 learned_addr learned my addr 172.21.15.28:0/3074263314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f00227fc640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0024100e30 0x7f0024198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0022ffd640 1 -- 172.21.15.28:0/3074263314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00241055b0 msgr2=0x7f002419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f00237fe640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00241055b0 0x7f002419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:17.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0022ffd640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00241055b0 0x7f002419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0022ffd640 1 -- 172.21.15.28:0/3074263314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0024100e30 msgr2=0x7f0024198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0022ffd640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0024100e30 0x7f0024198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0022ffd640 1 -- 172.21.15.28:0/3074263314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002419d400 con 0x7f0024100480 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f00227fc640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0024100e30 0x7f0024198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f00237fe640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00241055b0 0x7f002419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0022ffd640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100480 0x7f0024198450 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f001400c970 tx=0x7f001400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f000bfff640 1 -- 172.21.15.28:0/3074263314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014007bb0 con 0x7f0024100480 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002419d6f0 con 0x7f0024100480 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f000bfff640 1 -- 172.21.15.28:0/3074263314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0014007d50 con 0x7f0024100480 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f000bfff640 1 -- 172.21.15.28:0/3074263314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001400f450 con 0x7f0024100480 2024-09-15T22:04:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.424+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00241a4fd0 con 0x7f0024100480 2024-09-15T22:04:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.425+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0024100860 con 0x7f0024100480 2024-09-15T22:04:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.426+0000 7f000bfff640 1 -- 172.21.15.28:0/3074263314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f0014005020 con 0x7f0024100480 2024-09-15T22:04:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.427+0000 7f000bfff640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f000c0a33b0 0x7f000c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.427+0000 7f000bfff640 1 -- 172.21.15.28:0/3074263314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f00140cff70 con 0x7f0024100480 2024-09-15T22:04:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.427+0000 7f00227fc640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f000c0a33b0 0x7f000c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.428+0000 7f00227fc640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f000c0a33b0 0x7f000c0a5870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0024102e70 tx=0x7f0018005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.432+0000 7f000bfff640 1 -- 172.21.15.28:0/3074263314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0014014030 con 0x7f0024100480 2024-09-15T22:04:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.646+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00241087c0 con 0x7f0024100480 2024-09-15T22:04:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.647+0000 7f000bfff640 1 -- 172.21.15.28:0/3074263314 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11986 (secure 0 0 0) 0x7f001409c5a0 con 0x7f0024100480 2024-09-15T22:04:17.649 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:04:17.649 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"btime":"2024-09-15T22:01:08:357214+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:01:08.288094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:04:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-09-15T22:04:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f000c0a33b0 msgr2=0x7f000c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f000c0a33b0 0x7f000c0a5870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0024102e70 tx=0x7f0018005a20 comp rx=0 tx=0).stop 2024-09-15T22:04:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100480 msgr2=0x7f0024198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100480 0x7f0024198450 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f001400c970 tx=0x7f001400ce30 comp rx=0 tx=0).stop 2024-09-15T22:04:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 shutdown_connections 2024-09-15T22:04:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f000c0a33b0 0x7f000c0a5870 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00241055b0 0x7f002419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0024100e30 0x7f0024198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 --2- 172.21.15.28:0/3074263314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0024100480 0x7f0024198450 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.649+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 >> 172.21.15.28:0/3074263314 conn(0x7f0024077210 msgr2=0x7f0024103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.650+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 shutdown_connections 2024-09-15T22:04:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:17.650+0000 7f0029574640 1 -- 172.21.15.28:0/3074263314 wait complete. 2024-09-15T22:04:17.847 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T22:04:17.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T22:04:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:17 smithi028 ceph-mon[30969]: pgmap v213: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-09-15T22:04:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:17 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3074263314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:17 smithi203 ceph-mon[35546]: pgmap v213: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-09-15T22:04:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:17 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3074263314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:17 smithi174 ceph-mon[35244]: pgmap v213: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-09-15T22:04:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:17 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3074263314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:18.441 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id ebf3369a --await-for 45.0' 2024-09-15T22:04:18.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id ebf3369a --await-for 45.0 2024-09-15T22:04:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 -- 172.21.15.28:0/981740132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102d30 msgr2=0x7f6e3c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/981740132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102d30 0x7f6e3c103110 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c009900 tx=0x7f6e2c02a480 comp rx=0 tx=0).stop 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 -- 172.21.15.28:0/981740132 shutdown_connections 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/981740132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e3c10c160 0x7f6e3c10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/981740132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e3c103650 0x7f6e3c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/981740132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c102d30 0x7f6e3c103110 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 -- 172.21.15.28:0/981740132 >> 172.21.15.28:0/981740132 conn(0x7f6e3c0fc790 msgr2=0x7f6e3c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 -- 172.21.15.28:0/981740132 shutdown_connections 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.047+0000 7f6e41fc8640 1 -- 172.21.15.28:0/981740132 wait complete. 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.048+0000 7f6e41fc8640 1 Processor -- start 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.048+0000 7f6e41fc8640 1 -- start start 2024-09-15T22:04:19.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.048+0000 7f6e41fc8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e3c102d30 0x7f6e3c198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.048+0000 7f6e41fc8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e3c103650 0x7f6e3c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e41fc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c10c160 0x7f6e3c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e41fc8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c111f60 con 0x7f6e3c10c160 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e41fc8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c111de0 con 0x7f6e3c103650 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e41fc8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1120e0 con 0x7f6e3c102d30 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c10c160 0x7f6e3c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c10c160 0x7f6e3c19cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55748/0 (socket says 172.21.15.28:55748) 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 -- 172.21.15.28:0/896734677 learned_addr learned my addr 172.21.15.28:0/896734677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e3c102d30 0x7f6e3c198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3affd640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e3c103650 0x7f6e3c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 -- 172.21.15.28:0/896734677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e3c102d30 msgr2=0x7f6e3c198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e3c102d30 0x7f6e3c198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 -- 172.21.15.28:0/896734677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e3c103650 msgr2=0x7f6e3c198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e3c103650 0x7f6e3c198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 -- 172.21.15.28:0/896734677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3c19d2d0 con 0x7f6e3c10c160 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c10c160 0x7f6e3c19cbf0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400cce0 tx=0x7f6e3400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e38ff9640 1 -- 172.21.15.28:0/896734677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3400f450 con 0x7f6e3c10c160 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.049+0000 7f6e38ff9640 1 -- 172.21.15.28:0/896734677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e3400fad0 con 0x7f6e3c10c160 2024-09-15T22:04:19.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.050+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e3c19d5c0 con 0x7f6e3c10c160 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.050+0000 7f6e38ff9640 1 -- 172.21.15.28:0/896734677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e34018c10 con 0x7f6e3c10c160 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.050+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e3c1a4ea0 con 0x7f6e3c10c160 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.052+0000 7f6e443a1740 1 Processor -- start 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e38ff9640 1 -- 172.21.15.28:0/896734677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6e3400f5f0 con 0x7f6e3c10c160 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e443a1740 1 -- start start 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0826a40 0x5607a0826e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0835c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0836150 0x5607a0838540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e443a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5607a078c280 con 0x5607a0826a40 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e443a1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5607a0733b10 con 0x5607a0827360 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e443a1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5607a0788240 con 0x5607a0836150 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e3affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0826a40 0x5607a0826e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e3affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0826a40 0x5607a0826e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55754/0 (socket says 172.21.15.28:55754) 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.053+0000 7f6e3affd640 1 -- 172.21.15.28:0/3755077053 learned_addr learned my addr 172.21.15.28:0/3755077053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0835c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3affd640 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0836150 0x5607a0838540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3affd640 1 -- 172.21.15.28:0/3755077053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0836150 msgr2=0x5607a0838540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3affd640 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0836150 0x5607a0838540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3affd640 1 -- 172.21.15.28:0/3755077053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 msgr2=0x5607a0835c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3affd640 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0835c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3affd640 1 -- 172.21.15.28:0/3755077053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607a07401f0 con 0x5607a0826a40 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e38ff9640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6e240a33b0 0x7f6e240a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3affd640 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0826a40 0x5607a0826e20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6e30010370 tx=0x7f6e30008570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da9337e68446735 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6e240a33b0 0x7f6e240a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e38ff9640 1 -- 172.21.15.28:0/896734677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6e340d0880 con 0x7f6e3c10c160 2024-09-15T22:04:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e21ffb640 1 -- 172.21.15.28:0/3755077053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30005020 con 0x5607a0826a40 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e21ffb640 1 -- 172.21.15.28:0/3755077053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e300047a0 con 0x5607a0826a40 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.054+0000 7f6e21ffb640 1 -- 172.21.15.28:0/3755077053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e300128c0 con 0x5607a0826a40 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 -- 172.21.15.28:0/3755077053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0826a40 msgr2=0x5607a0826e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6e240a33b0 0x7f6e240a5870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x5607a0829750 tx=0x7f6e2c005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0826a40 0x5607a0826e20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6e30010370 tx=0x7f6e30008570 comp rx=0 tx=0).stop 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 -- 172.21.15.28:0/3755077053 shutdown_connections 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0836150 0x5607a0838540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0835c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3755077053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0826a40 0x5607a0826e20 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 -- 172.21.15.28:0/3755077053 >> 172.21.15.28:0/3755077053 conn(0x5607a08230f0 msgr2=0x5607a08234d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 -- 172.21.15.28:0/3755077053 shutdown_connections 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 -- 172.21.15.28:0/3755077053 wait complete. 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.055+0000 7f6e443a1740 1 -- start start 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 1 Processor -- start 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a08330c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0833600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a0833b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5607a0838d20 con 0x5607a0836150 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5607a0746f10 con 0x5607a0827360 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5607a0838ea0 con 0x5607a0826a40 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e443a1740 10 client.0 _ll_register_callbacks cb 0x5607a070fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:19.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a0833b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a0833b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55762/0 (socket says 172.21.15.28:55762) 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e3bfff640 1 -- 172.21.15.28:0/3075147184 learned_addr learned my addr 172.21.15.28:0/3075147184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e3affd640 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0833600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a08330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e3bfff640 1 -- 172.21.15.28:0/3075147184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 msgr2=0x5607a08330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.056+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a08330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e3bfff640 1 -- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 msgr2=0x5607a0833600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0833600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e3bfff640 1 -- 172.21.15.28:0/3075147184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607a04fd6a0 con 0x5607a0836150 2024-09-15T22:04:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a0833b40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6e34098bb0 tx=0x7f6e3409c0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff867bb7ae6b6fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e1affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e1affd640 1 -- 172.21.15.28:0/3075147184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340a0b00 con 0x5607a0836150 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e1affd640 1 -- 172.21.15.28:0/3075147184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6e34098dc0 con 0x5607a0836150 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e300047a0 con 0x5607a0836150 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e1affd640 1 -- 172.21.15.28:0/3075147184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e340a53c0 con 0x5607a0836150 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e443a1740 10 client.16686 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5607a0844a40 con 0x5607a0836150 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e1affd640 1 -- 172.21.15.28:0/3075147184 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e300047a0 con 0x5607a0836150 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.057+0000 7f6e1affd640 1 -- 172.21.15.28:0/3075147184 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f6e340a4f30 con 0x5607a0836150 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e443a1740 10 client.16686 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e443a1740 10 client.16686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e443a1740 20 client.16686 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a0848390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e443a1740 4 client.16686 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607a07997f0 con 0x5607a0845f50 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a0848390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a0848390 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c58d938f8d72711c server_cookie=e87345c6ae0a7f73 in_seq=0 out_seq=0 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.058+0000 7f6e1affd640 10 client.16686 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e1affd640 1 -- 172.21.15.28:0/3075147184 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5607a07997f0 con 0x5607a0845f50 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e1affd640 10 client.16686 handle_command_reply: tid=0 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 client.16686 shutdown 2024-09-15T22:04:19.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 msgr2=0x5607a0833b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a0833b40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6e34098bb0 tx=0x7f6e3409c0b0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 shutdown_connections 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a0848390 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a0833b40 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e1affd640 0 client.16686 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a0833600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3075147184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a08330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 >> 172.21.15.28:0/3075147184 conn(0x5607a08230f0 msgr2=0x5607a0828e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.059+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 shutdown_connections 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.060+0000 7f6e443a1740 1 -- 172.21.15.28:0/3075147184 wait complete. 2024-09-15T22:04:19.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.060+0000 7f6e443a1740 20 client.16686 trim_cache size 0 max 16384 2024-09-15T22:04:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 Processor -- start 2024-09-15T22:04:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 -- start start 2024-09-15T22:04:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a084dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0827360 0x5607a084e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0836150 0x5607a0830320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e3affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0827360 0x5607a084e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1acc40 con 0x5607a0827360 2024-09-15T22:04:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1ae780 con 0x5607a0836150 2024-09-15T22:04:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e443a1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1ad310 con 0x5607a0826a40 2024-09-15T22:04:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e3affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0827360 0x5607a084e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55776/0 (socket says 172.21.15.28:55776) 2024-09-15T22:04:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e3affd640 1 -- 172.21.15.28:0/1066554551 learned_addr learned my addr 172.21.15.28:0/1066554551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.098+0000 7f6e3b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a084dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0836150 0x5607a0830320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e3affd640 1 -- 172.21.15.28:0/1066554551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 msgr2=0x5607a084dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e3affd640 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a084dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e3affd640 1 -- 172.21.15.28:0/1066554551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0836150 msgr2=0x5607a0830320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e3affd640 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0836150 0x5607a0830320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e3affd640 1 -- 172.21.15.28:0/1066554551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607a0822b90 con 0x5607a0827360 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e3affd640 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0827360 0x5607a084e1a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6e30010fa0 tx=0x7f6e30014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828c95f7ad63517f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.099+0000 7f6e217fa640 1 -- 172.21.15.28:0/1066554551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30005310 con 0x5607a0827360 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.100+0000 7f6e217fa640 1 -- 172.21.15.28:0/1066554551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e30012b40 con 0x5607a0827360 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.100+0000 7f6e217fa640 1 -- 172.21.15.28:0/1066554551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30003af0 con 0x5607a0827360 2024-09-15T22:04:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 -- 172.21.15.28:0/1066554551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0827360 msgr2=0x5607a084e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0827360 0x5607a084e1a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6e30010fa0 tx=0x7f6e30014040 comp rx=0 tx=0).stop 2024-09-15T22:04:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 -- 172.21.15.28:0/1066554551 shutdown_connections 2024-09-15T22:04:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0836150 0x5607a0830320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0827360 0x5607a084e1a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 --2- 172.21.15.28:0/1066554551 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a084dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 -- 172.21.15.28:0/1066554551 >> 172.21.15.28:0/1066554551 conn(0x5607a08230f0 msgr2=0x5607a0835030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 -- 172.21.15.28:0/1066554551 shutdown_connections 2024-09-15T22:04:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 -- 172.21.15.28:0/1066554551 wait complete. 2024-09-15T22:04:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.101+0000 7f6e443a1740 1 -- start start 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.102+0000 7f6e443a1740 1 Processor -- start 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.102+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a083e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.102+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a083e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.102+0000 7f6e443a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a082ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.102+0000 7f6e443a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1ae0b0 con 0x5607a0836150 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.102+0000 7f6e443a1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1ad9e0 con 0x5607a0827360 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.102+0000 7f6e443a1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1aac40 con 0x5607a0826a40 2024-09-15T22:04:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a082ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a083e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a083e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:38916/0 (socket says 172.21.15.28:38916) 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 -- 172.21.15.28:0/3416390459 learned_addr learned my addr 172.21.15.28:0/3416390459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a083e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e443a1740 10 client.0 _ll_register_callbacks cb 0x7f6e340077a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 -- 172.21.15.28:0/3416390459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 msgr2=0x5607a083e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a083e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 -- 172.21.15.28:0/3416390459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 msgr2=0x5607a082ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a082ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 -- 172.21.15.28:0/3416390459 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607a082b450 con 0x5607a0826a40 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3bfff640 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a082ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e30012b40 con 0x5607a0826a40 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e3b7fe640 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a083e130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c02a9a0 tx=0x7f6e2c03d070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f14dd636952198e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.103+0000 7f6e1affd640 10 client.26167 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.104+0000 7f6e1affd640 1 -- 172.21.15.28:0/3416390459 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c05e050 con 0x5607a0826a40 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.104+0000 7f6e1affd640 1 -- 172.21.15.28:0/3416390459 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6e2c003dd0 con 0x5607a0826a40 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.104+0000 7f6e1affd640 1 -- 172.21.15.28:0/3416390459 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e2c035050 con 0x5607a0826a40 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.104+0000 7f6e443a1740 10 client.26167 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.104+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=32}) -- 0x5607a08b5bf0 con 0x5607a0826a40 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e1affd640 1 -- 172.21.15.28:0/3416390459 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f6e2c049430 con 0x5607a0826a40 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e443a1740 10 client.26167 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e443a1740 10 client.26167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e443a1740 20 client.26167 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a08b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e443a1740 4 client.26167 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5607a0793d00 con 0x5607a0845f50 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.105+0000 7f6e3affd640 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a08b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.106+0000 7f6e3affd640 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a08b81f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca04a89b47a36d83 server_cookie=d1133cf247c31c8c in_seq=0 out_seq=0 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.106+0000 7f6e1affd640 10 client.26167 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.107+0000 7f6e1affd640 1 -- 172.21.15.28:0/3416390459 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5607a0793d00 con 0x5607a0845f50 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.107+0000 7f6e1affd640 10 client.26167 handle_command_reply: tid=0 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.107+0000 7f6e443a1740 1 client.26167 shutdown 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 msgr2=0x5607a083e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a083e130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c02a9a0 tx=0x7f6e2c03d070 comp rx=0 tx=0).stop 2024-09-15T22:04:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 shutdown_connections 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5607a0845f50 0x5607a08b81f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607a0836150 0x5607a082ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5607a0827360 0x5607a083e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 --2- 172.21.15.28:0/3416390459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607a0826a40 0x5607a083e130 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 >> 172.21.15.28:0/3416390459 conn(0x5607a08230f0 msgr2=0x5607a0821a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e1affd640 0 client.26167 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 shutdown_connections 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 1 -- 172.21.15.28:0/3416390459 wait complete. 2024-09-15T22:04:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.108+0000 7f6e443a1740 20 client.26167 trim_cache size 0 max 16384 2024-09-15T22:04:19.118 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T22:04:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "1f439ed9-facf-4c87-aee7-1b16c01ad875", 2024-09-15T22:04:19.118 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T22:04:19.118 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6e240a33b0 msgr2=0x7f6e240a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6e240a33b0 0x7f6e240a5870 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x5607a0829750 tx=0x7f6e2c005a20 comp rx=0 tx=0).stop 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c10c160 msgr2=0x7f6e3c19cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c10c160 0x7f6e3c19cbf0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400cce0 tx=0x7f6e3400a620 comp rx=0 tx=0).stop 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 shutdown_connections 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6e240a33b0 0x7f6e240a5870 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c10c160 0x7f6e3c19cbf0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e3c103650 0x7f6e3c198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 --2- 172.21.15.28:0/896734677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e3c102d30 0x7f6e3c198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 >> 172.21.15.28:0/896734677 conn(0x7f6e3c0fc790 msgr2=0x7f6e3c0fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 shutdown_connections 2024-09-15T22:04:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.109+0000 7f6e41fc8640 1 -- 172.21.15.28:0/896734677 wait complete. 2024-09-15T22:04:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 -- 172.21.15.28:0/297867690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102ef0 msgr2=0x7f5bd410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/297867690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102ef0 0x7f5bd410da40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009900 tx=0x7f5bcc02a480 comp rx=0 tx=0).stop 2024-09-15T22:04:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 -- 172.21.15.28:0/297867690 shutdown_connections 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/297867690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102ef0 0x7f5bd410da40 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/297867690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bd4102570 0x7f5bd41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/297867690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bd4100e20 0x7f5bd4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 -- 172.21.15.28:0/297867690 >> 172.21.15.28:0/297867690 conn(0x7f5bd40fc7d0 msgr2=0x7f5bd40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.860+0000 7f5bdb9da640 1 -- 172.21.15.28:0/297867690 shutdown_connections 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 -- 172.21.15.28:0/297867690 wait complete. 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 Processor -- start 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 -- start start 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bd4100e20 0x7f5bd4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102570 0x7f5bd4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bd4102ef0 0x7f5bd419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4110f80 con 0x7f5bd4102570 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4110e00 con 0x7f5bd4100e20 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bdb9da640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd4111100 con 0x7f5bd4102ef0 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.861+0000 7f5bd8f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102570 0x7f5bd4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102570 0x7f5bd4198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55806/0 (socket says 172.21.15.28:55806) 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 -- 172.21.15.28:0/682657255 learned_addr learned my addr 172.21.15.28:0/682657255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bd4102ef0 0x7f5bd419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd9776640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bd4100e20 0x7f5bd4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 -- 172.21.15.28:0/682657255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bd4102ef0 msgr2=0x7f5bd419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bd4102ef0 0x7f5bd419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 -- 172.21.15.28:0/682657255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bd4100e20 msgr2=0x7f5bd4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bd4100e20 0x7f5bd4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 -- 172.21.15.28:0/682657255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd419d3e0 con 0x7f5bd4102570 2024-09-15T22:04:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102570 0x7f5bd4198970 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800d950 tx=0x7f5bc800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd27fc640 1 -- 172.21.15.28:0/682657255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8017070 con 0x7f5bd4102570 2024-09-15T22:04:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd27fc640 1 -- 172.21.15.28:0/682657255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc8009c30 con 0x7f5bd4102570 2024-09-15T22:04:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bd27fc640 1 -- 172.21.15.28:0/682657255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc800f6b0 con 0x7f5bd4102570 2024-09-15T22:04:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd419d6d0 con 0x7f5bd4102570 2024-09-15T22:04:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.862+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd41a4fb0 con 0x7f5bd4102570 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.865+0000 7f5bd27fc640 1 -- 172.21.15.28:0/682657255 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f5bc80109d0 con 0x7f5bd4102570 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.866+0000 7f5bd27fc640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bbc0a33b0 0x7f5bbc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.866+0000 7f5bd27fc640 1 -- 172.21.15.28:0/682657255 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f5bc80cfdc0 con 0x7f5bd4102570 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.866+0000 7f5bd9776640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bbc0a33b0 0x7f5bbc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.866+0000 7f5bdddb3740 1 Processor -- start 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.866+0000 7f5bd9776640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bbc0a33b0 0x7f5bbc0a5870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4005fd0 tx=0x7f5bc4009870 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.866+0000 7f5bdddb3740 1 -- start start 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c67280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c789e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d2f4bcd410 con 0x55d2f4c765f0 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d2f4b73e10 con 0x55d2f4c66ea0 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d2f4bc93d0 con 0x55d2f4c677c0 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd9f77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd9f77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c789e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55812/0 (socket says 172.21.15.28:55812) 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd9f77640 1 -- 172.21.15.28:0/1736793238 learned_addr learned my addr 172.21.15.28:0/1736793238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd9776640 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c67280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd8f75640 1 -- 172.21.15.28:0/1736793238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 msgr2=0x55d2f4c67280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c67280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd8f75640 1 -- 172.21.15.28:0/1736793238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 msgr2=0x55d2f4c789e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c789e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd8f75640 1 -- 172.21.15.28:0/1736793238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2f493e870 con 0x55d2f4c677c0 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.867+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.868+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c760b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80cd100 tx=0x7f5bc8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cc0a4f8441cf39b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.868+0000 7f5bbb7fe640 1 -- 172.21.15.28:0/1736793238 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8020040 con 0x55d2f4c677c0 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.868+0000 7f5bbb7fe640 1 -- 172.21.15.28:0/1736793238 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc80a0040 con 0x55d2f4c677c0 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.868+0000 7f5bbb7fe640 1 -- 172.21.15.28:0/1736793238 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc809b340 con 0x55d2f4c677c0 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 -- 172.21.15.28:0/1736793238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 msgr2=0x55d2f4c760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c760b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80cd100 tx=0x7f5bc8005f90 comp rx=0 tx=0).stop 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 -- 172.21.15.28:0/1736793238 shutdown_connections 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c760b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/1736793238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c67280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 -- 172.21.15.28:0/1736793238 >> 172.21.15.28:0/1736793238 conn(0x55d2f4c63590 msgr2=0x55d2f4c63970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 -- 172.21.15.28:0/1736793238 shutdown_connections 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 -- 172.21.15.28:0/1736793238 wait complete. 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 -- start start 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.869+0000 7f5bdddb3740 1 Processor -- start 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c735b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c677c0 0x55d2f4c73af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c74030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d2f4c79160 con 0x55d2f4c765f0 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d2f4b55c50 con 0x55d2f4c677c0 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d2f4c792e0 con 0x55d2f4c66ea0 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bdddb3740 10 client.0 _ll_register_callbacks cb 0x55d2f48dbd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9776640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9f77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c74030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9776640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c735b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:38946/0 (socket says 172.21.15.28:38946) 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9f77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c74030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55820/0 (socket says 172.21.15.28:55820) 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9776640 1 -- 172.21.15.28:0/3021763802 learned_addr learned my addr 172.21.15.28:0/3021763802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c677c0 0x55d2f4c73af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9f77640 1 -- 172.21.15.28:0/3021763802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c66ea0 msgr2=0x55d2f4c735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c735b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9f77640 1 -- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c677c0 msgr2=0x55d2f4c73af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c677c0 0x55d2f4c73af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9f77640 1 -- 172.21.15.28:0/3021763802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2f4b563c0 con 0x55d2f4c765f0 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd9776640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.870+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c677c0 0x55d2f4c73af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c74030 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006010 tx=0x7f5bcc004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66a94d524fabc76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3021763802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc035030 con 0x55d2f4c765f0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3021763802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f5bcc005220 con 0x55d2f4c765f0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc80a0040 con 0x55d2f4c765f0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3021763802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc04d050 con 0x55d2f4c765f0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bdddb3740 10 client.16704 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2f4c84e80 con 0x55d2f4c765f0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3021763802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc80a0040 con 0x55d2f4c765f0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3021763802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f5bcc0049b0 con 0x55d2f4c765f0 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bdddb3740 10 client.16704 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.871+0000 7f5bdddb3740 10 client.16704 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:04:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.872+0000 7f5bdddb3740 20 client.16704 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.872+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4c86340 0x55d2f4c88730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.872+0000 7f5bdddb3740 4 client.16704 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.872+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2f4bda980 con 0x55d2f4c86340 2024-09-15T22:04:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.872+0000 7f5bd9776640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4c86340 0x55d2f4c88730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.872+0000 7f5bd9776640 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4c86340 0x55d2f4c88730 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6acb56a4ec1f723 server_cookie=51ccfae389a5931 in_seq=0 out_seq=0 2024-09-15T22:04:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.872+0000 7f5bb8ff9640 10 client.16704 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3021763802 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d2f4bda980 con 0x55d2f4c86340 2024-09-15T22:04:19.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bb8ff9640 10 client.16704 handle_command_reply: tid=0 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 client.16704 shutdown 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 msgr2=0x55d2f4c74030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c74030 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006010 tx=0x7f5bcc004440 comp rx=0 tx=0).stop 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 shutdown_connections 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4c86340 0x55d2f4c88730 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c765f0 0x55d2f4c74030 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c677c0 0x55d2f4c73af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3021763802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 >> 172.21.15.28:0/3021763802 conn(0x55d2f4c63590 msgr2=0x55d2f4c692f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.873+0000 7f5bb8ff9640 0 client.16704 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.874+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 shutdown_connections 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.874+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3021763802 wait complete. 2024-09-15T22:04:19.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.874+0000 7f5bdddb3740 20 client.16704 trim_cache size 0 max 16384 2024-09-15T22:04:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 Processor -- start 2024-09-15T22:04:19.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 -- start start 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c8dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c8e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c765f0 0x55d2f4c61ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41acdb0 con 0x55d2f4c66ea0 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41ae8f0 con 0x55d2f4c765f0 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.891+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41ad480 con 0x55d2f4c677c0 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c8dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c8dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55824/0 (socket says 172.21.15.28:55824) 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 -- 172.21.15.28:0/2400486086 learned_addr learned my addr 172.21.15.28:0/2400486086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c765f0 0x55d2f4c61ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c8e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 -- 172.21.15.28:0/2400486086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 msgr2=0x55d2f4c8e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c8e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 -- 172.21.15.28:0/2400486086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c765f0 msgr2=0x55d2f4c61ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c765f0 0x55d2f4c61ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 -- 172.21.15.28:0/2400486086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2f4c5ef50 con 0x55d2f4c66ea0 2024-09-15T22:04:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c765f0 0x55d2f4c61ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bd9776640 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c8dfe0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4015040 tx=0x7f5bc4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d3565dbe4a70d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.892+0000 7f5bbaffd640 1 -- 172.21.15.28:0/2400486086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4021d90 con 0x55d2f4c66ea0 2024-09-15T22:04:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bbaffd640 1 -- 172.21.15.28:0/2400486086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc402d030 con 0x55d2f4c66ea0 2024-09-15T22:04:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bbaffd640 1 -- 172.21.15.28:0/2400486086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc40113e0 con 0x55d2f4c66ea0 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 -- 172.21.15.28:0/2400486086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c66ea0 msgr2=0x55d2f4c8dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c8dfe0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4015040 tx=0x7f5bc4016000 comp rx=0 tx=0).stop 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 -- 172.21.15.28:0/2400486086 shutdown_connections 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c765f0 0x55d2f4c61ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c677c0 0x55d2f4c8e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/2400486086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c8dfe0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 -- 172.21.15.28:0/2400486086 >> 172.21.15.28:0/2400486086 conn(0x55d2f4c63590 msgr2=0x55d2f4c6e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.893+0000 7f5bdddb3740 1 -- 172.21.15.28:0/2400486086 shutdown_connections 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 -- 172.21.15.28:0/2400486086 wait complete. 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 -- start start 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 Processor -- start 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c7e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c677c0 0x55d2f4c7ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c765f0 0x55d2f4c6fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41ae220 con 0x55d2f4c677c0 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41adb50 con 0x55d2f4c66ea0 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd41aadb0 con 0x55d2f4c765f0 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.894+0000 7f5bdddb3740 10 client.0 _ll_register_callbacks cb 0x7f5bcc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c677c0 0x55d2f4c7ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c677c0 0x55d2f4c7ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55828/0 (socket says 172.21.15.28:55828) 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 -- 172.21.15.28:0/3990560917 learned_addr learned my addr 172.21.15.28:0/3990560917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd9776640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c7e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c765f0 0x55d2f4c6fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 -- 172.21.15.28:0/3990560917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c765f0 msgr2=0x55d2f4c6fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c765f0 0x55d2f4c6fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 -- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 msgr2=0x55d2f4c7e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c7e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 -- 172.21.15.28:0/3990560917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2f4c70620 con 0x55d2f4c677c0 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd9f77640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c765f0 0x55d2f4c6fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bd8f75640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c677c0 0x55d2f4c7ed00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5bc809bc20 tx=0x7f5bc8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2c86036d72e610 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.895+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3990560917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80b9060 con 0x55d2f4c677c0 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3990560917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f5bc809b2e0 con 0x55d2f4c677c0 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc402d030 con 0x55d2f4c677c0 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3990560917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5bc8020040 con 0x55d2f4c677c0 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 10 client.16710 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2f4cf0ae0 con 0x55d2f4c677c0 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3990560917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc402d030 con 0x55d2f4c677c0 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3990560917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f5bc809ab50 con 0x55d2f4c677c0 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 10 client.16710 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 10 client.16710 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 20 client.16710 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4cf0e10 0x55d2f4cf3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 4 client.16710 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ebf3369a", "await_for": 45.0}] 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.896+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ebf3369a", "await_for": 45.0}) -- 0x55d2f4bd4e90 con 0x55d2f4cf0e10 2024-09-15T22:04:19.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.897+0000 7f5bd9776640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4cf0e10 0x55d2f4cf3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.897+0000 7f5bb8ff9640 10 client.16710 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.897+0000 7f5bd9776640 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4cf0e10 0x55d2f4cf3200 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5ec1c2fc8af2b9d server_cookie=9a8fbac0ba1f28e0 in_seq=0 out_seq=0 2024-09-15T22:04:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bb8ff9640 1 -- 172.21.15.28:0/3990560917 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+592 (crc 0 0 0) 0x55d2f4bd4e90 con 0x55d2f4cf0e10 2024-09-15T22:04:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bb8ff9640 10 client.16710 handle_command_reply: tid=0 2024-09-15T22:04:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 client.16710 shutdown 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c677c0 msgr2=0x55d2f4c7ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c677c0 0x55d2f4c7ed00 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5bc809bc20 tx=0x7f5bc8000bc0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 shutdown_connections 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d2f4cf0e10 0x55d2f4cf3200 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d2f4c765f0 0x55d2f4c6fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2f4c677c0 0x55d2f4c7ed00 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 --2- 172.21.15.28:0/3990560917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d2f4c66ea0 0x55d2f4c7e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bb8ff9640 0 client.16710 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 >> 172.21.15.28:0/3990560917 conn(0x55d2f4c63590 msgr2=0x55d2f4c61ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 shutdown_connections 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 1 -- 172.21.15.28:0/3990560917 wait complete. 2024-09-15T22:04:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.898+0000 7f5bdddb3740 20 client.16710 trim_cache size 0 max 16384 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 32, 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 11, 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "ebf3369a": { 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 11, 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-09-15T22:04:19.905 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 90.0, 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 139.6, 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 41.6 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:19.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:19.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:19.907 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bbc0a33b0 msgr2=0x7f5bbc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bbc0a33b0 0x7f5bbc0a5870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4005fd0 tx=0x7f5bc4009870 comp rx=0 tx=0).stop 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102570 msgr2=0x7f5bd4198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102570 0x7f5bd4198970 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800d950 tx=0x7f5bc800de10 comp rx=0 tx=0).stop 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 shutdown_connections 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5bbc0a33b0 0x7f5bbc0a5870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bd4102ef0 0x7f5bd419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bd4102570 0x7f5bd4198970 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 --2- 172.21.15.28:0/682657255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bd4100e20 0x7f5bd4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 >> 172.21.15.28:0/682657255 conn(0x7f5bd40fc7d0 msgr2=0x7f5bd406a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 shutdown_connections 2024-09-15T22:04:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:19.899+0000 7f5bdb9da640 1 -- 172.21.15.28:0/682657255 wait complete. 2024-09-15T22:04:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 -- 172.21.15.28:0/3866447319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564107b80 msgr2=0x7f4564107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 --2- 172.21.15.28:0/3866447319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564107b80 0x7f4564107f60 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f4560009930 tx=0x7f456002a420 comp rx=0 tx=0).stop 2024-09-15T22:04:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 -- 172.21.15.28:0/3866447319 shutdown_connections 2024-09-15T22:04:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 --2- 172.21.15.28:0/3866447319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4564102480 0x7f456410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 --2- 172.21.15.28:0/3866447319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4564101ae0 0x7f4564101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 --2- 172.21.15.28:0/3866447319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564107b80 0x7f4564107f60 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 -- 172.21.15.28:0/3866447319 >> 172.21.15.28:0/3866447319 conn(0x7f45640fd7e0 msgr2=0x7f45640ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 -- 172.21.15.28:0/3866447319 shutdown_connections 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.936+0000 7f456b5f8640 1 -- 172.21.15.28:0/3866447319 wait complete. 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 Processor -- start 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 -- start start 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564101ae0 0x7f4564198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4564102480 0x7f4564198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4564107b80 0x7f456419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4564111f30 con 0x7f4564101ae0 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4564111db0 con 0x7f4564102480 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f456b5f8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45641120b0 con 0x7f4564107b80 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564101ae0 0x7f4564198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564101ae0 0x7f4564198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55856/0 (socket says 172.21.15.28:55856) 2024-09-15T22:04:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 -- 172.21.15.28:0/2053641727 learned_addr learned my addr 172.21.15.28:0/2053641727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569b95640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4564107b80 0x7f456419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4568b93640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4564102480 0x7f4564198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 -- 172.21.15.28:0/2053641727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4564107b80 msgr2=0x7f456419cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4564107b80 0x7f456419cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 -- 172.21.15.28:0/2053641727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4564102480 msgr2=0x7f4564198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4564102480 0x7f4564198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569394640 1 -- 172.21.15.28:0/2053641727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456419d2f0 con 0x7f4564101ae0 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4569b95640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4564107b80 0x7f456419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.937+0000 7f4568b93640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4564102480 0x7f4564198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.938+0000 7f4569394640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564101ae0 0x7f4564198300 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4560009230 tx=0x7f4560004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.938+0000 7f455a7fc640 1 -- 172.21.15.28:0/2053641727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456003f030 con 0x7f4564101ae0 2024-09-15T22:04:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.938+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f456419d580 con 0x7f4564101ae0 2024-09-15T22:04:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.938+0000 7f455a7fc640 1 -- 172.21.15.28:0/2053641727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4560003ad0 con 0x7f4564101ae0 2024-09-15T22:04:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.938+0000 7f455a7fc640 1 -- 172.21.15.28:0/2053641727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560035070 con 0x7f4564101ae0 2024-09-15T22:04:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.938+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45641a4e70 con 0x7f4564101ae0 2024-09-15T22:04:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.940+0000 7f455a7fc640 1 -- 172.21.15.28:0/2053641727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f4560042030 con 0x7f4564101ae0 2024-09-15T22:04:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 Processor -- start 2024-09-15T22:04:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 -- start start 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd801ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f455a7fc640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f45500a3460 0x7f45500a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8028770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd802b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563dd7f822f0 con 0x563dd8028cb0 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563dd7f2a270 con 0x563dd801e410 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f456e468740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563dd7f7e2b0 con 0x563dd801daf0 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd802b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f4569394640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f45500a3460 0x7f45500a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd802b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55862/0 (socket says 172.21.15.28:55862) 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f4569b95640 1 -- 172.21.15.28:0/159297606 learned_addr learned my addr 172.21.15.28:0/159297606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f455a7fc640 1 -- 172.21.15.28:0/2053641727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f45600f06e0 con 0x7f4564101ae0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f4569b95640 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd801ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.941+0000 7f4568b93640 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8028770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f4569b95640 1 -- 172.21.15.28:0/159297606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 msgr2=0x563dd801ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f4569b95640 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd801ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f4569b95640 1 -- 172.21.15.28:0/159297606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 msgr2=0x563dd8028770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f4569b95640 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8028770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f4569b95640 1 -- 172.21.15.28:0/159297606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd7f29340 con 0x563dd8028cb0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f4569b95640 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd802b0a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f455c014c90 tx=0x7f455c009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c227b6e475ee1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f4569394640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f45500a3460 0x7f45500a5920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f45600bc2b0 tx=0x7f45600b1a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f454f7fe640 1 -- 172.21.15.28:0/159297606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c014310 con 0x563dd8028cb0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f454f7fe640 1 -- 172.21.15.28:0/159297606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f455c014e40 con 0x563dd8028cb0 2024-09-15T22:04:19.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.942+0000 7f454f7fe640 1 -- 172.21.15.28:0/159297606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c004020 con 0x563dd8028cb0 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 -- 172.21.15.28:0/159297606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 msgr2=0x563dd802b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd802b0a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f455c014c90 tx=0x7f455c009790 comp rx=0 tx=0).stop 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 -- 172.21.15.28:0/159297606 shutdown_connections 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd802b0a0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8028770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 --2- 172.21.15.28:0/159297606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd801ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 -- 172.21.15.28:0/159297606 >> 172.21.15.28:0/159297606 conn(0x563dd801a060 msgr2=0x563dd801a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 -- 172.21.15.28:0/159297606 shutdown_connections 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 -- 172.21.15.28:0/159297606 wait complete. 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 -- start start 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.943+0000 7f456e468740 1 Processor -- start 2024-09-15T22:04:19.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f456e468740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd8025c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f456e468740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8026140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f456e468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd8026680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f456e468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563dd802b880 con 0x563dd8028cb0 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f456e468740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563dd7f0b290 con 0x563dd801e410 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f456e468740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563dd802ba00 con 0x563dd801daf0 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd8026680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd8026680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55870/0 (socket says 172.21.15.28:55870) 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 -- 172.21.15.28:0/675930956 learned_addr learned my addr 172.21.15.28:0/675930956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f456e468740 10 client.0 _ll_register_callbacks cb 0x563dd7f3c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd8025c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4568b93640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8026140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 -- 172.21.15.28:0/675930956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 msgr2=0x563dd8025c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd8025c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 -- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 msgr2=0x563dd8026140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8026140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 -- 172.21.15.28:0/675930956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd7c95210 con 0x563dd8028cb0 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4568b93640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8026140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f4569b95640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd8026680 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f455c003a90 tx=0x7f455c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd23975a62f2220 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f454cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.944+0000 7f454cff9640 1 -- 172.21.15.28:0/675930956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c00fdd0 con 0x563dd8028cb0 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f454cff9640 1 -- 172.21.15.28:0/675930956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f455c004a40 con 0x563dd8028cb0 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563dd7f8fa20 con 0x563dd8028cb0 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f454cff9640 1 -- 172.21.15.28:0/675930956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f455c004c40 con 0x563dd8028cb0 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 10 client.16722 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:19.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563dd7d61620 con 0x563dd8028cb0 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f454cff9640 1 -- 172.21.15.28:0/675930956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563dd7f8fa20 con 0x563dd8028cb0 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f454cff9640 1 -- 172.21.15.28:0/675930956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f455c012450 con 0x563dd8028cb0 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 10 client.16722 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 10 client.16722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 20 client.16722 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd8038b20 0x563dd803af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 4 client.16722 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.945+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dd7f89d70 con 0x563dd8038b20 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.946+0000 7f4568b93640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd8038b20 0x563dd803af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.946+0000 7f4568b93640 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd8038b20 0x563dd803af10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87a0e5e58a293b81 server_cookie=7c926cd570067e3d in_seq=0 out_seq=0 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.946+0000 7f454cff9640 10 client.16722 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f454cff9640 1 -- 172.21.15.28:0/675930956 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563dd7f89d70 con 0x563dd8038b20 2024-09-15T22:04:19.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f454cff9640 10 client.16722 handle_command_reply: tid=0 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 client.16722 shutdown 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 msgr2=0x563dd8026680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd8026680 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f455c003a90 tx=0x7f455c003b90 comp rx=0 tx=0).stop 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 shutdown_connections 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd8038b20 0x563dd803af10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd8028cb0 0x563dd8026680 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f454cff9640 0 client.16722 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd8026140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 --2- 172.21.15.28:0/675930956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd801daf0 0x563dd8025c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 >> 172.21.15.28:0/675930956 conn(0x563dd801a060 msgr2=0x563dd801fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 shutdown_connections 2024-09-15T22:04:19.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 1 -- 172.21.15.28:0/675930956 wait complete. 2024-09-15T22:04:19.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.947+0000 7f456e468740 20 client.16722 trim_cache size 0 max 16384 2024-09-15T22:04:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.963+0000 7f456e468740 1 Processor -- start 2024-09-15T22:04:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.963+0000 7f456e468740 1 -- start start 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f456e468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd8020560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f456e468740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd80264b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f456e468740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 0x563dd8040630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f456e468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45641aef20 con 0x563dd801daf0 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f456e468740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45641b0a60 con 0x563dd801e410 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f456e468740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45641af5f0 con 0x563dd8028cb0 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd8020560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd8020560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55876/0 (socket says 172.21.15.28:55876) 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 -- 172.21.15.28:0/550496475 learned_addr learned my addr 172.21.15.28:0/550496475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4568b93640 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd80264b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 -- 172.21.15.28:0/550496475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 msgr2=0x563dd8040630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 0x563dd8040630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 -- 172.21.15.28:0/550496475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 msgr2=0x563dd80264b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd80264b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 -- 172.21.15.28:0/550496475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd8019e90 con 0x563dd801daf0 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4568b93640 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd80264b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f4569b95640 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd8020560 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f455c014c90 tx=0x7f455c009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f5658449e4cff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.964+0000 7f454effd640 1 -- 172.21.15.28:0/550496475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c009e90 con 0x563dd801daf0 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f454effd640 1 -- 172.21.15.28:0/550496475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f455c009c30 con 0x563dd801daf0 2024-09-15T22:04:19.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f454effd640 1 -- 172.21.15.28:0/550496475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c012cf0 con 0x563dd801daf0 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 -- 172.21.15.28:0/550496475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 msgr2=0x563dd8020560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd8020560 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f455c014c90 tx=0x7f455c009af0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 -- 172.21.15.28:0/550496475 shutdown_connections 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 0x563dd8040630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd80264b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 --2- 172.21.15.28:0/550496475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd8020560 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 -- 172.21.15.28:0/550496475 >> 172.21.15.28:0/550496475 conn(0x563dd801a060 msgr2=0x563dd8027c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 -- 172.21.15.28:0/550496475 shutdown_connections 2024-09-15T22:04:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.965+0000 7f456e468740 1 -- 172.21.15.28:0/550496475 wait complete. 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 -- start start 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 Processor -- start 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd802c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd802c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 0x563dd809b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45641b0390 con 0x563dd801daf0 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45641afcc0 con 0x563dd801e410 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45641acf20 con 0x563dd8028cb0 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f456e468740 10 client.0 _ll_register_callbacks cb 0x7f455c014640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd802c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f4569b95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd802c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55890/0 (socket says 172.21.15.28:55890) 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f4569b95640 1 -- 172.21.15.28:0/3921247424 learned_addr learned my addr 172.21.15.28:0/3921247424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:19.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.966+0000 7f4568b93640 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd802c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f4569b95640 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 0x563dd809b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f4569b95640 1 -- 172.21.15.28:0/3921247424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 msgr2=0x563dd809b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f4569b95640 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 0x563dd809b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f4569b95640 1 -- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 msgr2=0x563dd802c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f4569b95640 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd802c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f4569b95640 1 -- 172.21.15.28:0/3921247424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd809bc60 con 0x563dd801daf0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f4569b95640 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd802c420 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f455c0128a0 tx=0x7f455c005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5254fcfd4a0834d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f454cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f454cff9640 1 -- 172.21.15.28:0/3921247424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c01f480 con 0x563dd801daf0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f454cff9640 1 -- 172.21.15.28:0/3921247424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f455c01f620 con 0x563dd801daf0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f455c009c30 con 0x563dd801daf0 2024-09-15T22:04:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f454cff9640 1 -- 172.21.15.28:0/3921247424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f455c04c050 con 0x563dd801daf0 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f456e468740 10 client.16728 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563dd809bc60 con 0x563dd801daf0 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.967+0000 7f454cff9640 1 -- 172.21.15.28:0/3921247424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f455c009c30 con 0x563dd801daf0 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f454cff9640 1 -- 172.21.15.28:0/3921247424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f455c0378d0 con 0x563dd801daf0 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f456e468740 10 client.16728 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f456e468740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f456e468740 20 client.16728 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f456e468740 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd80a3320 0x563dd80a5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f456e468740 4 client.16728 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563dd7f06900 con 0x563dd80a3320 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f4568b93640 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd80a3320 0x563dd80a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f454cff9640 10 client.16728 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.968+0000 7f4568b93640 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd80a3320 0x563dd80a5710 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5391912ba5911a3 server_cookie=e5dfda217f57f01d in_seq=0 out_seq=0 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f454cff9640 1 -- 172.21.15.28:0/3921247424 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563dd7f06900 con 0x563dd80a3320 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f454cff9640 10 client.16728 handle_command_reply: tid=0 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 client.16728 shutdown 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 msgr2=0x563dd802c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd802c420 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f455c0128a0 tx=0x7f455c005d30 comp rx=0 tx=0).stop 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 shutdown_connections 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563dd80a3320 0x563dd80a5710 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563dd8028cb0 0x563dd809b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dd801e410 0x563dd802c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 --2- 172.21.15.28:0/3921247424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563dd801daf0 0x563dd802c420 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f454cff9640 0 client.16728 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 >> 172.21.15.28:0/3921247424 conn(0x563dd801a060 msgr2=0x563dd80189a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 shutdown_connections 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 1 -- 172.21.15.28:0/3921247424 wait complete. 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.978+0000 7f456e468740 20 client.16728 trim_cache size 0 max 16384 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:04:19.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:04:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f45500a3460 msgr2=0x7f45500a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f45500a3460 0x7f45500a5920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f45600bc2b0 tx=0x7f45600b1a40 comp rx=0 tx=0).stop 2024-09-15T22:04:19.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564101ae0 msgr2=0x7f4564198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564101ae0 0x7f4564198300 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4560009230 tx=0x7f4560004270 comp rx=0 tx=0).stop 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 shutdown_connections 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f45500a3460 0x7f45500a5920 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4564107b80 0x7f456419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4564102480 0x7f4564198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 --2- 172.21.15.28:0/2053641727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4564101ae0 0x7f4564198300 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 >> 172.21.15.28:0/2053641727 conn(0x7f45640fd7e0 msgr2=0x7f45640fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 shutdown_connections 2024-09-15T22:04:19.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:19.979+0000 7f456b5f8640 1 -- 172.21.15.28:0/2053641727 wait complete. 2024-09-15T22:04:20.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: pgmap v214: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 13 op/s 2024-09-15T22:04:20.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T22:04:20.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:19 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: pgmap v214: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 13 op/s 2024-09-15T22:04:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T22:04:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:19 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: pgmap v214: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 13 op/s 2024-09-15T22:04:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T22:04:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:04:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:19 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:04:20.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:04:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:21.408 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ebf3369a', quiesce took 174.3 seconds. Will release after: -154.75 2024-09-15T22:04:21.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:04:21.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 -- 172.21.15.28:0/2879085601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102570 msgr2=0x7fcfb41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 --2- 172.21.15.28:0/2879085601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102570 0x7fcfb41029b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4009900 tx=0x7fcfa402a480 comp rx=0 tx=0).stop 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 -- 172.21.15.28:0/2879085601 shutdown_connections 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 --2- 172.21.15.28:0/2879085601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfb4102ef0 0x7fcfb410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 --2- 172.21.15.28:0/2879085601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102570 0x7fcfb41029b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 --2- 172.21.15.28:0/2879085601 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb4100e20 0x7fcfb4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 -- 172.21.15.28:0/2879085601 >> 172.21.15.28:0/2879085601 conn(0x7fcfb40fc7d0 msgr2=0x7fcfb40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.672+0000 7fcfbc206640 1 -- 172.21.15.28:0/2879085601 shutdown_connections 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 -- 172.21.15.28:0/2879085601 wait complete. 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 Processor -- start 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 -- start start 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4100e20 0x7fcfb40fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb4102570 0x7fcfb40fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfb4102ef0 0x7fcfb40fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb4110f80 con 0x7fcfb4100e20 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb4110e00 con 0x7fcfb4102570 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfbc206640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb4111100 con 0x7fcfb4102ef0 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4100e20 0x7fcfb40fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4100e20 0x7fcfb40fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39652/0 (socket says 172.21.15.28:39652) 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.673+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/3515546591 learned_addr learned my addr 172.21.15.28:0/3515546591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfb4102ef0 0x7fcfb40fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb4102570 0x7fcfb40fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/3515546591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfb4102ef0 msgr2=0x7fcfb40fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfb4102ef0 0x7fcfb40fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/3515546591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb4102570 msgr2=0x7fcfb40fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb4102570 0x7fcfb40fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/3515546591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb41a6910 con 0x7fcfb4100e20 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb4102570 0x7fcfb40fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4100e20 0x7fcfb40fcc80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000c920 tx=0x7fcfb000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfaaffd640 1 -- 172.21.15.28:0/3515546591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0007c70 con 0x7fcfb4100e20 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb41a6c00 con 0x7fcfb4100e20 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfaaffd640 1 -- 172.21.15.28:0/3515546591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0007e10 con 0x7fcfb4100e20 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfaaffd640 1 -- 172.21.15.28:0/3515546591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0018890 con 0x7fcfb4100e20 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.674+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb41a7140 con 0x7fcfb4100e20 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfaaffd640 1 -- 172.21.15.28:0/3515546591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fcfb0018a70 con 0x7fcfb4100e20 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 Processor -- start 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 -- start start 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb9004a0 0x557ebb900880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb900dc0 0x557ebb90b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb90da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ebb8662e0 con 0x557ebb900dc0 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ebb80dce0 con 0x557ebb90b680 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557ebb8622a0 con 0x557ebb9004a0 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb900dc0 0x557ebb90b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb900dc0 0x557ebb90b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39656/0 (socket says 172.21.15.28:39656) 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/2803509970 learned_addr learned my addr 172.21.15.28:0/2803509970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.677+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb9004a0 0x557ebb900880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfaaffd640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfa00a33b0 0x7fcfa00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb90da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfa00a33b0 0x7fcfa00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfaaffd640 1 -- 172.21.15.28:0/3515546591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fcfb00d0220 con 0x7fcfb4100e20 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/2803509970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb9004a0 msgr2=0x557ebb900880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb9004a0 0x557ebb900880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/2803509970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 msgr2=0x557ebb90da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb90da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/2803509970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebb7f2440 con 0x557ebb900dc0 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb9004a0 0x557ebb900880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb900dc0 0x557ebb90b140 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0098b20 tx=0x7fcfb0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7dedfe3932817e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcf9ffff640 1 -- 172.21.15.28:0/2803509970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb009bab0 con 0x557ebb900dc0 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfa00a33b0 0x7fcfa00a5870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcfac005fd0 tx=0x7fcfac004ab0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcf9ffff640 1 -- 172.21.15.28:0/2803509970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb009c850 con 0x557ebb900dc0 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.678+0000 7fcf9ffff640 1 -- 172.21.15.28:0/2803509970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0018d60 con 0x557ebb900dc0 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 -- 172.21.15.28:0/2803509970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb900dc0 msgr2=0x557ebb90b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb900dc0 0x557ebb90b140 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0098b20 tx=0x7fcfb0000bc0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 -- 172.21.15.28:0/2803509970 shutdown_connections 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb90da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb900dc0 0x557ebb90b140 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/2803509970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb9004a0 0x557ebb900880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 -- 172.21.15.28:0/2803509970 >> 172.21.15.28:0/2803509970 conn(0x557ebb8fca30 msgr2=0x557ebb8fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 -- 172.21.15.28:0/2803509970 shutdown_connections 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.679+0000 7fcfbe5df740 1 -- 172.21.15.28:0/2803509970 wait complete. 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 -- start start 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 Processor -- start 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb90afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb908280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9087e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557ebb90e250 con 0x557ebb9004a0 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557ebb821050 con 0x557ebb90b680 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557ebb90e3d0 con 0x557ebb900dc0 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfbe5df740 10 client.0 _ll_register_callbacks cb 0x557ebb861160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb90afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb90afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39668/0 (socket says 172.21.15.28:39668) 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1342619675 learned_addr learned my addr 172.21.15.28:0/1342619675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb908280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.680+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1342619675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 msgr2=0x557ebb908280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb908280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 msgr2=0x557ebb9087e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9087e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1342619675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebb5d73c0 con 0x557ebb9004a0 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb908280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb90afd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00985b0 tx=0x7fcfb00cd560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd7e67a1b5cc2f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcf9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/1342619675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb009bab0 con 0x557ebb9004a0 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/1342619675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fcfb00bc0c0 con 0x557ebb9004a0 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb009c850 con 0x557ebb9004a0 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/1342619675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcfb00a0500 con 0x557ebb9004a0 2024-09-15T22:04:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfbe5df740 10 client.16740 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557ebb919c50 con 0x557ebb9004a0 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/1342619675 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb009c850 con 0x557ebb9004a0 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/1342619675 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fcfb00a00d0 con 0x557ebb9004a0 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfbe5df740 10 client.16740 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.681+0000 7fcfbe5df740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.682+0000 7fcfbe5df740 20 client.16740 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.682+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb91afa0 0x557ebb91d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.682+0000 7fcfbe5df740 4 client.16740 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.682+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ebb873850 con 0x557ebb91afa0 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.682+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb91afa0 0x557ebb91d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.682+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb91afa0 0x557ebb91d390 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e282b0985a3104d server_cookie=e4df21e056e6c0 in_seq=0 out_seq=0 2024-09-15T22:04:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.682+0000 7fcf9d7fa640 10 client.16740 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/1342619675 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557ebb873850 con 0x557ebb91afa0 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcf9d7fa640 10 client.16740 handle_command_reply: tid=0 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 client.16740 shutdown 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 msgr2=0x557ebb90afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb90afd0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00985b0 tx=0x7fcfb00cd560 comp rx=0 tx=0).stop 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 shutdown_connections 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb91afa0 0x557ebb91d390 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb908280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcf9d7fa640 0 client.16740 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1342619675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb90afd0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 >> 172.21.15.28:0/1342619675 conn(0x557ebb8fca30 msgr2=0x557ebb8fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 shutdown_connections 2024-09-15T22:04:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1342619675 wait complete. 2024-09-15T22:04:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.683+0000 7fcfbe5df740 20 client.16740 trim_cache size 0 max 16384 2024-09-15T22:04:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.702+0000 7fcfbe5df740 1 Processor -- start 2024-09-15T22:04:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.702+0000 7fcfbe5df740 1 -- start start 2024-09-15T22:04:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb906b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb9203b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9208f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb41b1140 con 0x557ebb9004a0 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb41b2c80 con 0x557ebb90b680 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb41b1810 con 0x557ebb900dc0 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb906b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb906b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39670/0 (socket says 172.21.15.28:39670) 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1245828003 learned_addr learned my addr 172.21.15.28:0/1245828003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb9203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9208f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1245828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 msgr2=0x557ebb9203b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb9203b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1245828003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 msgr2=0x557ebb9208f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9208f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/1245828003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ebb907230 con 0x557ebb9004a0 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9208f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.703+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb906b50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fcfb009a400 tx=0x7fcfb0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f484c24e1dd1e17f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.704+0000 7fcf9f7fe640 1 -- 172.21.15.28:0/1245828003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00d7070 con 0x557ebb9004a0 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.704+0000 7fcf9f7fe640 1 -- 172.21.15.28:0/1245828003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb009c850 con 0x557ebb9004a0 2024-09-15T22:04:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.704+0000 7fcf9f7fe640 1 -- 172.21.15.28:0/1245828003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00d3b20 con 0x557ebb9004a0 2024-09-15T22:04:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1245828003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 msgr2=0x557ebb906b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb906b50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fcfb009a400 tx=0x7fcfb0000bc0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1245828003 shutdown_connections 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb9208f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb9203b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/1245828003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb906b50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1245828003 >> 172.21.15.28:0/1245828003 conn(0x557ebb8fca30 msgr2=0x557ebb903890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1245828003 shutdown_connections 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 -- 172.21.15.28:0/1245828003 wait complete. 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 -- start start 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.705+0000 7fcfbe5df740 1 Processor -- start 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb91e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb91e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfbe5df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb91ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb41b25b0 con 0x557ebb9004a0 2024-09-15T22:04:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb41b1ee0 con 0x557ebb90b680 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfbe5df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb41af140 con 0x557ebb900dc0 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfbe5df740 10 client.0 _ll_register_callbacks cb 0x7fcfa4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb91e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfb9fa2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb91e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39682/0 (socket says 172.21.15.28:39682) 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/584824204 learned_addr learned my addr 172.21.15.28:0/584824204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfba7a3640 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb91ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.706+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb91e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/584824204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 msgr2=0x557ebb91e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb91e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 msgr2=0x557ebb91ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb91ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfb9fa2640 1 -- 172.21.15.28:0/584824204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ebb9848e0 con 0x557ebb9004a0 2024-09-15T22:04:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfb9fa2640 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb91e140 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0098b20 tx=0x7fcfb00a4cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118280185745e432 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcf9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/584824204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00d7070 con 0x557ebb9004a0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/584824204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fcfb00a00d0 con 0x557ebb9004a0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb009c850 con 0x557ebb9004a0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/584824204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcfb0098960 con 0x557ebb9004a0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfbe5df740 10 client.16746 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557ebb9848e0 con 0x557ebb9004a0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/584824204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb009c850 con 0x557ebb9004a0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/584824204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fcfb009b5a0 con 0x557ebb9004a0 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfbe5df740 10 client.16746 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfbe5df740 10 client.16746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.707+0000 7fcfbe5df740 20 client.16746 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.708+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb985dd0 0x557ebb9881c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.708+0000 7fcfbe5df740 4 client.16746 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.708+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557ebb86dd60 con 0x557ebb985dd0 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.708+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb985dd0 0x557ebb9881c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcf9d7fa640 10 client.16746 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfb97a1640 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb985dd0 0x557ebb9881c0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=309ac55e3fbd1df9 server_cookie=49e6eb95074eaa79 in_seq=0 out_seq=0 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcf9d7fa640 1 -- 172.21.15.28:0/584824204 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557ebb86dd60 con 0x557ebb985dd0 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcf9d7fa640 10 client.16746 handle_command_reply: tid=0 2024-09-15T22:04:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 client.16746 shutdown 2024-09-15T22:04:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 msgr2=0x557ebb91e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb91e140 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0098b20 tx=0x7fcfb00a4cf0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 shutdown_connections 2024-09-15T22:04:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557ebb985dd0 0x557ebb9881c0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557ebb90b680 0x557ebb91ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcf9d7fa640 0 client.16746 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557ebb900dc0 0x557ebb91e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 --2- 172.21.15.28:0/584824204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ebb9004a0 0x557ebb91e140 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 >> 172.21.15.28:0/584824204 conn(0x557ebb8fca30 msgr2=0x557ebb902680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 shutdown_connections 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.709+0000 7fcfbe5df740 1 -- 172.21.15.28:0/584824204 wait complete. 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbe5df740 20 client.16746 trim_cache size 0 max 16384 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stdout: "1f439ed9-facf-4c87-aee7-1b16c01ad875": { 2024-09-15T22:04:21.716 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-15T22:04:21.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "1f439ed9-facf-4c87-aee7-1b16c01ad875", 2024-09-15T22:04:21.717 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-15T22:04:21.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:21.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:21.717 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfa00a33b0 msgr2=0x7fcfa00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfa00a33b0 0x7fcfa00a5870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcfac005fd0 tx=0x7fcfac004ab0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4100e20 msgr2=0x7fcfb40fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4100e20 0x7fcfb40fcc80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000c920 tx=0x7fcfb000cde0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 shutdown_connections 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfa00a33b0 0x7fcfa00a5870 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfb4102ef0 0x7fcfb40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb4102570 0x7fcfb40fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 --2- 172.21.15.28:0/3515546591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4100e20 0x7fcfb40fcc80 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 >> 172.21.15.28:0/3515546591 conn(0x7fcfb40fc7d0 msgr2=0x7fcfb406a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:21.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 shutdown_connections 2024-09-15T22:04:21.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:21.710+0000 7fcfbc206640 1 -- 172.21.15.28:0/3515546591 wait complete. 2024-09-15T22:04:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:21 smithi203 ceph-mon[35546]: pgmap v215: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 8.0 KiB/s wr, 23 op/s 2024-09-15T22:04:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:21 smithi174 ceph-mon[35244]: pgmap v215: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 8.0 KiB/s wr, 23 op/s 2024-09-15T22:04:22.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:21 smithi028 ceph-mon[30969]: pgmap v215: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 21 KiB/s rd, 8.0 KiB/s wr, 23 op/s 2024-09-15T22:04:22.399 INFO:tasks.cephfs.filesystem:scrub status for tag:1f439ed9-facf-4c87-aee7-1b16c01ad875 - {'path': '/', 'tag': '1f439ed9-facf-4c87-aee7-1b16c01ad875', 'options': 'recursive,force'} 2024-09-15T22:04:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 -- 172.21.15.28:0/281536399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560100a60 msgr2=0x7f1560100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 --2- 172.21.15.28:0/281536399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560100a60 0x7f1560100ec0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f1554009930 tx=0x7f155402a420 comp rx=0 tx=0).stop 2024-09-15T22:04:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 -- 172.21.15.28:0/281536399 shutdown_connections 2024-09-15T22:04:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 --2- 172.21.15.28:0/281536399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1560101400 0x7f156010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 --2- 172.21.15.28:0/281536399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560100a60 0x7f1560100ec0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 --2- 172.21.15.28:0/281536399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560106a60 0x7f1560106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 -- 172.21.15.28:0/281536399 >> 172.21.15.28:0/281536399 conn(0x7f15600fc780 msgr2=0x7f15600feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 -- 172.21.15.28:0/281536399 shutdown_connections 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.996+0000 7f1567316640 1 -- 172.21.15.28:0/281536399 wait complete. 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 Processor -- start 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 -- start start 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1560100a60 0x7f1560199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560101400 0x7f1560199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560106a60 0x7f15601a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1560110e20 con 0x7f1560106a60 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1560110ca0 con 0x7f1560101400 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f1567316640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1560110fa0 con 0x7f1560100a60 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560106a60 0x7f15601a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560106a60 0x7f15601a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39692/0 (socket says 172.21.15.28:39692) 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 -- 172.21.15.28:0/4123106777 learned_addr learned my addr 172.21.15.28:0/4123106777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15650b2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1560100a60 0x7f1560199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15648b1640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560101400 0x7f1560199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 -- 172.21.15.28:0/4123106777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1560100a60 msgr2=0x7f1560199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1560100a60 0x7f1560199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 -- 172.21.15.28:0/4123106777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560101400 msgr2=0x7f1560199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560101400 0x7f1560199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 -- 172.21.15.28:0/4123106777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15601a48f0 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.997+0000 7f15658b3640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560106a60 0x7f15601a43b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f155800eaa0 tx=0x7f155800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.998+0000 7f15527fc640 1 -- 172.21.15.28:0/4123106777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155800cdb0 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.998+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15601a4ac0 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.998+0000 7f15527fc640 1 -- 172.21.15.28:0/4123106777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1558005020 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.998+0000 7f15527fc640 1 -- 172.21.15.28:0/4123106777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15580116b0 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.998+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15601a4f80 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:22.999+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1560100ec0 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.000+0000 7f15527fc640 1 -- 172.21.15.28:0/4123106777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f1558005560 con 0x7f1560106a60 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.001+0000 7f15527fc640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f154c0a33b0 0x7f154c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.001+0000 7f15650b2640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f154c0a33b0 0x7f154c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.001+0000 7f15527fc640 1 -- 172.21.15.28:0/4123106777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f15580d0660 con 0x7f1560106a60 2024-09-15T22:04:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.001+0000 7f15650b2640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f154c0a33b0 0x7f154c0a5870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f155c00bb60 tx=0x7f155c00b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.004+0000 7f15527fc640 1 -- 172.21.15.28:0/4123106777 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1558016030 con 0x7f1560106a60 2024-09-15T22:04:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.217+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1560108ff0 con 0x7f1560106a60 2024-09-15T22:04:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.217+0000 7f15527fc640 1 -- 172.21.15.28:0/4123106777 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11986 (secure 0 0 0) 0x7f1558098020 con 0x7f1560106a60 2024-09-15T22:04:23.219 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:04:23.219 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"btime":"2024-09-15T22:01:08:357214+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:01:08.288094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:04:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-09-15T22:04:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.219+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f154c0a33b0 msgr2=0x7f154c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.219+0000 7f1567316640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f154c0a33b0 0x7f154c0a5870 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f155c00bb60 tx=0x7f155c00b390 comp rx=0 tx=0).stop 2024-09-15T22:04:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.219+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560106a60 msgr2=0x7f15601a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.219+0000 7f1567316640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560106a60 0x7f15601a43b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f155800eaa0 tx=0x7f155800ef60 comp rx=0 tx=0).stop 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 shutdown_connections 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f154c0a33b0 0x7f154c0a5870 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1560106a60 0x7f15601a43b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1560101400 0x7f1560199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 --2- 172.21.15.28:0/4123106777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1560100a60 0x7f1560199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 >> 172.21.15.28:0/4123106777 conn(0x7f15600fc780 msgr2=0x7f15600fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 shutdown_connections 2024-09-15T22:04:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:23.220+0000 7f1567316640 1 -- 172.21.15.28:0/4123106777 wait complete. 2024-09-15T22:04:23.840 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --set-id ebf3369a --release --await' 2024-09-15T22:04:23.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --set-id ebf3369a --release --await 2024-09-15T22:04:24.139 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:24.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:23 smithi028 ceph-mon[30969]: pgmap v216: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 51 KiB/s rd, 5.9 KiB/s wr, 52 op/s 2024-09-15T22:04:24.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:23 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4123106777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:23 smithi203 ceph-mon[35546]: pgmap v216: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 51 KiB/s rd, 5.9 KiB/s wr, 52 op/s 2024-09-15T22:04:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:23 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4123106777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:23 smithi174 ceph-mon[35244]: pgmap v216: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 51 KiB/s rd, 5.9 KiB/s wr, 52 op/s 2024-09-15T22:04:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:23 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4123106777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 -- 172.21.15.28:0/3573622161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8102790 msgr2=0x7faed81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 --2- 172.21.15.28:0/3573622161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8102790 0x7faed81067e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7faecc009900 tx=0x7faecc02a410 comp rx=0 tx=0).stop 2024-09-15T22:04:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 -- 172.21.15.28:0/3573622161 shutdown_connections 2024-09-15T22:04:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 --2- 172.21.15.28:0/3573622161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed8106f10 0x7faed810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 --2- 172.21.15.28:0/3573622161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8102790 0x7faed81067e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 --2- 172.21.15.28:0/3573622161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faed8101de0 0x7faed81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 -- 172.21.15.28:0/3573622161 >> 172.21.15.28:0/3573622161 conn(0x7faed80fd8c0 msgr2=0x7faed80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 -- 172.21.15.28:0/3573622161 shutdown_connections 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 -- 172.21.15.28:0/3573622161 wait complete. 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 Processor -- start 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.660+0000 7faede8fa640 1 -- start start 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faede8fa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed8101de0 0x7faed80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faede8fa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faed8102790 0x7faed8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faede8fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8106f10 0x7faed8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faede8fa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed8111f90 con 0x7faed8106f10 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faede8fa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faed8111e10 con 0x7faed8101de0 2024-09-15T22:04:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faede8fa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faed8112110 con 0x7faed8102790 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8106f10 0x7faed8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8106f10 0x7faed8074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39710/0 (socket says 172.21.15.28:39710) 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 -- 172.21.15.28:0/2908032820 learned_addr learned my addr 172.21.15.28:0/2908032820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faed7fff640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed8101de0 0x7faed80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faed77fe640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faed8102790 0x7faed8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 -- 172.21.15.28:0/2908032820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faed8102790 msgr2=0x7faed8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faed8102790 0x7faed8074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 -- 172.21.15.28:0/2908032820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed8101de0 msgr2=0x7faed80744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed8101de0 0x7faed80744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 -- 172.21.15.28:0/2908032820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed81a6950 con 0x7faed8106f10 2024-09-15T22:04:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.661+0000 7faedce97640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8106f10 0x7faed8074f70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7faed000eaa0 tx=0x7faed000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.662+0000 7faed57fa640 1 -- 172.21.15.28:0/2908032820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0004d20 con 0x7faed8106f10 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.662+0000 7faed57fa640 1 -- 172.21.15.28:0/2908032820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faed0005020 con 0x7faed8106f10 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.662+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed81a6c40 con 0x7faed8106f10 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.662+0000 7faed57fa640 1 -- 172.21.15.28:0/2908032820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed00116d0 con 0x7faed8106f10 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.662+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed81a7100 con 0x7faed8106f10 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.664+0000 7faed57fa640 1 -- 172.21.15.28:0/2908032820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7faed00051d0 con 0x7faed8106f10 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.664+0000 7faee0cd3740 1 Processor -- start 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.664+0000 7faee0cd3740 1 -- start start 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec55440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55980 0x55e12ec5fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 0x55e12ec62630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faee0cd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e12ebba620 con 0x55e12ec55060 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faee0cd3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e12eb61e50 con 0x55e12ec55980 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faee0cd3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e12ebb65e0 con 0x55e12ec60240 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec55440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec55440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39718/0 (socket says 172.21.15.28:39718) 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 -- 172.21.15.28:0/3742062866 learned_addr learned my addr 172.21.15.28:0/3742062866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed57fa640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7faec00a33b0 0x7faec00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed7fff640 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55980 0x55e12ec5fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faedce97640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7faec00a33b0 0x7faec00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 -- 172.21.15.28:0/3742062866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 msgr2=0x55e12ec62630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 0x55e12ec62630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 -- 172.21.15.28:0/3742062866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55980 msgr2=0x55e12ec5fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55980 0x55e12ec5fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 -- 172.21.15.28:0/3742062866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12eb52d40 con 0x55e12ec55060 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed7fff640 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55980 0x55e12ec5fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed57fa640 1 -- 172.21.15.28:0/2908032820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7faed00d0290 con 0x7faed8106f10 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.665+0000 7faed77fe640 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec55440 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7faecc02a930 tx=0x7faecc031660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187fdcf703089f2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faedce97640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7faec00a33b0 0x7faec00a5870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faed0009270 tx=0x7faed006df40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faebe7fc640 1 -- 172.21.15.28:0/3742062866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc035040 con 0x55e12ec55060 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faebe7fc640 1 -- 172.21.15.28:0/3742062866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faecc005020 con 0x55e12ec55060 2024-09-15T22:04:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faebe7fc640 1 -- 172.21.15.28:0/3742062866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc004b20 con 0x55e12ec55060 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 -- 172.21.15.28:0/3742062866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 msgr2=0x55e12ec55440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec55440 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7faecc02a930 tx=0x7faecc031660 comp rx=0 tx=0).stop 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 -- 172.21.15.28:0/3742062866 shutdown_connections 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 0x55e12ec62630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55980 0x55e12ec5fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3742062866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec55440 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 -- 172.21.15.28:0/3742062866 >> 172.21.15.28:0/3742062866 conn(0x55e12ec515f0 msgr2=0x55e12ec519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 -- 172.21.15.28:0/3742062866 shutdown_connections 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.666+0000 7faee0cd3740 1 -- 172.21.15.28:0/3742062866 wait complete. 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 -- start start 2024-09-15T22:04:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 Processor -- start 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec5d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 0x55e12ec5dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e12ec62e10 con 0x55e12ec55060 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e12eb42f10 con 0x55e12ec60240 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e12ec62f90 con 0x55e12ec55980 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faee0cd3740 10 client.0 _ll_register_callbacks cb 0x55e12eb66db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec5d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec5d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39726/0 (socket says 172.21.15.28:39726) 2024-09-15T22:04:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.667+0000 7faed77fe640 1 -- 172.21.15.28:0/3254679330 learned_addr learned my addr 172.21.15.28:0/3254679330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed77fe640 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 0x55e12ec5dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed7fff640 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed77fe640 1 -- 172.21.15.28:0/3254679330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 msgr2=0x55e12ec5d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed77fe640 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed77fe640 1 -- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 msgr2=0x55e12ec5dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed77fe640 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 0x55e12ec5dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed77fe640 1 -- 172.21.15.28:0/3254679330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12e8c9580 con 0x55e12ec55060 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faed77fe640 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec5d130 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7faecc0055c0 tx=0x7faecc0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fdd9882fb1a7763 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faeb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faeb77fe640 1 -- 172.21.15.28:0/3254679330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc004cb0 con 0x55e12ec55060 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faeb77fe640 1 -- 172.21.15.28:0/3254679330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7faecc045440 con 0x55e12ec55060 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faecc005020 con 0x55e12ec55060 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faeb77fe640 1 -- 172.21.15.28:0/3254679330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faecc004e50 con 0x55e12ec55060 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faee0cd3740 10 client.16764 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e12e993960 con 0x55e12ec55060 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.668+0000 7faeb77fe640 1 -- 172.21.15.28:0/3254679330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faecc005020 con 0x55e12ec55060 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faeb77fe640 1 -- 172.21.15.28:0/3254679330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7faecc03b7d0 con 0x55e12ec55060 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faee0cd3740 10 client.16764 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faee0cd3740 10 client.16764 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faee0cd3740 20 client.16764 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ec6fc70 0x55e12ec72060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faee0cd3740 4 client.16764 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e12ebc7b90 con 0x55e12ec6fc70 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faed7fff640 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ec6fc70 0x55e12ec72060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faed7fff640 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ec6fc70 0x55e12ec72060 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f19adf84172e54c server_cookie=4bcf639e89ee3bad in_seq=0 out_seq=0 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.669+0000 7faeb77fe640 10 client.16764 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faeb77fe640 1 -- 172.21.15.28:0/3254679330 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e12ebc7b90 con 0x55e12ec6fc70 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faeb77fe640 10 client.16764 handle_command_reply: tid=0 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 client.16764 shutdown 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 msgr2=0x55e12ec5d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec5d130 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7faecc0055c0 tx=0x7faecc0045b0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 shutdown_connections 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ec6fc70 0x55e12ec72060 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 0x55e12ec5dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.670+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3254679330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec5d130 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.671+0000 7faeb77fe640 0 client.16764 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.671+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 >> 172.21.15.28:0/3254679330 conn(0x55e12ec515f0 msgr2=0x55e12ec57320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.671+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 shutdown_connections 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.671+0000 7faee0cd3740 1 -- 172.21.15.28:0/3254679330 wait complete. 2024-09-15T22:04:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.671+0000 7faee0cd3740 20 client.16764 trim_cache size 0 max 16384 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 Processor -- start 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 -- start start 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec50640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 0x55e12ec5a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed81b11b0 con 0x55e12ec55060 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faed81b2cf0 con 0x55e12ec60240 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faee0cd3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faed81b1880 con 0x55e12ec55980 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec50640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec50640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39732/0 (socket says 172.21.15.28:39732) 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.688+0000 7faed77fe640 1 -- 172.21.15.28:0/2824115578 learned_addr learned my addr 172.21.15.28:0/2824115578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faed7fff640 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faed77fe640 1 -- 172.21.15.28:0/2824115578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 msgr2=0x55e12ec5a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faed77fe640 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faed77fe640 1 -- 172.21.15.28:0/2824115578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 msgr2=0x55e12ec5a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faed77fe640 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 0x55e12ec5a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faed77fe640 1 -- 172.21.15.28:0/2824115578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e12ec5faa0 con 0x55e12ec55060 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faed77fe640 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec50640 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7faecc031470 tx=0x7faecc005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6acc179352b87f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faebdffb640 1 -- 172.21.15.28:0/2824115578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc035030 con 0x55e12ec55060 2024-09-15T22:04:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faebdffb640 1 -- 172.21.15.28:0/2824115578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faecc04f040 con 0x55e12ec55060 2024-09-15T22:04:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.689+0000 7faebdffb640 1 -- 172.21.15.28:0/2824115578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc03d7f0 con 0x55e12ec55060 2024-09-15T22:04:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 -- 172.21.15.28:0/2824115578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 msgr2=0x55e12ec50640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec50640 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7faecc031470 tx=0x7faecc005670 comp rx=0 tx=0).stop 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 -- 172.21.15.28:0/2824115578 shutdown_connections 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec60240 0x55e12ec5a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec55980 0x55e12ec5a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 --2- 172.21.15.28:0/2824115578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55060 0x55e12ec50640 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 -- 172.21.15.28:0/2824115578 >> 172.21.15.28:0/2824115578 conn(0x55e12ec515f0 msgr2=0x55e12ec4ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 -- 172.21.15.28:0/2824115578 shutdown_connections 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 -- 172.21.15.28:0/2824115578 wait complete. 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 -- start start 2024-09-15T22:04:24.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.690+0000 7faee0cd3740 1 Processor -- start 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55060 0x55e12ec73660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55980 0x55e12ec73ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faee0cd3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 0x55e12ec74ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faee0cd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed81b2620 con 0x55e12ec55980 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faee0cd3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faed81b1f50 con 0x55e12ec55060 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faee0cd3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faed81af1b0 con 0x55e12ec60240 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faee0cd3740 10 client.0 _ll_register_callbacks cb 0x7faecc0051d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55980 0x55e12ec73ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55980 0x55e12ec73ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39736/0 (socket says 172.21.15.28:39736) 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55060 0x55e12ec73660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 -- 172.21.15.28:0/3596281936 learned_addr learned my addr 172.21.15.28:0/3596281936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed77fe640 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 0x55e12ec74ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 -- 172.21.15.28:0/3596281936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 msgr2=0x55e12ec74ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 0x55e12ec74ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 -- 172.21.15.28:0/3596281936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55060 msgr2=0x55e12ec73660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55060 0x55e12ec73660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 -- 172.21.15.28:0/3596281936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e12ec75310 con 0x55e12ec55980 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faed7fff640 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55980 0x55e12ec73ba0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x55e12ec5e770 tx=0x7faec8016430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445c1849e30dee76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.691+0000 7faeb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faeb77fe640 1 -- 172.21.15.28:0/3596281936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8025410 con 0x55e12ec55980 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faeb77fe640 1 -- 172.21.15.28:0/3596281936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7faec80255b0 con 0x55e12ec55980 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faecc04f040 con 0x55e12ec55980 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faeb77fe640 1 -- 172.21.15.28:0/3596281936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faec802e050 con 0x55e12ec55980 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 10 client.16770 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e12ecdb1d0 con 0x55e12ec55980 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faeb77fe640 1 -- 172.21.15.28:0/3596281936 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faecc04f040 con 0x55e12ec55980 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faeb77fe640 1 -- 172.21.15.28:0/3596281936 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7faec80158c0 con 0x55e12ec55980 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 10 client.16770 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 10 client.16770 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:04:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 20 client.16770 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ecdb900 0x55e12ecddd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 4 client.16770 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ebf3369a", "release": true, "await": true}] 2024-09-15T22:04:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ebf3369a", "release": true, "await": true}) -- 0x55e12ebc20a0 con 0x55e12ecdb900 2024-09-15T22:04:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.692+0000 7faed77fe640 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ecdb900 0x55e12ecddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.693+0000 7faeb77fe640 10 client.16770 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.693+0000 7faed77fe640 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ecdb900 0x55e12ecddd20 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df9ea641d2f34af2 server_cookie=5ba471ff23e016f5 in_seq=0 out_seq=0 2024-09-15T22:04:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.724+0000 7faeb77fe640 1 -- 172.21.15.28:0/3596281936 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55e12ebc20a0 con 0x55e12ecdb900 2024-09-15T22:04:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.724+0000 7faeb77fe640 10 client.16770 handle_command_reply: tid=0 2024-09-15T22:04:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 client.16770 shutdown 2024-09-15T22:04:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55980 msgr2=0x55e12ec73ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55980 0x55e12ec73ba0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x55e12ec5e770 tx=0x7faec8016430 comp rx=0 tx=0).stop 2024-09-15T22:04:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 shutdown_connections 2024-09-15T22:04:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e12ecdb900 0x55e12ecddd20 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faeb77fe640 0 client.16770 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e12ec60240 0x55e12ec74ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e12ec55980 0x55e12ec73ba0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 --2- 172.21.15.28:0/3596281936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e12ec55060 0x55e12ec73660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.725+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 >> 172.21.15.28:0/3596281936 conn(0x55e12ec515f0 msgr2=0x55e12ec50750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:24.732 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 32, 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 13, 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "ebf3369a": { 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 13, 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:04:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 90.0, 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 139.6, 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:04:24.735 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:24.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.726+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 shutdown_connections 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.726+0000 7faee0cd3740 1 -- 172.21.15.28:0/3596281936 wait complete. 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.726+0000 7faee0cd3740 20 client.16770 trim_cache size 0 max 16384 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7faec00a33b0 msgr2=0x7faec00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7faec00a33b0 0x7faec00a5870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faed0009270 tx=0x7faed006df40 comp rx=0 tx=0).stop 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8106f10 msgr2=0x7faed8074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8106f10 0x7faed8074f70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7faed000eaa0 tx=0x7faed000eea0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 shutdown_connections 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7faec00a33b0 0x7faec00a5870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8106f10 0x7faed8074f70 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faed8102790 0x7faed8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 --2- 172.21.15.28:0/2908032820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faed8101de0 0x7faed80744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:24.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 >> 172.21.15.28:0/2908032820 conn(0x7faed80fd8c0 msgr2=0x7faed80fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:24.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 shutdown_connections 2024-09-15T22:04:24.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:24.729+0000 7faede8fa640 1 -- 172.21.15.28:0/2908032820 wait complete. 2024-09-15T22:04:25.801 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ebf3369a', total seconds elapsed: 178.7 2024-09-15T22:04:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:25 smithi028 ceph-mon[30969]: pgmap v217: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 51 KiB/s rd, 6.3 KiB/s wr, 53 op/s 2024-09-15T22:04:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:25 smithi174 ceph-mon[35244]: pgmap v217: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 51 KiB/s rd, 6.3 KiB/s wr, 53 op/s 2024-09-15T22:04:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:25 smithi203 ceph-mon[35546]: pgmap v217: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 51 KiB/s rd, 6.3 KiB/s wr, 53 op/s 2024-09-15T22:04:28.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:27 smithi174 ceph-mon[35244]: pgmap v218: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 48 KiB/s rd, 178 KiB/s wr, 91 op/s 2024-09-15T22:04:28.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:27 smithi028 ceph-mon[30969]: pgmap v218: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 48 KiB/s rd, 178 KiB/s wr, 91 op/s 2024-09-15T22:04:28.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:27 smithi203 ceph-mon[35546]: pgmap v218: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 48 KiB/s rd, 178 KiB/s wr, 91 op/s 2024-09-15T22:04:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:28 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:04:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:28 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:04:29.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:28 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:04:29.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:28 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:04:29.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:28 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:04:29.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:28 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:04:29.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:28 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:04:29.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:28 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:04:29.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:28 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:04:30.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:29 smithi028 ceph-mon[30969]: pgmap v219: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 2.8 MiB/s wr, 715 op/s 2024-09-15T22:04:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:29 smithi174 ceph-mon[35244]: pgmap v219: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 2.8 MiB/s wr, 715 op/s 2024-09-15T22:04:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:29 smithi203 ceph-mon[35546]: pgmap v219: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 44 KiB/s rd, 2.8 MiB/s wr, 715 op/s 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:Syncing()...5 sec 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:FFSB benchmark finished at: Sun Sep 15 22:04:30 2024 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:Results: 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:Benchmark took 316.39 sec 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:Total Results 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout:=============== 2024-09-15T22:04:30.638 INFO:tasks.workunit.client.0.smithi028.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout: ======= ============ ========= ======= =========== ========== 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout: write : 3709440 11724.36 100.000% 100.000% 45.8MB/sec 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:- 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:11724.36 Transactions per Second 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:Throughput Results 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:=================== 2024-09-15T22:04:30.639 INFO:tasks.workunit.client.0.smithi028.stdout:Write Throughput: 45.8MB/sec 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout:System Call Latency statistics in millisecs 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout:===== 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout: Min Avg Max Total Calls 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout: ======== ======== ======== ============ 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout:[ open] 0.005000 0.558577 76.213997 2898 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-15T22:04:30.659 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 1 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 18 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 40 2024-09-15T22:04:30.660 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 44 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 27 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 10 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 8 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 5 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 15 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 532 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 1669 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 528 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout:[ write] 0.001000 2.675380 143609.031250 3709440 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 52 2024-09-15T22:04:30.661 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 103 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 79 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 99 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 1110 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 3693 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 363 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 954 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 794 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 702 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 843 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 522 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 485 2024-09-15T22:04:30.662 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 431 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 493 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 1345 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 6346 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 19664 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 3671236 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout:[ lseek] 0.000000 0.001755 81.886002 3709440 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-15T22:04:30.663 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 5 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 73 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 75 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 57 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 66 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 43 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 52 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 53 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 51 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 85 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 500 2024-09-15T22:04:30.664 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 826 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 3707554 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout:[ close] 0.001000 0.003894 0.069000 2898 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-15T22:04:30.665 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[15] 0.050000 - 0.100000 : 1 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[16] 0.020000 - 0.050000 : 2 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[17] 0.010000 - 0.020000 : 9 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: msec_range[18] 0.000000 - 0.010000 : 2886 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.666 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.667 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-15T22:04:30.667 INFO:tasks.workunit.client.0.smithi028.stdout:1.1% User Time 2024-09-15T22:04:30.667 INFO:tasks.workunit.client.0.smithi028.stdout:4.1% System Time 2024-09-15T22:04:30.667 INFO:tasks.workunit.client.0.smithi028.stdout:5.2% CPU Utilization 2024-09-15T22:04:30.714 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd .. 2024-09-15T22:04:30.718 INFO:tasks.workunit.client.0.smithi028.stderr:+ rm -r tmp ffsb 2024-09-15T22:04:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:31 smithi174 ceph-mon[35244]: pgmap v220: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 42 KiB/s rd, 5.4 MiB/s wr, 1.32k op/s 2024-09-15T22:04:32.311 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T22:04:32.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T22:04:32.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:31 smithi028 ceph-mon[30969]: pgmap v220: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 42 KiB/s rd, 5.4 MiB/s wr, 1.32k op/s 2024-09-15T22:04:32.383 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-09-15T22:04:32.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T22:04:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:31 smithi203 ceph-mon[35546]: pgmap v220: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 42 KiB/s rd, 5.4 MiB/s wr, 1.32k op/s 2024-09-15T22:04:33.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 -- 172.21.15.28:0/392238587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301091f0 msgr2=0x7f1d30110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 --2- 172.21.15.28:0/392238587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301091f0 0x7f1d30110250 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009900 tx=0x7f1d2002a440 comp rx=0 tx=0).stop 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 -- 172.21.15.28:0/392238587 shutdown_connections 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 --2- 172.21.15.28:0/392238587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 0x7f1d3011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 --2- 172.21.15.28:0/392238587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d301095d0 0x7f1d30109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 --2- 172.21.15.28:0/392238587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301091f0 0x7f1d30110250 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 -- 172.21.15.28:0/392238587 >> 172.21.15.28:0/392238587 conn(0x7f1d3006cc60 msgr2=0x7f1d3006d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 -- 172.21.15.28:0/392238587 shutdown_connections 2024-09-15T22:04:33.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.171+0000 7f1d35f11640 1 -- 172.21.15.28:0/392238587 wait complete. 2024-09-15T22:04:33.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.172+0000 7f1d35f11640 1 Processor -- start 2024-09-15T22:04:33.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.174+0000 7f1d35f11640 1 -- start start 2024-09-15T22:04:33.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.174+0000 7f1d35f11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d301091f0 0x7f1d301a70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.174+0000 7f1d35f11640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301095d0 0x7f1d301a7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.177 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.174+0000 7f1d35f11640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 0x7f1d301ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.174+0000 7f1d35f11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30120b60 con 0x7f1d301091f0 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.174+0000 7f1d35f11640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d301209e0 con 0x7f1d3011b140 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.174+0000 7f1d35f11640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30120ce0 con 0x7f1d301095d0 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 0x7f1d301ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 0x7f1d301ab9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:37886/0 (socket says 172.21.15.28:37886) 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 -- 172.21.15.28:0/787108130 learned_addr learned my addr 172.21.15.28:0/787108130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2effd640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301095d0 0x7f1d301a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 -- 172.21.15.28:0/787108130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301095d0 msgr2=0x7f1d301a7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301095d0 0x7f1d301a7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 -- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d301091f0 msgr2=0x7f1d301a70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d301091f0 0x7f1d301a70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.175+0000 7f1d2ffff640 1 -- 172.21.15.28:0/787108130 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d301ac0e0 con 0x7f1d3011b140 2024-09-15T22:04:33.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.176+0000 7f1d2ffff640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 0x7f1d301ab9b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e9d0 tx=0x7f1d2800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.178+0000 7f1d2cff9640 1 -- 172.21.15.28:0/787108130 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2800cd80 con 0x7f1d3011b140 2024-09-15T22:04:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.178+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d301ac3d0 con 0x7f1d3011b140 2024-09-15T22:04:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.178+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d301b3c60 con 0x7f1d3011b140 2024-09-15T22:04:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.178+0000 7f1d2cff9640 1 -- 172.21.15.28:0/787108130 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d28005020 con 0x7f1d3011b140 2024-09-15T22:04:33.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.178+0000 7f1d2cff9640 1 -- 172.21.15.28:0/787108130 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d280116d0 con 0x7f1d3011b140 2024-09-15T22:04:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.184+0000 7f1d38ca5740 1 Processor -- start 2024-09-15T22:04:33.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.185+0000 7f1d38ca5740 1 -- start start 2024-09-15T22:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.186+0000 7f1d2cff9640 1 -- 172.21.15.28:0/787108130 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f1d28004a40 con 0x7f1d3011b140 2024-09-15T22:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.186+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e7410a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.186+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.186+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e74e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.186+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56381e6a52f0 con 0x56381e74be80 2024-09-15T22:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.186+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56381e64d270 con 0x56381e7415e0 2024-09-15T22:04:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.186+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56381e6a12b0 con 0x56381e740cc0 2024-09-15T22:04:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.187+0000 7f1d2cff9640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1d180a35c0 0x7f1d180a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.187+0000 7f1d2cff9640 1 -- 172.21.15.28:0/787108130 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f1d280d0210 con 0x7f1d3011b140 2024-09-15T22:04:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.187+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.187+0000 7f1d2effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e7410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.187+0000 7f1d2effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e7410a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:41432/0 (socket says 172.21.15.28:41432) 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.187+0000 7f1d2effd640 1 -- 172.21.15.28:0/2686417839 learned_addr learned my addr 172.21.15.28:0/2686417839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.188+0000 7f1d2effd640 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e74e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.188+0000 7f1d2effd640 1 -- 172.21.15.28:0/2686417839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 msgr2=0x56381e74b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.188+0000 7f1d2effd640 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.188+0000 7f1d2effd640 1 -- 172.21.15.28:0/2686417839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 msgr2=0x56381e74e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.188+0000 7f1d2effd640 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e74e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.188+0000 7f1d2effd640 1 -- 172.21.15.28:0/2686417839 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56381e64c340 con 0x56381e740cc0 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.189+0000 7f1d2effd640 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e7410a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d24012780 tx=0x7f1d240096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2caa7238a1b89ef2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.189+0000 7f1d2ffff640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1d180a35c0 0x7f1d180a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.189+0000 7f1d15ffb640 1 -- 172.21.15.28:0/2686417839 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24004030 con 0x56381e740cc0 2024-09-15T22:04:33.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.189+0000 7f1d15ffb640 1 -- 172.21.15.28:0/2686417839 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d24010cb0 con 0x56381e740cc0 2024-09-15T22:04:33.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.190+0000 7f1d15ffb640 1 -- 172.21.15.28:0/2686417839 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d240039d0 con 0x56381e740cc0 2024-09-15T22:04:33.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.191+0000 7f1d38ca5740 1 -- 172.21.15.28:0/2686417839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 msgr2=0x56381e7410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.191+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e7410a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d24012780 tx=0x7f1d240096e0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.192+0000 7f1d2ffff640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1d180a35c0 0x7f1d180a5a80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d280cd5b0 tx=0x7f1d2800c7f0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 -- 172.21.15.28:0/2686417839 shutdown_connections 2024-09-15T22:04:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e74e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/2686417839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e7410a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 -- 172.21.15.28:0/2686417839 >> 172.21.15.28:0/2686417839 conn(0x56381e73d230 msgr2=0x56381e73d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 -- 172.21.15.28:0/2686417839 shutdown_connections 2024-09-15T22:04:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 -- 172.21.15.28:0/2686417839 wait complete. 2024-09-15T22:04:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.194+0000 7f1d38ca5740 1 -- start start 2024-09-15T22:04:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 1 Processor -- start 2024-09-15T22:04:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e748dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e749310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e749850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56381e74ea50 con 0x56381e74be80 2024-09-15T22:04:33.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56381e62e290 con 0x56381e7415e0 2024-09-15T22:04:33.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56381e74ebd0 con 0x56381e740cc0 2024-09-15T22:04:33.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.196+0000 7f1d38ca5740 10 client.0 _ll_register_callbacks cb 0x56381e65f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.197+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e749310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.197+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e749310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:37898/0 (socket says 172.21.15.28:37898) 2024-09-15T22:04:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.197+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1119405779 learned_addr learned my addr 172.21.15.28:0/1119405779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.197+0000 7f1d2effd640 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e748dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.198+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1119405779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 msgr2=0x56381e748dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.198+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e748dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.198+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1119405779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 msgr2=0x56381e749850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.198+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e749850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.198+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1119405779 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56381e3b8210 con 0x56381e7415e0 2024-09-15T22:04:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.198+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e749310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009900 tx=0x7f1d20009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285be1142ebcf87a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.198+0000 7f1d0effd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:04:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.199+0000 7f1d0effd640 1 -- 172.21.15.28:0/1119405779 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20004a00 con 0x56381e7415e0 2024-09-15T22:04:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.199+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d24010cb0 con 0x56381e7415e0 2024-09-15T22:04:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.199+0000 7f1d0effd640 1 -- 172.21.15.28:0/1119405779 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f1d20005000 con 0x56381e7415e0 2024-09-15T22:04:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.199+0000 7f1d0effd640 1 -- 172.21.15.28:0/1119405779 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1d20005200 con 0x56381e7415e0 2024-09-15T22:04:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.200+0000 7f1d38ca5740 10 client.26267 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:33.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.200+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56381e484620 con 0x56381e7415e0 2024-09-15T22:04:33.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d0effd640 1 -- 172.21.15.28:0/1119405779 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d24010cb0 con 0x56381e7415e0 2024-09-15T22:04:33.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d0effd640 1 -- 172.21.15.28:0/1119405779 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1d20046430 con 0x56381e7415e0 2024-09-15T22:04:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d38ca5740 10 client.26267 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d38ca5740 10 client.26267 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d38ca5740 20 client.26267 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e75c3f0 0x56381e75e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d38ca5740 4 client.26267 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56381e6b2a20 con 0x56381e75c3f0 2024-09-15T22:04:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.202+0000 7f1d2effd640 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e75c3f0 0x56381e75e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.209+0000 7f1d2effd640 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e75c3f0 0x56381e75e810 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8ee13d748fe1c9 server_cookie=801d6c2596dac24c in_seq=0 out_seq=0 2024-09-15T22:04:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.209+0000 7f1d0effd640 10 client.26267 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.210+0000 7f1d0effd640 1 -- 172.21.15.28:0/1119405779 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56381e6b2a20 con 0x56381e75c3f0 2024-09-15T22:04:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.210+0000 7f1d0effd640 10 client.26267 handle_command_reply: tid=0 2024-09-15T22:04:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.210+0000 7f1d38ca5740 1 client.26267 shutdown 2024-09-15T22:04:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.211+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 msgr2=0x56381e749310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.211+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e749310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009900 tx=0x7f1d20009630 comp rx=0 tx=0).stop 2024-09-15T22:04:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 shutdown_connections 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e75c3f0 0x56381e75e810 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e749850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e749310 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1119405779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e748dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 >> 172.21.15.28:0/1119405779 conn(0x56381e73d230 msgr2=0x56381e742f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d0effd640 0 client.26267 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 shutdown_connections 2024-09-15T22:04:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.215+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1119405779 wait complete. 2024-09-15T22:04:33.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.219+0000 7f1d38ca5740 20 client.26267 trim_cache size 0 max 16384 2024-09-15T22:04:33.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.247+0000 7f1d38ca5740 1 Processor -- start 2024-09-15T22:04:33.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.247+0000 7f1d38ca5740 1 -- start start 2024-09-15T22:04:33.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e740cc0 0x56381e749680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e7415e0 0x56381e7640e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e74be80 0x56381e764620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c010020 con 0x56381e7415e0 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c011b60 con 0x56381e740cc0 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c0106f0 con 0x56381e74be80 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e7415e0 0x56381e7640e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e7415e0 0x56381e7640e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59606/0 (socket says 172.21.15.28:59606) 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1090555032 learned_addr learned my addr 172.21.15.28:0/1090555032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2effd640 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e740cc0 0x56381e749680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2effd640 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e74be80 0x56381e764620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1090555032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e74be80 msgr2=0x56381e764620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e74be80 0x56381e764620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1090555032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e740cc0 msgr2=0x56381e749680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e740cc0 0x56381e749680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/1090555032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56381e758f00 con 0x56381e7415e0 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.248+0000 7f1d2effd640 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e740cc0 0x56381e749680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.249+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e7415e0 0x56381e7640e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1d20006010 tx=0x7f1d2002a940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c7fce7d03d4a1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.249+0000 7f1d157fa640 1 -- 172.21.15.28:0/1090555032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20046430 con 0x56381e7415e0 2024-09-15T22:04:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.249+0000 7f1d157fa640 1 -- 172.21.15.28:0/1090555032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d200465d0 con 0x56381e7415e0 2024-09-15T22:04:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.249+0000 7f1d157fa640 1 -- 172.21.15.28:0/1090555032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20005690 con 0x56381e7415e0 2024-09-15T22:04:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.250+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1090555032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e7415e0 msgr2=0x56381e7640e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.250+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e7415e0 0x56381e7640e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1d20006010 tx=0x7f1d2002a940 comp rx=0 tx=0).stop 2024-09-15T22:04:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.250+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1090555032 shutdown_connections 2024-09-15T22:04:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.250+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e74be80 0x56381e764620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.251+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e7415e0 0x56381e7640e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.251+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/1090555032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e740cc0 0x56381e749680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.251+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1090555032 >> 172.21.15.28:0/1090555032 conn(0x56381e73d230 msgr2=0x56381e747680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.253+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1090555032 shutdown_connections 2024-09-15T22:04:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.253+0000 7f1d38ca5740 1 -- 172.21.15.28:0/1090555032 wait complete. 2024-09-15T22:04:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.254+0000 7f1d38ca5740 1 -- start start 2024-09-15T22:04:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.254+0000 7f1d38ca5740 1 Processor -- start 2024-09-15T22:04:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.255+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e74eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.255+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.255+0000 7f1d38ca5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e74f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.255+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c011490 con 0x56381e74be80 2024-09-15T22:04:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.255+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c010dc0 con 0x56381e7415e0 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.255+0000 7f1d38ca5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c00e020 con 0x56381e740cc0 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d38ca5740 10 client.0 _ll_register_callbacks cb 0x7f1d20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:37910/0 (socket says 172.21.15.28:37910) 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/3371913227 learned_addr learned my addr 172.21.15.28:0/3371913227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2effd640 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e74eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/3371913227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 msgr2=0x56381e74eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e74eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/3371913227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 msgr2=0x56381e74f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e74f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 -- 172.21.15.28:0/3371913227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56381e7c5250 con 0x56381e7415e0 2024-09-15T22:04:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.256+0000 7f1d2f7fe640 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74f3f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d20046a10 tx=0x7f1d20046ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f46534086ee530c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.260+0000 7f1d0effd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:04:33.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.260+0000 7f1d0effd640 1 -- 172.21.15.28:0/3371913227 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d20062070 con 0x56381e7415e0 2024-09-15T22:04:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.260+0000 7f1d0effd640 1 -- 172.21.15.28:0/3371913227 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f1d2002acc0 con 0x56381e7415e0 2024-09-15T22:04:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.261+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d200465d0 con 0x56381e7415e0 2024-09-15T22:04:33.263 DEBUG:teuthology.parallel:result is None 2024-09-15T22:04:33.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T22:04:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.267+0000 7f1d0effd640 1 -- 172.21.15.28:0/3371913227 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1d2006c050 con 0x56381e7415e0 2024-09-15T22:04:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.267+0000 7f1d38ca5740 10 client.26273 fetch_fsmap learned FSMap version 32 2024-09-15T22:04:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.267+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56381e7c6500 con 0x56381e7415e0 2024-09-15T22:04:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.270+0000 7f1d0effd640 1 -- 172.21.15.28:0/3371913227 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d20004a10 con 0x56381e7415e0 2024-09-15T22:04:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.271+0000 7f1d0effd640 1 -- 172.21.15.28:0/3371913227 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1d20031b20 con 0x56381e7415e0 2024-09-15T22:04:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.271+0000 7f1d38ca5740 10 client.26273 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T22:04:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.271+0000 7f1d38ca5740 10 client.26273 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.271+0000 7f1d38ca5740 20 client.26273 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:33.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.271+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e7c6a30 0x56381e7c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.271+0000 7f1d38ca5740 4 client.26273 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:04:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.271+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f1d200465d0 con 0x56381e7c6a30 2024-09-15T22:04:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.272+0000 7f1d2effd640 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e7c6a30 0x56381e7c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.281+0000 7f1d2effd640 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e7c6a30 0x56381e7c8e50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d780545776449555 server_cookie=6fb922ee8865df49 in_seq=0 out_seq=0 2024-09-15T22:04:33.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.281+0000 7f1d0effd640 10 client.26273 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:33.404 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T22:04:33.404 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-15T22:04:33.416 INFO:tasks.quiescer:joining Quiescers 2024-09-15T22:04:33.416 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-15T22:04:33.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:04:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.437+0000 7f1d0effd640 1 -- 172.21.15.28:0/3371913227 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f1d200465d0 con 0x56381e7c6a30 2024-09-15T22:04:33.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.437+0000 7f1d0effd640 10 client.26273 handle_command_reply: tid=0 2024-09-15T22:04:33.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.437+0000 7f1d38ca5740 1 client.26273 shutdown 2024-09-15T22:04:33.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 msgr2=0x56381e74f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74f3f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d20046a10 tx=0x7f1d20046ed0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 shutdown_connections 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56381e7c6a30 0x56381e7c8e50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56381e74be80 0x56381e74f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56381e7415e0 0x56381e74f3f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 --2- 172.21.15.28:0/3371913227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56381e740cc0 0x56381e74eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 >> 172.21.15.28:0/3371913227 conn(0x56381e73d230 msgr2=0x56381e73ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d0effd640 0 client.26273 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 shutdown_connections 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.438+0000 7f1d38ca5740 1 -- 172.21.15.28:0/3371913227 wait complete. 2024-09-15T22:04:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.439+0000 7f1d38ca5740 20 client.26273 trim_cache size 0 max 16384 2024-09-15T22:04:33.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:04:33.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:04:33.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:04:33.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:04:33.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.439+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1d180a35c0 msgr2=0x7f1d180a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1d180a35c0 0x7f1d180a5a80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d280cd5b0 tx=0x7f1d2800c7f0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 msgr2=0x7f1d301ab9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 0x7f1d301ab9b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800e9d0 tx=0x7f1d2800ee90 comp rx=0 tx=0).stop 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 shutdown_connections 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1d180a35c0 0x7f1d180a5a80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3011b140 0x7f1d301ab9b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d301095d0 0x7f1d301a7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 --2- 172.21.15.28:0/787108130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d301091f0 0x7f1d301a70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 >> 172.21.15.28:0/787108130 conn(0x7f1d3006cc60 msgr2=0x7f1d3011b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 shutdown_connections 2024-09-15T22:04:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:33.440+0000 7f1d35f11640 1 -- 172.21.15.28:0/787108130 wait complete. 2024-09-15T22:04:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:34 smithi174 ceph-mon[35244]: pgmap v221: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1018 GiB / 1.0 TiB avail; 190 KiB/s rd, 9.3 MiB/s wr, 2.19k op/s 2024-09-15T22:04:34.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:34 smithi028 ceph-mon[30969]: pgmap v221: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1018 GiB / 1.0 TiB avail; 190 KiB/s rd, 9.3 MiB/s wr, 2.19k op/s 2024-09-15T22:04:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:34 smithi203 ceph-mon[35546]: pgmap v221: 129 pgs: 129 active+clean; 19 GiB data, 54 GiB used, 1018 GiB / 1.0 TiB avail; 190 KiB/s rd, 9.3 MiB/s wr, 2.19k op/s 2024-09-15T22:04:35.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:35 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:04:35.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:35 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:04:35.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:35 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:04:36.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:36 smithi028 ceph-mon[30969]: pgmap v222: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 160 KiB/s rd, 9.4 MiB/s wr, 2.21k op/s 2024-09-15T22:04:36.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:36 smithi203 ceph-mon[35546]: pgmap v222: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 160 KiB/s rd, 9.4 MiB/s wr, 2.21k op/s 2024-09-15T22:04:36.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:36 smithi174 ceph-mon[35244]: pgmap v222: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 160 KiB/s rd, 9.4 MiB/s wr, 2.21k op/s 2024-09-15T22:04:38.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:38 smithi028 ceph-mon[30969]: pgmap v223: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 160 KiB/s rd, 9.4 MiB/s wr, 2.22k op/s 2024-09-15T22:04:38.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:38 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:04:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:38 smithi203 ceph-mon[35546]: pgmap v223: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 160 KiB/s rd, 9.4 MiB/s wr, 2.22k op/s 2024-09-15T22:04:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:38 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:04:38.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:38 smithi174 ceph-mon[35244]: pgmap v223: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 160 KiB/s rd, 9.4 MiB/s wr, 2.22k op/s 2024-09-15T22:04:38.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:38 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:04:40.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:40 smithi028 ceph-mon[30969]: pgmap v224: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 317 KiB/s rd, 9.3 MiB/s wr, 2.30k op/s 2024-09-15T22:04:40.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:40 smithi203 ceph-mon[35546]: pgmap v224: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 317 KiB/s rd, 9.3 MiB/s wr, 2.30k op/s 2024-09-15T22:04:40.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:40 smithi174 ceph-mon[35244]: pgmap v224: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 317 KiB/s rd, 9.3 MiB/s wr, 2.30k op/s 2024-09-15T22:04:41.115 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:04:41.116 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"btime":"2024-09-15T22:04:37:022811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:04:36.713632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:04:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 -- 172.21.15.28:0/2457000779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 msgr2=0x7f29941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 --2- 172.21.15.28:0/2457000779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f29941067e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f297c009900 tx=0x7f297c02a410 comp rx=0 tx=0).stop 2024-09-15T22:04:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 -- 172.21.15.28:0/2457000779 shutdown_connections 2024-09-15T22:04:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 --2- 172.21.15.28:0/2457000779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2994106f10 0x7f299410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 --2- 172.21.15.28:0/2457000779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f29941067e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 --2- 172.21.15.28:0/2457000779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2994101de0 0x7f29941021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 -- 172.21.15.28:0/2457000779 >> 172.21.15.28:0/2457000779 conn(0x7f29940fd8c0 msgr2=0x7f29940ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 -- 172.21.15.28:0/2457000779 shutdown_connections 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 -- 172.21.15.28:0/2457000779 wait complete. 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.911+0000 7f299a82f640 1 Processor -- start 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f299a82f640 1 -- start start 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f299a82f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2994101de0 0x7f2994198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f299a82f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f2994198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f299a82f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2994106f10 0x7f299419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f299a82f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2994111fc0 con 0x7f2994102790 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f299a82f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2994111e40 con 0x7f2994106f10 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f299a82f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2994112140 con 0x7f2994101de0 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f2994198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f2994198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59630/0 (socket says 172.21.15.28:59630) 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 -- 172.21.15.28:0/1441537317 learned_addr learned my addr 172.21.15.28:0/1441537317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f2998dcc640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2994106f10 0x7f299419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f2993fff640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2994101de0 0x7f2994198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 -- 172.21.15.28:0/1441537317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2994101de0 msgr2=0x7f2994198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2994101de0 0x7f2994198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 -- 172.21.15.28:0/1441537317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2994106f10 msgr2=0x7f299419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2994106f10 0x7f299419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 -- 172.21.15.28:0/1441537317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299419d3d0 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f2998dcc640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2994106f10 0x7f299419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.912+0000 7f298bfff640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f2994198960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f297c004770 tx=0x7f297c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.913+0000 7f2991ffb640 1 -- 172.21.15.28:0/1441537317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c03f070 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.913+0000 7f2991ffb640 1 -- 172.21.15.28:0/1441537317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f297c031ca0 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.913+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299419d660 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.913+0000 7f2991ffb640 1 -- 172.21.15.28:0/1441537317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c035030 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.913+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29941a4fa0 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.914+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29941032c0 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.919+0000 7f2991ffb640 1 -- 172.21.15.28:0/1441537317 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f297c042070 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.931+0000 7f2991ffb640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f29800a33b0 0x7f29800a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.932+0000 7f2991ffb640 1 -- 172.21.15.28:0/1441537317 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f297c0f0a00 con 0x7f2994102790 2024-09-15T22:04:41.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.932+0000 7f2993fff640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f29800a33b0 0x7f29800a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.939+0000 7f2993fff640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f29800a33b0 0x7f29800a5870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2984005fd0 tx=0x7f298400b590 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:39.942+0000 7f2991ffb640 1 -- 172.21.15.28:0/1441537317 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f297c0bcfd0 con 0x7f2994102790 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.276+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f299410a150 con 0x7f2994102790 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.277+0000 7f2991ffb640 1 -- 172.21.15.28:0/1441537317 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11988 (secure 0 0 0) 0x7f297c0bc8d0 con 0x7f2994102790 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f29800a33b0 msgr2=0x7f29800a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f29800a33b0 0x7f29800a5870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2984005fd0 tx=0x7f298400b590 comp rx=0 tx=0).stop 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 msgr2=0x7f2994198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f2994198960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f297c004770 tx=0x7f297c003c10 comp rx=0 tx=0).stop 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 shutdown_connections 2024-09-15T22:04:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f29800a33b0 0x7f29800a5870 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2994106f10 0x7f299419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2994102790 0x7f2994198960 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 --2- 172.21.15.28:0/1441537317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2994101de0 0x7f2994198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 >> 172.21.15.28:0/1441537317 conn(0x7f29940fd8c0 msgr2=0x7f29940fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 shutdown_connections 2024-09-15T22:04:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:40.280+0000 7f299a82f640 1 -- 172.21.15.28:0/1441537317 wait complete. 2024-09-15T22:04:41.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:41 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1441537317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:41.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:41 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1441537317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:41.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:41 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1441537317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:04:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:42 smithi028 ceph-mon[30969]: pgmap v225: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 315 KiB/s rd, 6.7 MiB/s wr, 1.69k op/s 2024-09-15T22:04:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:42 smithi203 ceph-mon[35546]: pgmap v225: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 315 KiB/s rd, 6.7 MiB/s wr, 1.69k op/s 2024-09-15T22:04:42.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:42 smithi174 ceph-mon[35244]: pgmap v225: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 315 KiB/s rd, 6.7 MiB/s wr, 1.69k op/s 2024-09-15T22:04:44.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:44 smithi028 ceph-mon[30969]: pgmap v226: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 323 KiB/s rd, 4.1 MiB/s wr, 1.14k op/s 2024-09-15T22:04:44.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:44 smithi203 ceph-mon[35546]: pgmap v226: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 323 KiB/s rd, 4.1 MiB/s wr, 1.14k op/s 2024-09-15T22:04:44.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:44 smithi174 ceph-mon[35244]: pgmap v226: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 323 KiB/s rd, 4.1 MiB/s wr, 1.14k op/s 2024-09-15T22:04:45.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:45 smithi028 ceph-mon[30969]: pgmap v227: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 166 KiB/s rd, 223 KiB/s wr, 276 op/s 2024-09-15T22:04:45.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:45 smithi203 ceph-mon[35546]: pgmap v227: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 166 KiB/s rd, 223 KiB/s wr, 276 op/s 2024-09-15T22:04:45.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:45 smithi174 ceph-mon[35244]: pgmap v227: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 166 KiB/s rd, 223 KiB/s wr, 276 op/s 2024-09-15T22:04:46.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 -- 172.21.15.28:0/2845519330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd01008d0 msgr2=0x7f3fd0100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 --2- 172.21.15.28:0/2845519330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd01008d0 0x7f3fd0100cb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009900 tx=0x7f3fc002a410 comp rx=0 tx=0).stop 2024-09-15T22:04:46.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 -- 172.21.15.28:0/2845519330 shutdown_connections 2024-09-15T22:04:46.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 --2- 172.21.15.28:0/2845519330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd010e480 0x7f3fd0110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 --2- 172.21.15.28:0/2845519330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fd0101280 0x7f3fd010deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 --2- 172.21.15.28:0/2845519330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd01008d0 0x7f3fd0100cb0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 -- 172.21.15.28:0/2845519330 >> 172.21.15.28:0/2845519330 conn(0x7f3fd00fc6b0 msgr2=0x7f3fd00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 -- 172.21.15.28:0/2845519330 shutdown_connections 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.553+0000 7f3fd5642640 1 -- 172.21.15.28:0/2845519330 wait complete. 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 Processor -- start 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 -- start start 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd01008d0 0x7f3fd019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd0101280 0x7f3fd019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fd010e480 0x7f3fd019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd01140e0 con 0x7f3fd0101280 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd0113f60 con 0x7f3fd01008d0 2024-09-15T22:04:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fd5642640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd0114260 con 0x7f3fd010e480 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fc7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd0101280 0x7f3fd019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fc7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd0101280 0x7f3fd019aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35774/0 (socket says 172.21.15.28:35774) 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fc7fff640 1 -- 172.21.15.28:0/1242041346 learned_addr learned my addr 172.21.15.28:0/1242041346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.554+0000 7f3fceffd640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd01008d0 0x7f3fd019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fd010e480 0x7f3fd019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fc7fff640 1 -- 172.21.15.28:0/1242041346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fd010e480 msgr2=0x7f3fd019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fc7fff640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fd010e480 0x7f3fd019ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fc7fff640 1 -- 172.21.15.28:0/1242041346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd01008d0 msgr2=0x7f3fd019a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fc7fff640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd01008d0 0x7f3fd019a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fc7fff640 1 -- 172.21.15.28:0/1242041346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd019f5a0 con 0x7f3fd0101280 2024-09-15T22:04:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fc7fff640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd0101280 0x7f3fd019aaf0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800d950 tx=0x7f3fb800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fccff9640 1 -- 172.21.15.28:0/1242041346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8017030 con 0x7f3fd0101280 2024-09-15T22:04:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fccff9640 1 -- 172.21.15.28:0/1242041346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fb8004020 con 0x7f3fd0101280 2024-09-15T22:04:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd019f890 con 0x7f3fd0101280 2024-09-15T22:04:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fccff9640 1 -- 172.21.15.28:0/1242041346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80047a0 con 0x7f3fd0101280 2024-09-15T22:04:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.555+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd01a7120 con 0x7f3fd0101280 2024-09-15T22:04:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.557+0000 7f3fccff9640 1 -- 172.21.15.28:0/1242041346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f3fb80041d0 con 0x7f3fd0101280 2024-09-15T22:04:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 Processor -- start 2024-09-15T22:04:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 -- start start 2024-09-15T22:04:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd46294ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.566 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd4629fcb0 0x55dd462a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd461f92f0 con 0x55dd46294af0 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd461a1270 con 0x55dd4629fcb0 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dd461f52b0 con 0x55dd46295410 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd46294ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd46294ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35778/0 (socket says 172.21.15.28:35778) 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fc7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd4629fcb0 0x55dd462a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/1216277614 learned_addr learned my addr 172.21.15.28:0/1216277614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fccff9640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3fbc0a34e0 0x7f3fbc0a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.558+0000 7f3fceffd640 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/1216277614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 msgr2=0x55dd4629f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/1216277614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd4629fcb0 msgr2=0x55dd462a20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd4629fcb0 0x55dd462a20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/1216277614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd461a0340 con 0x55dd46294af0 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fccff9640 1 -- 172.21.15.28:0/1242041346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f3fb80cf980 con 0x7f3fd0101280 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3fbc0a34e0 0x7f3fbc0a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd46294ed0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8010470 tx=0x7f3fc8010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f31ad4e100bb641 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fc57fa640 1 -- 172.21.15.28:0/1216277614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc801d400 con 0x55dd46294af0 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fc57fa640 1 -- 172.21.15.28:0/1216277614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fc801da80 con 0x55dd46294af0 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.559+0000 7f3fc57fa640 1 -- 172.21.15.28:0/1216277614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc801c820 con 0x55dd46294af0 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 -- 172.21.15.28:0/1216277614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 msgr2=0x55dd46294ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3fbc0a34e0 0x7f3fbc0a59a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8021000 tx=0x7f3fc801e000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd46294ed0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8010470 tx=0x7f3fc8010930 comp rx=0 tx=0).stop 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 -- 172.21.15.28:0/1216277614 shutdown_connections 2024-09-15T22:04:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd4629fcb0 0x55dd462a20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/1216277614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd46294ed0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 -- 172.21.15.28:0/1216277614 >> 172.21.15.28:0/1216277614 conn(0x55dd46291060 msgr2=0x55dd46291440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 -- 172.21.15.28:0/1216277614 shutdown_connections 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 -- 172.21.15.28:0/1216277614 wait complete. 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 -- start start 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.560+0000 7f3fd83d3740 1 Processor -- start 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd4629cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd462a2880 con 0x55dd4629fcb0 2024-09-15T22:04:46.569 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dd46182290 con 0x55dd46294af0 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dd462a2a00 con 0x55dd46295410 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fd83d3740 10 client.0 _ll_register_callbacks cb 0x55dd461b3140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fceffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd4629cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fc7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fc7fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:55704/0 (socket says 172.21.15.28:55704) 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fceffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd4629cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:39292/0 (socket says 172.21.15.28:39292) 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35784/0 (socket says 172.21.15.28:35784) 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fc7fff640 1 -- 172.21.15.28:0/2348096667 learned_addr learned my addr 172.21.15.28:0/2348096667 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/2348096667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 msgr2=0x55dd4629d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 msgr2=0x55dd4629cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd4629cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/2348096667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd45f0c210 con 0x55dd4629fcb0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fceffd640 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd4629cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d640 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8010e30 tx=0x7f3fc801cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84168e5e39619434 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.561+0000 7f3fb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fb67fc640 1 -- 172.21.15.28:0/2348096667 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8036060 con 0x55dd4629fcb0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fb67fc640 1 -- 172.21.15.28:0/2348096667 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f3fc801c7d0 con 0x55dd4629fcb0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fc801da80 con 0x55dd4629fcb0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fb67fc640 1 -- 172.21.15.28:0/2348096667 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3fc8009bc0 con 0x55dd4629fcb0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 10 client.16797 fetch_fsmap learned FSMap version 34 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55dd45fd8620 con 0x55dd4629fcb0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fb67fc640 1 -- 172.21.15.28:0/2348096667 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fc801da80 con 0x55dd4629fcb0 2024-09-15T22:04:46.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fb67fc640 1 -- 172.21.15.28:0/2348096667 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f3fc800f040 con 0x55dd4629fcb0 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 10 client.16797 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 10 client.16797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 20 client.16797 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd462afb10 0x55dd462b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 4 client.16797 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.562+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd46206a20 con 0x55dd462afb10 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.563+0000 7f3fceffd640 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd462afb10 0x55dd462b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.563+0000 7f3fb67fc640 10 client.16797 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.563+0000 7f3fceffd640 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd462afb10 0x55dd462b1f00 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2567f9e904f12205 server_cookie=3a4c4fdde6eef2a in_seq=0 out_seq=0 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fb67fc640 1 -- 172.21.15.28:0/2348096667 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dd46206a20 con 0x55dd462afb10 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fb67fc640 10 client.16797 handle_command_reply: tid=0 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 client.16797 shutdown 2024-09-15T22:04:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 msgr2=0x55dd4629d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d640 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8010e30 tx=0x7f3fc801cdb0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 shutdown_connections 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd462afb10 0x55dd462b1f00 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d640 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd4629d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fb67fc640 0 client.16797 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2348096667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd4629cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 >> 172.21.15.28:0/2348096667 conn(0x55dd46291060 msgr2=0x55dd46296d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 shutdown_connections 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2348096667 wait complete. 2024-09-15T22:04:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.564+0000 7f3fd83d3740 20 client.16797 trim_cache size 0 max 16384 2024-09-15T22:04:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.578+0000 7f3fd83d3740 1 Processor -- start 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.578+0000 7f3fd83d3740 1 -- start start 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd46297560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd46291440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd01b11d0 con 0x55dd4629fcb0 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd01b2d40 con 0x55dd46294af0 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd01b18a0 con 0x55dd46295410 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35796/0 (socket says 172.21.15.28:35796) 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/2795938033 learned_addr learned my addr 172.21.15.28:0/2795938033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fceffd640 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd46297560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/2795938033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 msgr2=0x55dd46291440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd46291440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/2795938033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 msgr2=0x55dd46297560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd46297560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 -- 172.21.15.28:0/2795938033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd4629bb50 con 0x55dd4629fcb0 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fceffd640 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd46297560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d470 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8010970 tx=0x7f3fc8014fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8413452089f3efea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fc4ff9640 1 -- 172.21.15.28:0/2795938033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8036040 con 0x55dd4629fcb0 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fc4ff9640 1 -- 172.21.15.28:0/2795938033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fc802b070 con 0x55dd4629fcb0 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.579+0000 7f3fc4ff9640 1 -- 172.21.15.28:0/2795938033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8003970 con 0x55dd4629fcb0 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2795938033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 msgr2=0x55dd4629d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d470 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8010970 tx=0x7f3fc8014fc0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2795938033 shutdown_connections 2024-09-15T22:04:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd4629fcb0 0x55dd4629d470 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd46295410 0x55dd46291440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/2795938033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46294af0 0x55dd46297560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2795938033 >> 172.21.15.28:0/2795938033 conn(0x55dd46291060 msgr2=0x55dd4629ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2795938033 shutdown_connections 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 -- 172.21.15.28:0/2795938033 wait complete. 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.580+0000 7f3fd83d3740 1 -- start start 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 1 Processor -- start 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd462a3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46295410 0x55dd462a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd4629fcb0 0x55dd463125f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd01b2670 con 0x55dd46294af0 2024-09-15T22:04:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd01b1fa0 con 0x55dd46295410 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd01af1d0 con 0x55dd4629fcb0 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fd83d3740 10 client.0 _ll_register_callbacks cb 0x7f3fc8020020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd462a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd462a3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35812/0 (socket says 172.21.15.28:35812) 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 -- 172.21.15.28:0/467012986 learned_addr learned my addr 172.21.15.28:0/467012986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fcf7fe640 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd4629fcb0 0x55dd463125f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fc7fff640 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46295410 0x55dd462a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 -- 172.21.15.28:0/467012986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd4629fcb0 msgr2=0x55dd463125f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd4629fcb0 0x55dd463125f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 -- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46295410 msgr2=0x55dd462a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46295410 0x55dd462a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 -- 172.21.15.28:0/467012986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd46312c30 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fceffd640 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd462a3400 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009900 tx=0x7f3fc00095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb94244b87948cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fb67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.581+0000 7f3fb67fc640 1 -- 172.21.15.28:0/467012986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc004d020 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fb67fc640 1 -- 172.21.15.28:0/467012986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f3fc0004240 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fc802b070 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fb67fc640 1 -- 172.21.15.28:0/467012986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3fc0058050 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 10 client.16803 fetch_fsmap learned FSMap version 34 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55dd46319fa0 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fb67fc640 1 -- 172.21.15.28:0/467012986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fc802b070 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fb67fc640 1 -- 172.21.15.28:0/467012986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f3fc0048590 con 0x55dd46294af0 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 10 client.16803 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 10 client.16803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:46.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 20 client.16803 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd4631a750 0x55dd4631cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 4 client.16803 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:04:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd46200d70 con 0x55dd4631a750 2024-09-15T22:04:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.582+0000 7f3fc7fff640 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd4631a750 0x55dd4631cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.583+0000 7f3fc7fff640 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd4631a750 0x55dd4631cb70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80106f97eddd1732 server_cookie=5f587186d1aa4e6c in_seq=0 out_seq=0 2024-09-15T22:04:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.583+0000 7f3fb67fc640 10 client.16803 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fb67fc640 1 -- 172.21.15.28:0/467012986 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd46200d70 con 0x55dd4631a750 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fb67fc640 10 client.16803 handle_command_reply: tid=0 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 client.16803 shutdown 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 msgr2=0x55dd462a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd462a3400 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009900 tx=0x7f3fc00095b0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 shutdown_connections 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55dd4631a750 0x55dd4631cb70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dd4629fcb0 0x55dd463125f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dd46295410 0x55dd462a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 --2- 172.21.15.28:0/467012986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd46294af0 0x55dd462a3400 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 >> 172.21.15.28:0/467012986 conn(0x55dd46291060 msgr2=0x55dd4628f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fb67fc640 0 client.16803 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 shutdown_connections 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 1 -- 172.21.15.28:0/467012986 wait complete. 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.720+0000 7f3fd83d3740 20 client.16803 trim_cache size 0 max 16384 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:04:46.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3fbc0a34e0 msgr2=0x7f3fbc0a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3fbc0a34e0 0x7f3fbc0a59a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8021000 tx=0x7f3fc801e000 comp rx=0 tx=0).stop 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd0101280 msgr2=0x7f3fd019aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd0101280 0x7f3fd019aaf0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800d950 tx=0x7f3fb800de10 comp rx=0 tx=0).stop 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 shutdown_connections 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3fbc0a34e0 0x7f3fbc0a59a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fd010e480 0x7f3fd019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fd0101280 0x7f3fd019aaf0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 --2- 172.21.15.28:0/1242041346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd01008d0 0x7f3fd019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:46.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 >> 172.21.15.28:0/1242041346 conn(0x7f3fd00fc6b0 msgr2=0x7f3fd00feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:46.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 shutdown_connections 2024-09-15T22:04:46.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:46.722+0000 7f3fd5642640 1 -- 172.21.15.28:0/1242041346 wait complete. 2024-09-15T22:04:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:47 smithi028 ceph-mon[30969]: pgmap v228: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 165 KiB/s rd, 88 KiB/s wr, 239 op/s 2024-09-15T22:04:47.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:47 smithi203 ceph-mon[35546]: pgmap v228: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 165 KiB/s rd, 88 KiB/s wr, 239 op/s 2024-09-15T22:04:47.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:47 smithi174 ceph-mon[35244]: pgmap v228: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 165 KiB/s rd, 88 KiB/s wr, 239 op/s 2024-09-15T22:04:49.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:49 smithi028 ceph-mon[30969]: pgmap v229: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 170 KiB/s rd, 93 KiB/s wr, 277 op/s 2024-09-15T22:04:49.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:49 smithi203 ceph-mon[35546]: pgmap v229: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 170 KiB/s rd, 93 KiB/s wr, 277 op/s 2024-09-15T22:04:49.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:49 smithi174 ceph-mon[35244]: pgmap v229: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 170 KiB/s rd, 93 KiB/s wr, 277 op/s 2024-09-15T22:04:51.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:51 smithi028 ceph-mon[30969]: pgmap v230: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 189 op/s 2024-09-15T22:04:51.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:51 smithi203 ceph-mon[35546]: pgmap v230: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 189 op/s 2024-09-15T22:04:51.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:51 smithi174 ceph-mon[35244]: pgmap v230: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 189 op/s 2024-09-15T22:04:52.013 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24485 quiesce db --cancel --all' 2024-09-15T22:04:52.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24485 quiesce db --cancel --all 2024-09-15T22:04:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:04:52.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:04:53.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:53 smithi203 ceph-mon[35546]: pgmap v231: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 224 op/s 2024-09-15T22:04:53.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:53 smithi174 ceph-mon[35244]: pgmap v231: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 224 op/s 2024-09-15T22:04:53.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:53 smithi028 ceph-mon[30969]: pgmap v231: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 224 op/s 2024-09-15T22:04:54.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:54 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:04:54.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:54 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:04:54.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:54 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:04:55.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:55 smithi203 ceph-mon[35546]: pgmap v232: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 KiB/s wr, 162 op/s 2024-09-15T22:04:55.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:55 smithi174 ceph-mon[35244]: pgmap v232: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 KiB/s wr, 162 op/s 2024-09-15T22:04:55.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:55 smithi028 ceph-mon[30969]: pgmap v232: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 KiB/s wr, 162 op/s 2024-09-15T22:04:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:04:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:04:56.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:04:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:04:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:04:56.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:04:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:04:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:04:56.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:04:58.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:57 smithi174 ceph-mon[35244]: pgmap v233: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 KiB/s wr, 146 op/s 2024-09-15T22:04:58.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:57 smithi028 ceph-mon[30969]: pgmap v233: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 KiB/s wr, 146 op/s 2024-09-15T22:04:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 -- 172.21.15.28:0/4099497102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708100e20 msgr2=0x7fd708101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 --2- 172.21.15.28:0/4099497102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708100e20 0x7fd708101200 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd704009900 tx=0x7fd70402a480 comp rx=0 tx=0).stop 2024-09-15T22:04:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 -- 172.21.15.28:0/4099497102 shutdown_connections 2024-09-15T22:04:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 --2- 172.21.15.28:0/4099497102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd708102ef0 0x7fd70810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 --2- 172.21.15.28:0/4099497102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd708102570 0x7fd7081029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 --2- 172.21.15.28:0/4099497102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708100e20 0x7fd708101200 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 -- 172.21.15.28:0/4099497102 >> 172.21.15.28:0/4099497102 conn(0x7fd7080fc7d0 msgr2=0x7fd7080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 -- 172.21.15.28:0/4099497102 shutdown_connections 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.447+0000 7fd71064c640 1 -- 172.21.15.28:0/4099497102 wait complete. 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 Processor -- start 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 -- start start 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd708100e20 0x7fd708196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd708102570 0x7fd7081968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708102ef0 0x7fd70819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd70810fe80 con 0x7fd708102ef0 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd70810fd00 con 0x7fd708100e20 2024-09-15T22:04:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd71064c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd708110000 con 0x7fd708102570 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd70ebe9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708102ef0 0x7fd70819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd70ebe9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708102ef0 0x7fd70819ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40810/0 (socket says 172.21.15.28:40810) 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd70ebe9640 1 -- 172.21.15.28:0/898779578 learned_addr learned my addr 172.21.15.28:0/898779578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd708100e20 0x7fd708196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.448+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd708102570 0x7fd7081968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd70ebe9640 1 -- 172.21.15.28:0/898779578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd708102570 msgr2=0x7fd7081968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd70ebe9640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd708102570 0x7fd7081968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd70ebe9640 1 -- 172.21.15.28:0/898779578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd708100e20 msgr2=0x7fd708196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd70ebe9640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd708100e20 0x7fd708196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd70ebe9640 1 -- 172.21.15.28:0/898779578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70819b310 con 0x7fd708102ef0 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd70ebe9640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708102ef0 0x7fd70819ac30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd70000bfb0 tx=0x7fd7000114a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd6ff7fe640 1 -- 172.21.15.28:0/898779578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700011dc0 con 0x7fd708102ef0 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd6ff7fe640 1 -- 172.21.15.28:0/898779578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd700005020 con 0x7fd708102ef0 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70819b600 con 0x7fd708102ef0 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd6ff7fe640 1 -- 172.21.15.28:0/898779578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700017650 con 0x7fd708102ef0 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.449+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7081a2ee0 con 0x7fd708102ef0 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd6ff7fe640 1 -- 172.21.15.28:0/898779578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd70001e020 con 0x7fd708102ef0 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 Processor -- start 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 -- start start 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f142de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55767f0a7200 con 0x55767f14dbe0 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55767f04ddc0 con 0x55767f143320 2024-09-15T22:04:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.452+0000 7fd712a25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55767f0a31c0 con 0x55767f142a00 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40822/0 (socket says 172.21.15.28:40822) 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1033681069 learned_addr learned my addr 172.21.15.28:0/1033681069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd6ff7fe640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6f40a3430 0x7fd6f40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1033681069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 msgr2=0x55767f142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f142de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1033681069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 msgr2=0x55767f14d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1033681069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55767ee18620 con 0x55767f14dbe0 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70ebe9640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6f40a3430 0x7fd6f40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd6ff7fe640 1 -- 172.21.15.28:0/898779578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fd7000d38f0 con 0x7fd708102ef0 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.453+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ffd0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800a830 tx=0x7fd6f8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369a3770b2f6a8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd6fcff9640 1 -- 172.21.15.28:0/1033681069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8004250 con 0x55767f14dbe0 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd6fcff9640 1 -- 172.21.15.28:0/1033681069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f80043f0 con 0x55767f14dbe0 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd6fcff9640 1 -- 172.21.15.28:0/1033681069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8004d70 con 0x55767f14dbe0 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd70ebe9640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6f40a3430 0x7fd6f40a58f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd7000d0d40 tx=0x7fd70008a050 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd712a25740 1 -- 172.21.15.28:0/1033681069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 msgr2=0x55767f14ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd712a25740 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ffd0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800a830 tx=0x7fd6f8005f90 comp rx=0 tx=0).stop 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd712a25740 1 -- 172.21.15.28:0/1033681069 shutdown_connections 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd712a25740 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ffd0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd712a25740 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd712a25740 1 --2- 172.21.15.28:0/1033681069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.454+0000 7fd712a25740 1 -- 172.21.15.28:0/1033681069 >> 172.21.15.28:0/1033681069 conn(0x55767f13ef90 msgr2=0x55767f13f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 -- 172.21.15.28:0/1033681069 shutdown_connections 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 -- 172.21.15.28:0/1033681069 wait complete. 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 -- start start 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 Processor -- start 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f14d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55767f1507b0 con 0x55767f14dbe0 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55767f041c50 con 0x55767f143320 2024-09-15T22:04:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55767f150930 con 0x55767f142a00 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd712a25740 10 client.0 _ll_register_callbacks cb 0x55767f053ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40834/0 (socket says 172.21.15.28:40834) 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.455+0000 7fd70dbe7640 1 -- 172.21.15.28:0/881809972 learned_addr learned my addr 172.21.15.28:0/881809972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f14d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70dbe7640 1 -- 172.21.15.28:0/881809972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 msgr2=0x55767f14d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f14d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70dbe7640 1 -- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 msgr2=0x55767f14a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70dbe7640 1 -- 172.21.15.28:0/881809972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55767f14b590 con 0x55767f14dbe0 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ad60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8001eb0 tx=0x7fd6f8004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747c2e6d37a933f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd6f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/881809972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8020040 con 0x55767f14dbe0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/881809972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fd6f8005570 con 0x55767f14dbe0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6f80043f0 con 0x55767f14dbe0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/881809972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd6f8038050 con 0x55767f14dbe0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd712a25740 10 client.16815 fetch_fsmap learned FSMap version 34 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.456+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55767f15c360 con 0x55767f14dbe0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/881809972 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f80043f0 con 0x55767f14dbe0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/881809972 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd6f8026040 con 0x55767f14dbe0 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd712a25740 10 client.16815 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd712a25740 10 client.16815 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd712a25740 20 client.16815 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd712a25740 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f15d760 0x55767f15fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd712a25740 4 client.16815 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55767f0b4770 con 0x55767f15d760 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.457+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f15d760 0x55767f15fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.458+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f15d760 0x55767f15fb50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3c3caacc75234a8 server_cookie=4f3a6f784df03a33 in_seq=0 out_seq=0 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.458+0000 7fd6f1ffb640 10 client.16815 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.458+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/881809972 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55767f0b4770 con 0x55767f15d760 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.458+0000 7fd6f1ffb640 10 client.16815 handle_command_reply: tid=0 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.458+0000 7fd712a25740 1 client.16815 shutdown 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 msgr2=0x55767f14ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ad60 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8001eb0 tx=0x7fd6f8004950 comp rx=0 tx=0).stop 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 shutdown_connections 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f15d760 0x55767f15fb50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f14dbe0 0x55767f14ad60 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f14a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd6f1ffb640 0 client.16815 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 --2- 172.21.15.28:0/881809972 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f142a00 0x55767f14d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 >> 172.21.15.28:0/881809972 conn(0x55767f13ef90 msgr2=0x55767f13e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 shutdown_connections 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 1 -- 172.21.15.28:0/881809972 wait complete. 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.459+0000 7fd712a25740 20 client.16815 trim_cache size 0 max 16384 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 Processor -- start 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 -- start start 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f1460d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f159a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 0x55767f165120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081ace50 con 0x55767f142a00 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081ae9c0 con 0x55767f143320 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.476+0000 7fd712a25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081ad520 con 0x55767f14dbe0 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f1460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f1460d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40844/0 (socket says 172.21.15.28:40844) 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1813555712 learned_addr learned my addr 172.21.15.28:0/1813555712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1813555712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 msgr2=0x55767f165120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 0x55767f165120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1813555712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 msgr2=0x55767f159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 -- 172.21.15.28:0/1813555712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55767f1508e0 con 0x55767f142a00 2024-09-15T22:04:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f1460d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8005c20 tx=0x7fd6f8004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62dec1da91d6954 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd6f3fff640 1 -- 172.21.15.28:0/1813555712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8020040 con 0x55767f142a00 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd6f3fff640 1 -- 172.21.15.28:0/1813555712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f802f070 con 0x55767f142a00 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.477+0000 7fd6f3fff640 1 -- 172.21.15.28:0/1813555712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f80055d0 con 0x55767f142a00 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 -- 172.21.15.28:0/1813555712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 msgr2=0x55767f1460d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f1460d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8005c20 tx=0x7fd6f8004a20 comp rx=0 tx=0).stop 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 -- 172.21.15.28:0/1813555712 shutdown_connections 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 0x55767f165120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 --2- 172.21.15.28:0/1813555712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f1460d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 -- 172.21.15.28:0/1813555712 >> 172.21.15.28:0/1813555712 conn(0x55767f13ef90 msgr2=0x55767f14c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 -- 172.21.15.28:0/1813555712 shutdown_connections 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 -- 172.21.15.28:0/1813555712 wait complete. 2024-09-15T22:04:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 -- start start 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.478+0000 7fd712a25740 1 Processor -- start 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd712a25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f156320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd712a25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f151650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd712a25740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 0x55767f151b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd712a25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081ae2f0 con 0x55767f142a00 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd712a25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081adc20 con 0x55767f143320 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd712a25740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7081aae50 con 0x55767f14dbe0 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd712a25740 10 client.0 _ll_register_callbacks cb 0x7fd704009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40848/0 (socket says 172.21.15.28:40848) 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 -- 172.21.15.28:0/250306976 learned_addr learned my addr 172.21.15.28:0/250306976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 -- 172.21.15.28:0/250306976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 msgr2=0x55767f151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 0x55767f151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 -- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 msgr2=0x55767f151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 -- 172.21.15.28:0/250306976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55767f150600 con 0x55767f142a00 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd70dbe7640 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f156320 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800b720 tx=0x7fd6f8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebe66baf19f2a15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.479+0000 7fd6f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/250306976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f801c9a0 con 0x55767f142a00 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/250306976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fd6f8004660 con 0x55767f142a00 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6f802f070 con 0x55767f142a00 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/250306976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd6f804e050 con 0x55767f142a00 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 10 client.16821 fetch_fsmap learned FSMap version 34 2024-09-15T22:04:58.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55767f1c7f60 con 0x55767f142a00 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/250306976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f802f070 con 0x55767f142a00 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/250306976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd6f8026070 con 0x55767f142a00 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 10 client.16821 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 10 client.16821 resolve_mds: validated gid 24485 aka daemon mds.b 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 34, 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24485, 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 13, 2024-09-15T22:04:58.222 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": {} 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 20 client.16821 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f1c83f0 0x55767f1ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 4 client.16821 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.480+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55767f0aec80 con 0x55767f1c83f0 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.481+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f1c83f0 0x55767f1ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.481+0000 7fd6f1ffb640 10 client.16821 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.481+0000 7fd70e3e8640 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f1c83f0 0x55767f1ca7e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d510b157a19674d5 server_cookie=a24c47e2f2d862a3 in_seq=0 out_seq=0 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.481+0000 7fd6f1ffb640 1 -- 172.21.15.28:0/250306976 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55767f0aec80 con 0x55767f1c83f0 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd6f1ffb640 10 client.16821 handle_command_reply: tid=0 2024-09-15T22:04:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 client.16821 shutdown 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 msgr2=0x55767f156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f156320 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800b720 tx=0x7fd6f8005030 comp rx=0 tx=0).stop 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 shutdown_connections 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55767f1c83f0 0x55767f1ca7e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55767f14dbe0 0x55767f151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55767f143320 0x55767f151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd6f1ffb640 0 client.16821 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 --2- 172.21.15.28:0/250306976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55767f142a00 0x55767f156320 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 >> 172.21.15.28:0/250306976 conn(0x55767f13ef90 msgr2=0x55767f13e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 shutdown_connections 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 1 -- 172.21.15.28:0/250306976 wait complete. 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd712a25740 20 client.16821 trim_cache size 0 max 16384 2024-09-15T22:04:58.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6f40a3430 msgr2=0x7fd6f40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd71064c640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6f40a3430 0x7fd6f40a58f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd7000d0d40 tx=0x7fd70008a050 comp rx=0 tx=0).stop 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708102ef0 msgr2=0x7fd70819ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.482+0000 7fd71064c640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708102ef0 0x7fd70819ac30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd70000bfb0 tx=0x7fd7000114a0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 shutdown_connections 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd6f40a3430 0x7fd6f40a58f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd708102ef0 0x7fd70819ac30 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd708102570 0x7fd7081968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 --2- 172.21.15.28:0/898779578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd708100e20 0x7fd708196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 >> 172.21.15.28:0/898779578 conn(0x7fd7080fc7d0 msgr2=0x7fd7080fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 shutdown_connections 2024-09-15T22:04:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:04:57.483+0000 7fd71064c640 1 -- 172.21.15.28:0/898779578 wait complete. 2024-09-15T22:04:58.226 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:57 smithi203 ceph-mon[35546]: pgmap v233: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 KiB/s wr, 146 op/s 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 -- 172.21.15.28:0/2593640417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630101ad0 msgr2=0x7f3630101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 --2- 172.21.15.28:0/2593640417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630101ad0 0x7f3630101f30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3624009930 tx=0x7f362402a420 comp rx=0 tx=0).stop 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 -- 172.21.15.28:0/2593640417 shutdown_connections 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 --2- 172.21.15.28:0/2593640417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3630102470 0x7f363010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 --2- 172.21.15.28:0/2593640417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630101ad0 0x7f3630101f30 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 --2- 172.21.15.28:0/2593640417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3630107b70 0x7f3630107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 -- 172.21.15.28:0/2593640417 >> 172.21.15.28:0/2593640417 conn(0x7f36300fd7d0 msgr2=0x7f36300ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:59.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 -- 172.21.15.28:0/2593640417 shutdown_connections 2024-09-15T22:04:59.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.830+0000 7f3636847640 1 -- 172.21.15.28:0/2593640417 wait complete. 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 Processor -- start 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 -- start start 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3630101ad0 0x7f36300744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630102470 0x7f3630074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3630107b70 0x7f3630074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3630111ee0 con 0x7f3630102470 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3630111d60 con 0x7f3630107b70 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3636847640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3630112060 con 0x7f3630101ad0 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630102470 0x7f3630074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630102470 0x7f3630074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40868/0 (socket says 172.21.15.28:40868) 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f362f7fe640 1 -- 172.21.15.28:0/3508315682 learned_addr learned my addr 172.21.15.28:0/3508315682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f362ffff640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3630101ad0 0x7f36300744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.831+0000 7f3634de4640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3630107b70 0x7f3630074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362f7fe640 1 -- 172.21.15.28:0/3508315682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3630101ad0 msgr2=0x7f36300744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362f7fe640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3630101ad0 0x7f36300744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362f7fe640 1 -- 172.21.15.28:0/3508315682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3630107b70 msgr2=0x7f3630074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362f7fe640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3630107b70 0x7f3630074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362f7fe640 1 -- 172.21.15.28:0/3508315682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36301a68b0 con 0x7f3630102470 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362f7fe640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630102470 0x7f3630074a30 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3624009230 tx=0x7f3624004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362d7fa640 1 -- 172.21.15.28:0/3508315682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362403f030 con 0x7f3630102470 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362d7fa640 1 -- 172.21.15.28:0/3508315682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3624003ad0 con 0x7f3630102470 2024-09-15T22:04:59.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36301a6b40 con 0x7f3630102470 2024-09-15T22:04:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f362d7fa640 1 -- 172.21.15.28:0/3508315682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624035070 con 0x7f3630102470 2024-09-15T22:04:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.832+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36301a7030 con 0x7f3630102470 2024-09-15T22:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f362d7fa640 1 -- 172.21.15.28:0/3508315682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f3624042030 con 0x7f3630102470 2024-09-15T22:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 Processor -- start 2024-09-15T22:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 -- start start 2024-09-15T22:04:59.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe442bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe442c410 0x563fe4436770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe4436cb0 0x563fe44390a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563fe43902f0 con 0x563fe442c410 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563fe4338270 con 0x563fe4436cb0 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f36395db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563fe438c2b0 con 0x563fe442baf0 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f362ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe442c410 0x563fe4436770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe4436cb0 0x563fe44390a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f362ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe442c410 0x563fe4436770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40870/0 (socket says 172.21.15.28:40870) 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.835+0000 7f362ffff640 1 -- 172.21.15.28:0/4254253450 learned_addr learned my addr 172.21.15.28:0/4254253450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f3634de4640 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe442bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362d7fa640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f36180a3460 0x7f36180a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362ffff640 1 -- 172.21.15.28:0/4254253450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 msgr2=0x563fe442bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362ffff640 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe442bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362ffff640 1 -- 172.21.15.28:0/4254253450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe4436cb0 msgr2=0x563fe44390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362ffff640 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe4436cb0 0x563fe44390a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362ffff640 1 -- 172.21.15.28:0/4254253450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fe4337340 con 0x563fe442c410 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362f7fe640 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe4436cb0 0x563fe44390a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362d7fa640 1 -- 172.21.15.28:0/3508315682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f36240f0520 con 0x7f3630102470 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f3634de4640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f36180a3460 0x7f36180a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f362ffff640 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe442c410 0x563fe4436770 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f362000bc90 tx=0x7f36200107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edab713b6ea4436 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f36167fc640 1 -- 172.21.15.28:0/4254253450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362001c480 con 0x563fe442c410 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f36167fc640 1 -- 172.21.15.28:0/4254253450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f362001cb00 con 0x563fe442c410 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.836+0000 7f36167fc640 1 -- 172.21.15.28:0/4254253450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620011bb0 con 0x563fe442c410 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 -- 172.21.15.28:0/4254253450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe442c410 msgr2=0x563fe4436770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f3634de4640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f36180a3460 0x7f36180a5920 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3628005fd0 tx=0x7f3628004a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe442c410 0x563fe4436770 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f362000bc90 tx=0x7f36200107a0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 -- 172.21.15.28:0/4254253450 shutdown_connections 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe4436cb0 0x563fe44390a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe442c410 0x563fe4436770 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 --2- 172.21.15.28:0/4254253450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe442bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 -- 172.21.15.28:0/4254253450 >> 172.21.15.28:0/4254253450 conn(0x563fe4428060 msgr2=0x563fe4428440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 -- 172.21.15.28:0/4254253450 shutdown_connections 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 -- 172.21.15.28:0/4254253450 wait complete. 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.837+0000 7f36395db740 1 -- start start 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 1 Processor -- start 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe4433b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe44345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563fe4439880 con 0x563fe4436cb0 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563fe4319290 con 0x563fe442baf0 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563fe4439a00 con 0x563fe442c410 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f36395db740 10 client.0 _ll_register_callbacks cb 0x563fe434a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe44345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe44345d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40880/0 (socket says 172.21.15.28:40880) 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 -- 172.21.15.28:0/1544723251 learned_addr learned my addr 172.21.15.28:0/1544723251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f362ffff640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe4433b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f362f7fe640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 -- 172.21.15.28:0/1544723251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 msgr2=0x563fe4434090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 -- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 msgr2=0x563fe4433b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe4433b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f3634de4640 1 -- 172.21.15.28:0/1544723251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fe40a3210 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.838+0000 7f362f7fe640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f3634de4640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe44345d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f36280116c0 tx=0x7f3628004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6535b78a3e745bf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f360f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f360f7fe640 1 -- 172.21.15.28:0/1544723251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628010980 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f360f7fe640 1 -- 172.21.15.28:0/1544723251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f3628010b20 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f362001cb00 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f360f7fe640 1 -- 172.21.15.28:0/1544723251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3628030050 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f36395db740 10 client.16833 fetch_fsmap learned FSMap version 34 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563fe416f620 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f360f7fe640 1 -- 172.21.15.28:0/1544723251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f362001cb00 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.839+0000 7f360f7fe640 1 -- 172.21.15.28:0/1544723251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f362801f420 con 0x563fe4436cb0 2024-09-15T22:04:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f36395db740 10 client.16833 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f36395db740 10 client.16833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f36395db740 20 client.16833 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f36395db740 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe4446c20 0x563fe4449090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f36395db740 4 client.16833 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fe439da20 con 0x563fe4446c20 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f362ffff640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe4446c20 0x563fe4449090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f362ffff640 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe4446c20 0x563fe4449090 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77caad675753d98a server_cookie=e35f6a818d18c94c in_seq=0 out_seq=0 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.840+0000 7f360f7fe640 10 client.16833 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f360f7fe640 1 -- 172.21.15.28:0/1544723251 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563fe439da20 con 0x563fe4446c20 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f360f7fe640 10 client.16833 handle_command_reply: tid=0 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 client.16833 shutdown 2024-09-15T22:04:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 msgr2=0x563fe44345d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe44345d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f36280116c0 tx=0x7f3628004ab0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 shutdown_connections 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe4446c20 0x563fe4449090 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe44345d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f360f7fe640 0 client.16833 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 --2- 172.21.15.28:0/1544723251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe4433b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 >> 172.21.15.28:0/1544723251 conn(0x563fe4428060 msgr2=0x563fe442dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.841+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 shutdown_connections 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.842+0000 7f36395db740 1 -- 172.21.15.28:0/1544723251 wait complete. 2024-09-15T22:04:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.842+0000 7f36395db740 20 client.16833 trim_cache size 0 max 16384 2024-09-15T22:04:59.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 Processor -- start 2024-09-15T22:04:59.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 -- start start 2024-09-15T22:04:59.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe442e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe444e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36301b1150 con 0x563fe4436cb0 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36301b2cc0 con 0x563fe442baf0 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f36395db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36301b1820 con 0x563fe442c410 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.856+0000 7f3634de4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe444e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe444e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40886/0 (socket says 172.21.15.28:40886) 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 -- 172.21.15.28:0/2615551252 learned_addr learned my addr 172.21.15.28:0/2615551252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f362ffff640 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe442e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 -- 172.21.15.28:0/2615551252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 msgr2=0x563fe4434400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f362f7fe640 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 -- 172.21.15.28:0/2615551252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 msgr2=0x563fe442e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe442e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 -- 172.21.15.28:0/2615551252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fe4427e90 con 0x563fe4436cb0 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f362ffff640 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe442e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f362f7fe640 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3634de4640 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe444e9f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f362801f840 tx=0x7f3628009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7723bf373b306eff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3615ffb640 1 -- 172.21.15.28:0/2615551252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362801f620 con 0x563fe4436cb0 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3615ffb640 1 -- 172.21.15.28:0/2615551252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3628039070 con 0x563fe4436cb0 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.857+0000 7f3615ffb640 1 -- 172.21.15.28:0/2615551252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628034430 con 0x563fe4436cb0 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 -- 172.21.15.28:0/2615551252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 msgr2=0x563fe444e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe444e9f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f362801f840 tx=0x7f3628009fd0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 -- 172.21.15.28:0/2615551252 shutdown_connections 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe444e9f0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442c410 0x563fe4434400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 --2- 172.21.15.28:0/2615551252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442baf0 0x563fe442e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 -- 172.21.15.28:0/2615551252 >> 172.21.15.28:0/2615551252 conn(0x563fe4428060 msgr2=0x563fe4435bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 -- 172.21.15.28:0/2615551252 shutdown_connections 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 -- 172.21.15.28:0/2615551252 wait complete. 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 -- start start 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.858+0000 7f36395db740 1 Processor -- start 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe443f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442c410 0x563fe4430860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe4430da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36301b25f0 con 0x563fe4436cb0 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36301b1f20 con 0x563fe442c410 2024-09-15T22:04:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36301af150 con 0x563fe442baf0 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 10 client.0 _ll_register_callbacks cb 0x7f3628009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe4430da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe4430da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40900/0 (socket says 172.21.15.28:40900) 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 -- 172.21.15.28:0/1196988041 learned_addr learned my addr 172.21.15.28:0/1196988041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f362ffff640 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe443f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f362f7fe640 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442c410 0x563fe4430860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 -- 172.21.15.28:0/1196988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 msgr2=0x563fe443f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe443f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 -- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442c410 msgr2=0x563fe4430860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442c410 0x563fe4430860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 -- 172.21.15.28:0/1196988041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fe44315d0 con 0x563fe4436cb0 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f3634de4640 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe4430da0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3628000f80 tx=0x7f3628004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7236fc71a714cab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:04:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f360f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f360f7fe640 1 -- 172.21.15.28:0/1196988041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628041020 con 0x563fe4436cb0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f360f7fe640 1 -- 172.21.15.28:0/1196988041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f362801f420 con 0x563fe4436cb0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.859+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3628039070 con 0x563fe4436cb0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f360f7fe640 1 -- 172.21.15.28:0/1196988041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f362801f620 con 0x563fe4436cb0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 10 client.16839 fetch_fsmap learned FSMap version 34 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563fe44b1940 con 0x563fe4436cb0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f360f7fe640 1 -- 172.21.15.28:0/1196988041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3628039070 con 0x563fe4436cb0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f360f7fe640 1 -- 172.21.15.28:0/1196988041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f3628005200 con 0x563fe4436cb0 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 10 client.16839 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 20 client.16839 populate_metadata read hostname 'smithi028' 2024-09-15T22:04:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe44b1ff0 0x563fe44b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:04:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 4 client.16839 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T22:04:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563fe4397d70 con 0x563fe44b1ff0 2024-09-15T22:04:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.860+0000 7f362ffff640 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe44b1ff0 0x563fe44b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:04:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.861+0000 7f362ffff640 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe44b1ff0 0x563fe44b43e0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=489a1e6f5a20ea27 server_cookie=9c2d5c905b7f2c4 in_seq=0 out_seq=0 2024-09-15T22:04:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.861+0000 7f360f7fe640 10 client.16839 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f360f7fe640 1 -- 172.21.15.28:0/1196988041 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563fe4397d70 con 0x563fe44b1ff0 2024-09-15T22:04:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f360f7fe640 10 client.16839 handle_command_reply: tid=0 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 client.16839 shutdown 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 msgr2=0x563fe4430da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe4430da0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3628000f80 tx=0x7f3628004790 comp rx=0 tx=0).stop 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 shutdown_connections 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563fe44b1ff0 0x563fe44b43e0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563fe4436cb0 0x563fe4430da0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fe442c410 0x563fe4430860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f360f7fe640 0 client.16839 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 --2- 172.21.15.28:0/1196988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fe442baf0 0x563fe443f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.868+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 >> 172.21.15.28:0/1196988041 conn(0x563fe4428060 msgr2=0x563fe44269a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 shutdown_connections 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f36395db740 1 -- 172.21.15.28:0/1196988041 wait complete. 2024-09-15T22:04:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f36395db740 20 client.16839 trim_cache size 0 max 16384 2024-09-15T22:04:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-09-15T22:04:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-09-15T22:04:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-09-15T22:04:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f36180a3460 msgr2=0x7f36180a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f36180a3460 0x7f36180a5920 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3628005fd0 tx=0x7f3628004a40 comp rx=0 tx=0).stop 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630102470 msgr2=0x7f3630074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630102470 0x7f3630074a30 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f3624009230 tx=0x7f3624004270 comp rx=0 tx=0).stop 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 shutdown_connections 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f36180a3460 0x7f36180a5920 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3630107b70 0x7f3630074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3630102470 0x7f3630074a30 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 --2- 172.21.15.28:0/3508315682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3630101ad0 0x7f36300744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:04:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 >> 172.21.15.28:0/3508315682 conn(0x7f36300fd7d0 msgr2=0x7f36300fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:04:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 shutdown_connections 2024-09-15T22:04:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-09-15T22:04:59.869+0000 7f3636847640 1 -- 172.21.15.28:0/3508315682 wait complete. 2024-09-15T22:05:00.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:04:59 smithi028 ceph-mon[30969]: pgmap v234: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 139 op/s 2024-09-15T22:05:00.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:04:59 smithi203 ceph-mon[35546]: pgmap v234: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 139 op/s 2024-09-15T22:05:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:04:59 smithi174 ceph-mon[35244]: pgmap v234: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 11 KiB/s rd, 10 KiB/s wr, 139 op/s 2024-09-15T22:05:02.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:01 smithi028 ceph-mon[30969]: pgmap v235: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.2 KiB/s wr, 85 op/s 2024-09-15T22:05:02.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:01 smithi203 ceph-mon[35546]: pgmap v235: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.2 KiB/s wr, 85 op/s 2024-09-15T22:05:02.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:01 smithi174 ceph-mon[35244]: pgmap v235: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.2 KiB/s wr, 85 op/s 2024-09-15T22:05:03.479 INFO:tasks.quiescer:done joining Quiescers 2024-09-15T22:05:03.479 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-09-15T22:05:03.490 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T22:05:03.500 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-09-15T22:05:03.512 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-09-15T22:05:03.513 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T22:05:03.513 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T22:05:03.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-09-15T22:05:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:03.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-09-15T22:05:03.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:03 smithi028 ceph-mon[30969]: pgmap v236: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 28 KiB/s wr, 80 op/s 2024-09-15T22:05:04.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:03 smithi203 ceph-mon[35546]: pgmap v236: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 28 KiB/s wr, 80 op/s 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 -- 172.21.15.28:0/1945377429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0103660 msgr2=0x7f3de010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 --2- 172.21.15.28:0/1945377429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0103660 0x7f3de010bb50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4009900 tx=0x7f3dd402a480 comp rx=0 tx=0).stop 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 -- 172.21.15.28:0/1945377429 shutdown_connections 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 --2- 172.21.15.28:0/1945377429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3de010c170 0x7f3de010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 --2- 172.21.15.28:0/1945377429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0103660 0x7f3de010bb50 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 --2- 172.21.15.28:0/1945377429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de0102d40 0x7f3de0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 -- 172.21.15.28:0/1945377429 >> 172.21.15.28:0/1945377429 conn(0x7f3de00fc7a0 msgr2=0x7f3de00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 -- 172.21.15.28:0/1945377429 shutdown_connections 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.064+0000 7f3de4d11640 1 -- 172.21.15.28:0/1945377429 wait complete. 2024-09-15T22:05:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 Processor -- start 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 -- start start 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0102d40 0x7f3de01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3de0103660 0x7f3de0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de010c170 0x7f3de0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3de0111f70 con 0x7f3de0102d40 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3de0111df0 con 0x7f3de0103660 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3de4d11640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01120f0 con 0x7f3de010c170 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0102d40 0x7f3de01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0102d40 0x7f3de01009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52036/0 (socket says 172.21.15.28:52036) 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.065+0000 7f3dde59c640 1 -- 172.21.15.28:0/3214479043 learned_addr learned my addr 172.21.15.28:0/3214479043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3de0103660 0x7f3de0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dded9d640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de010c170 0x7f3de0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dde59c640 1 -- 172.21.15.28:0/3214479043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de010c170 msgr2=0x7f3de0101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dde59c640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de010c170 0x7f3de0101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dde59c640 1 -- 172.21.15.28:0/3214479043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3de0103660 msgr2=0x7f3de0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dde59c640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3de0103660 0x7f3de0100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dde59c640 1 -- 172.21.15.28:0/3214479043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de01a68f0 con 0x7f3de0102d40 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dde59c640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0102d40 0x7f3de01009a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000d770 tx=0x7f3dd000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dcf7fe640 1 -- 172.21.15.28:0/3214479043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0017070 con 0x7f3de0102d40 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de01a6be0 con 0x7f3de0102d40 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dcf7fe640 1 -- 172.21.15.28:0/3214479043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd0005500 con 0x7f3de0102d40 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3dcf7fe640 1 -- 172.21.15.28:0/3214479043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0004e00 con 0x7f3de0102d40 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.066+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de01a7120 con 0x7f3de0102d40 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.068+0000 7f3dcf7fe640 1 -- 172.21.15.28:0/3214479043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f3dd0004fa0 con 0x7f3de0102d40 2024-09-15T22:05:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 Processor -- start 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 -- start start 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef7291c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72927140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef72929a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef728822e0 con 0x55ef7291cdc0 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ef72829ce0 con 0x55ef7291c4a0 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3de70ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ef7287e2a0 con 0x55ef72927680 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3dcf7fe640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3dc80a33b0 0x7f3dc80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72927140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72927140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52052/0 (socket says 172.21.15.28:52052) 2024-09-15T22:05:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3dde59c640 1 -- 172.21.15.28:0/1134736591 learned_addr learned my addr 172.21.15.28:0/1134736591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.069+0000 7f3dded9d640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3dc80a33b0 0x7f3dc80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef72929a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dded9d640 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef7291c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dcf7fe640 1 -- 172.21.15.28:0/3214479043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f3dd00cfc00 con 0x7f3de0102d40 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dde59c640 1 -- 172.21.15.28:0/1134736591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 msgr2=0x55ef72929a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dde59c640 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef72929a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dde59c640 1 -- 172.21.15.28:0/1134736591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 msgr2=0x55ef7291c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dde59c640 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef7291c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dde59c640 1 -- 172.21.15.28:0/1134736591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef7280e440 con 0x55ef7291cdc0 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dded9d640 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef7291c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dde59c640 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72927140 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0097f90 tx=0x7f3dd009ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5535635ef349b476 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dded9d640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3dc80a33b0 0x7f3dc80a5870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80126d0 tx=0x7f3dd800a290 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dccff9640 1 -- 172.21.15.28:0/1134736591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0020070 con 0x55ef7291cdc0 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dccff9640 1 -- 172.21.15.28:0/1134736591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd009bb30 con 0x55ef7291cdc0 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.070+0000 7f3dccff9640 1 -- 172.21.15.28:0/1134736591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd009ce60 con 0x55ef7291cdc0 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 -- 172.21.15.28:0/1134736591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 msgr2=0x55ef72927140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72927140 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0097f90 tx=0x7f3dd009ba30 comp rx=0 tx=0).stop 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 -- 172.21.15.28:0/1134736591 shutdown_connections 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef72929a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72927140 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 --2- 172.21.15.28:0/1134736591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef7291c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 -- 172.21.15.28:0/1134736591 >> 172.21.15.28:0/1134736591 conn(0x55ef72918a30 msgr2=0x55ef72918e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 -- 172.21.15.28:0/1134736591 shutdown_connections 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 -- 172.21.15.28:0/1134736591 wait complete. 2024-09-15T22:05:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 -- start start 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.071+0000 7f3de70ea740 1 Processor -- start 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef72926fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72924270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef729247b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3de70ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef7292a1f0 con 0x55ef7291cdc0 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3de70ea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ef7283d050 con 0x55ef7291c4a0 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3de70ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ef7292a370 con 0x55ef72927680 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3de70ea740 10 client.0 _ll_register_callbacks cb 0x55ef7287d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72924270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72924270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52058/0 (socket says 172.21.15.28:52058) 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dded9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef729247b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 -- 172.21.15.28:0/2013991764 learned_addr learned my addr 172.21.15.28:0/2013991764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dde59c640 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef72926fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 -- 172.21.15.28:0/2013991764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 msgr2=0x55ef729247b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef729247b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 -- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 msgr2=0x55ef72926fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef72926fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 -- 172.21.15.28:0/2013991764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef725f33c0 con 0x55ef7291cdc0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72924270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3dd402a9a0 tx=0x7f3dd40319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f102086fb4f4002 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.072+0000 7f3dc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/2013991764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4035040 con 0x55ef7291cdc0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/2013991764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f3dd4004240 con 0x55ef7291cdc0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dd009bb30 con 0x55ef7291cdc0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/2013991764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3dd404e050 con 0x55ef7291cdc0 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 10 client.16851 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ef72935bf0 con 0x55ef7291cdc0 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/2013991764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd009bb30 con 0x55ef7291cdc0 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/2013991764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f3dd4004ac0 con 0x55ef7291cdc0 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 10 client.16851 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 20 client.16851 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729370b0 0x55ef72939510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 4 client.16851 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.073+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef7288f850 con 0x55ef729370b0 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.074+0000 7f3dde59c640 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729370b0 0x55ef72939510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.074+0000 7f3dde59c640 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729370b0 0x55ef72939510 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f42f6544d68d3a5 server_cookie=98353bdbbd074944 in_seq=0 out_seq=0 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.074+0000 7f3dc5ffb640 10 client.16851 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/2013991764 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ef7288f850 con 0x55ef729370b0 2024-09-15T22:05:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3dc5ffb640 10 client.16851 handle_command_reply: tid=0 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 client.16851 shutdown 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 msgr2=0x55ef72924270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72924270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3dd402a9a0 tx=0x7f3dd40319b0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 shutdown_connections 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729370b0 0x55ef72939510 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef72927680 0x55ef729247b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291cdc0 0x55ef72924270 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3dc5ffb640 0 client.16851 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 --2- 172.21.15.28:0/2013991764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291c4a0 0x55ef72926fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 >> 172.21.15.28:0/2013991764 conn(0x55ef72918a30 msgr2=0x55ef7291e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 shutdown_connections 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 1 -- 172.21.15.28:0/2013991764 wait complete. 2024-09-15T22:05:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.075+0000 7f3de70ea740 20 client.16851 trim_cache size 0 max 16384 2024-09-15T22:05:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 Processor -- start 2024-09-15T22:05:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 -- start start 2024-09-15T22:05:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291c4a0 0x55ef72933450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.245 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:05:04.245 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:05:04.245 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:05:04.245 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291cdc0 0x55ef7293ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef72927680 0x55ef7293f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01b11d0 con 0x55ef72927680 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01b2d40 con 0x55ef7291cdc0 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.092+0000 7f3de70ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01b18a0 con 0x55ef7291c4a0 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef72927680 0x55ef7293f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef72927680 0x55ef7293f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52060/0 (socket says 172.21.15.28:52060) 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 -- 172.21.15.28:0/481364221 learned_addr learned my addr 172.21.15.28:0/481364221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291cdc0 0x55ef7293ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 -- 172.21.15.28:0/481364221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291c4a0 msgr2=0x55ef72933450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dde59c640 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291c4a0 0x55ef72933450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291c4a0 0x55ef72933450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 -- 172.21.15.28:0/481364221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291cdc0 msgr2=0x55ef7293ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291cdc0 0x55ef7293ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 -- 172.21.15.28:0/481364221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef72939b90 con 0x55ef72927680 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291cdc0 0x55ef7293ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dde59c640 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291c4a0 0x55ef72933450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dded9d640 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef72927680 0x55ef7293f320 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80123b0 tx=0x7f3dd8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9b4ac726292b32 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dc7fff640 1 -- 172.21.15.28:0/481364221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8004d20 con 0x55ef72927680 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dc7fff640 1 -- 172.21.15.28:0/481364221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd8010e20 con 0x55ef72927680 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.093+0000 7f3dc7fff640 1 -- 172.21.15.28:0/481364221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80053a0 con 0x55ef72927680 2024-09-15T22:05:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- 172.21.15.28:0/481364221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef72927680 msgr2=0x55ef7293f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef72927680 0x55ef7293f320 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80123b0 tx=0x7f3dd8011000 comp rx=0 tx=0).stop 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- 172.21.15.28:0/481364221 shutdown_connections 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef72927680 0x55ef7293f320 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef7291cdc0 0x55ef7293ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 --2- 172.21.15.28:0/481364221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291c4a0 0x55ef72933450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- 172.21.15.28:0/481364221 >> 172.21.15.28:0/481364221 conn(0x55ef72918a30 msgr2=0x55ef7291f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- 172.21.15.28:0/481364221 shutdown_connections 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- 172.21.15.28:0/481364221 wait complete. 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- start start 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 Processor -- start 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291c4a0 0x55ef7292fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291cdc0 0x55ef7292b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef72927680 0x55ef7292b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01b2670 con 0x55ef7291c4a0 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01b1fa0 con 0x55ef72927680 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.094+0000 7f3de70ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3de01af1d0 con 0x55ef7291cdc0 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3de70ea740 10 client.0 _ll_register_callbacks cb 0x7f3dd4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291c4a0 0x55ef7292fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291c4a0 0x55ef7292fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52070/0 (socket says 172.21.15.28:52070) 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 -- 172.21.15.28:0/3176121868 learned_addr learned my addr 172.21.15.28:0/3176121868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dded9d640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef72927680 0x55ef7292b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291cdc0 0x55ef7292b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 -- 172.21.15.28:0/3176121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291cdc0 msgr2=0x55ef7292b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291cdc0 0x55ef7292b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 -- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef72927680 msgr2=0x55ef7292b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef72927680 0x55ef7292b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 -- 172.21.15.28:0/3176121868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef7292bf80 con 0x55ef7291c4a0 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dded9d640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef72927680 0x55ef7292b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291cdc0 0x55ef7292b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dde59c640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291c4a0 0x55ef7292fe30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3dd009bac0 tx=0x7f3dd00048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb26dc7cdc5e17 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dc5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/3176121868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00b8070 con 0x55ef7291c4a0 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/3176121868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f3dd0009cc0 con 0x55ef7291c4a0 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dd8010e20 con 0x55ef7291c4a0 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/3176121868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f3dd009ce60 con 0x55ef7291c4a0 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3de70ea740 10 client.16857 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.095+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ef729a1e40 con 0x55ef7291c4a0 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/3176121868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dd8010e20 con 0x55ef7291c4a0 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/3176121868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f3dd0086110 con 0x55ef7291c4a0 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3de70ea740 10 client.16857 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3de70ea740 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3de70ea740 20 client.16857 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3de70ea740 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729a2180 0x55ef729a4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3de70ea740 4 client.16857 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ef72889d60 con 0x55ef729a2180 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729a2180 0x55ef729a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3dc5ffb640 10 client.16857 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.096+0000 7f3dddd9b640 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729a2180 0x55ef729a4570 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=912ccedbe4d101c server_cookie=d920c0fa863b671e in_seq=0 out_seq=0 2024-09-15T22:05:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3dc5ffb640 1 -- 172.21.15.28:0/3176121868 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ef72889d60 con 0x55ef729a2180 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3dc5ffb640 10 client.16857 handle_command_reply: tid=0 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 client.16857 shutdown 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291c4a0 msgr2=0x55ef7292fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291c4a0 0x55ef7292fe30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3dd009bac0 tx=0x7f3dd00048a0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 shutdown_connections 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ef729a2180 0x55ef729a4570 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ef72927680 0x55ef7292b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ef7291cdc0 0x55ef7292b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3dc5ffb640 0 client.16857 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 --2- 172.21.15.28:0/3176121868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef7291c4a0 0x55ef7292fe30 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 >> 172.21.15.28:0/3176121868 conn(0x55ef72918a30 msgr2=0x55ef7291def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 shutdown_connections 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 1 -- 172.21.15.28:0/3176121868 wait complete. 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.097+0000 7f3de70ea740 20 client.16857 trim_cache size 0 max 16384 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3dc80a33b0 msgr2=0x7f3dc80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3dc80a33b0 0x7f3dc80a5870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80126d0 tx=0x7f3dd800a290 comp rx=0 tx=0).stop 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0102d40 msgr2=0x7f3de01009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0102d40 0x7f3de01009a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000d770 tx=0x7f3dd000dc30 comp rx=0 tx=0).stop 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 shutdown_connections 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f3dc80a33b0 0x7f3dc80a5870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3de010c170 0x7f3de0101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3de0103660 0x7f3de0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 --2- 172.21.15.28:0/3214479043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de0102d40 0x7f3de01009a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 >> 172.21.15.28:0/3214479043 conn(0x7f3de00fc7a0 msgr2=0x7f3de00fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 shutdown_connections 2024-09-15T22:05:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:04.098+0000 7f3de4d11640 1 -- 172.21.15.28:0/3214479043 wait complete. 2024-09-15T22:05:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:03 smithi174 ceph-mon[35244]: pgmap v236: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 28 KiB/s wr, 80 op/s 2024-09-15T22:05:04.994 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:05:04.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:05:05.254 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 -- 172.21.15.28:0/1267221603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd010e480 msgr2=0x7f2cd01108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/1267221603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd010e480 0x7f2cd01108b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8009ca0 tx=0x7f2cc802a8c0 comp rx=0 tx=0).stop 2024-09-15T22:05:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 -- 172.21.15.28:0/1267221603 shutdown_connections 2024-09-15T22:05:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/1267221603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd010e480 0x7f2cd01108b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/1267221603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cd0075110 0x7f2cd010deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/1267221603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cd00747f0 0x7f2cd0074bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 -- 172.21.15.28:0/1267221603 >> 172.21.15.28:0/1267221603 conn(0x7f2cd00fd8d0 msgr2=0x7f2cd00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 -- 172.21.15.28:0/1267221603 shutdown_connections 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.793+0000 7f2cd73e7640 1 -- 172.21.15.28:0/1267221603 wait complete. 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 Processor -- start 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 -- start start 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cd00747f0 0x7f2cd019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd0075110 0x7f2cd019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cd010e480 0x7f2cd019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01141f0 con 0x7f2cd0075110 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd0114070 con 0x7f2cd010e480 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd73e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd0114370 con 0x7f2cd00747f0 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd4982640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd0075110 0x7f2cd019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd4982640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd0075110 0x7f2cd019aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52094/0 (socket says 172.21.15.28:52094) 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd4982640 1 -- 172.21.15.28:0/249346145 learned_addr learned my addr 172.21.15.28:0/249346145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd5984640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cd010e480 0x7f2cd019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.794+0000 7f2cd5183640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cd00747f0 0x7f2cd019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd4982640 1 -- 172.21.15.28:0/249346145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cd00747f0 msgr2=0x7f2cd019a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd4982640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cd00747f0 0x7f2cd019a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd4982640 1 -- 172.21.15.28:0/249346145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cd010e480 msgr2=0x7f2cd019ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd4982640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cd010e480 0x7f2cd019ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd4982640 1 -- 172.21.15.28:0/249346145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd019f560 con 0x7f2cd0075110 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd4982640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd0075110 0x7f2cd019aaf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000c890 tx=0x7f2cc000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cc67fc640 1 -- 172.21.15.28:0/249346145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0004220 con 0x7f2cd0075110 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cc67fc640 1 -- 172.21.15.28:0/249346145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc00043c0 con 0x7f2cd0075110 2024-09-15T22:05:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd019f850 con 0x7f2cd0075110 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cc67fc640 1 -- 172.21.15.28:0/249346145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc00049e0 con 0x7f2cd0075110 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.795+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd01a7130 con 0x7f2cd0075110 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.796+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cd0074bd0 con 0x7f2cd0075110 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.797+0000 7f2cc67fc640 1 -- 172.21.15.28:0/249346145 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f2cc00079b0 con 0x7f2cd0075110 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.798+0000 7f2cc67fc640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2cbc0a33b0 0x7f2cbc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.798+0000 7f2cd5183640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2cbc0a33b0 0x7f2cbc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.798+0000 7f2cc67fc640 1 -- 172.21.15.28:0/249346145 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f2cc00d02f0 con 0x7f2cd0075110 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.798+0000 7f2cd5183640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2cbc0a33b0 0x7f2cbc0a5870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc005e10 tx=0x7f2ccc005da0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:05.801+0000 7f2cc67fc640 1 -- 172.21.15.28:0/249346145 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2cc0014030 con 0x7f2cd0075110 2024-09-15T22:05:06.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.014+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cd019b910 con 0x7f2cd0075110 2024-09-15T22:05:06.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:05 smithi028 ceph-mon[30969]: pgmap v237: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-09-15T22:05:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.014+0000 7f2cc67fc640 1 -- 172.21.15.28:0/249346145 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11988 (secure 0 0 0) 0x7f2cc009c920 con 0x7f2cd0075110 2024-09-15T22:05:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:05:06.017 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"btime":"2024-09-15T22:04:37:022811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:04:36.713632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:05:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-09-15T22:05:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.016+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2cbc0a33b0 msgr2=0x7f2cbc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.016+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2cbc0a33b0 0x7f2cbc0a5870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc005e10 tx=0x7f2ccc005da0 comp rx=0 tx=0).stop 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.016+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd0075110 msgr2=0x7f2cd019aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.016+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd0075110 0x7f2cd019aaf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000c890 tx=0x7f2cc000cd50 comp rx=0 tx=0).stop 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 shutdown_connections 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2cbc0a33b0 0x7f2cbc0a5870 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cd010e480 0x7f2cd019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd0075110 0x7f2cd019aaf0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 --2- 172.21.15.28:0/249346145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cd00747f0 0x7f2cd019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 >> 172.21.15.28:0/249346145 conn(0x7f2cd00fd8d0 msgr2=0x7f2cd00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 shutdown_connections 2024-09-15T22:05:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:06.017+0000 7f2cd73e7640 1 -- 172.21.15.28:0/249346145 wait complete. 2024-09-15T22:05:06.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:05 smithi203 ceph-mon[35546]: pgmap v237: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-09-15T22:05:06.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:05 smithi174 ceph-mon[35244]: pgmap v237: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-09-15T22:05:06.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:05:06.914 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:06.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:06 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/249346145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:05:07.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:06 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/249346145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:05:07.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:06 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/249346145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:05:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 -- 172.21.15.28:0/2140469436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8100e20 msgr2=0x7f6ab8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 --2- 172.21.15.28:0/2140469436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8100e20 0x7f6ab8101200 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009900 tx=0x7f6aa802a480 comp rx=0 tx=0).stop 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 -- 172.21.15.28:0/2140469436 shutdown_connections 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 --2- 172.21.15.28:0/2140469436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ab8102ef0 0x7f6ab810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 --2- 172.21.15.28:0/2140469436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab8102570 0x7f6ab81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 --2- 172.21.15.28:0/2140469436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8100e20 0x7f6ab8101200 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 -- 172.21.15.28:0/2140469436 >> 172.21.15.28:0/2140469436 conn(0x7f6ab80fc7d0 msgr2=0x7f6ab80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 -- 172.21.15.28:0/2140469436 shutdown_connections 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.418+0000 7f6abe576640 1 -- 172.21.15.28:0/2140469436 wait complete. 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 Processor -- start 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 -- start start 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab8100e20 0x7f6ab80fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8102570 0x7f6ab80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ab8102ef0 0x7f6ab80fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab8110f50 con 0x7f6ab8102570 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab8110dd0 con 0x7f6ab8102ef0 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abe576640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab81110d0 con 0x7f6ab8100e20 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8102570 0x7f6ab80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8102570 0x7f6ab80fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52124/0 (socket says 172.21.15.28:52124) 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6aaffff640 1 -- 172.21.15.28:0/556040524 learned_addr learned my addr 172.21.15.28:0/556040524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab8100e20 0x7f6ab80fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.419+0000 7f6abcb13640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ab8102ef0 0x7f6ab80fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6aaffff640 1 -- 172.21.15.28:0/556040524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab8100e20 msgr2=0x7f6ab80fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6aaffff640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab8100e20 0x7f6ab80fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6aaffff640 1 -- 172.21.15.28:0/556040524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ab8102ef0 msgr2=0x7f6ab80fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6aaffff640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ab8102ef0 0x7f6ab80fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6aaffff640 1 -- 172.21.15.28:0/556040524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab81a6a20 con 0x7f6ab8102570 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6abcb13640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ab8102ef0 0x7f6ab80fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6aaffff640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8102570 0x7f6ab80fd1c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000c890 tx=0x7f6aa000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6ab5ffb640 1 -- 172.21.15.28:0/556040524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0004220 con 0x7f6ab8102570 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab81a6d10 con 0x7f6ab8102570 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6ab5ffb640 1 -- 172.21.15.28:0/556040524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6aa00043c0 con 0x7f6ab8102570 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6ab5ffb640 1 -- 172.21.15.28:0/556040524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0004a00 con 0x7f6ab8102570 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.420+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab81a7240 con 0x7f6ab8102570 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.422+0000 7f6ab5ffb640 1 -- 172.21.15.28:0/556040524 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6aa00079b0 con 0x7f6ab8102570 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 Processor -- start 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 -- start start 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80186090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca801865d0 0x55ca80186a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80191370 0x55ca80193760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca800ec2f0 con 0x55ca801865d0 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ca80093cf0 con 0x55ca80187b40 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ac094f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ca800e82b0 con 0x55ca80191370 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ab5ffb640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6aa40a33b0 0x7f6aa40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ab7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca801865d0 0x55ca80186a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ab7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca801865d0 0x55ca80186a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52132/0 (socket says 172.21.15.28:52132) 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ab7fff640 1 -- 172.21.15.28:0/728133842 learned_addr learned my addr 172.21.15.28:0/728133842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6aaffff640 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80191370 0x55ca80193760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6abcb13640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6aa40a33b0 0x7f6aa40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6ab5ffb640 1 -- 172.21.15.28:0/556040524 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6aa00cfdd0 con 0x7f6ab8102570 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.423+0000 7f6abcb13640 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80186090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6ab7fff640 1 -- 172.21.15.28:0/728133842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80191370 msgr2=0x55ca80193760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80191370 0x55ca80193760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6ab7fff640 1 -- 172.21.15.28:0/728133842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 msgr2=0x55ca80186090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80186090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6ab7fff640 1 -- 172.21.15.28:0/728133842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca800783a0 con 0x55ca801865d0 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca801865d0 0x55ca80186a10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009900 tx=0x7f6aa8004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806090ca8603c943 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6abcb13640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6aa40a33b0 0x7f6aa40a5870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009370 tx=0x7f6ab0009590 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6aae7fc640 1 -- 172.21.15.28:0/728133842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8035030 con 0x55ca801865d0 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6aae7fc640 1 -- 172.21.15.28:0/728133842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6aa8040070 con 0x55ca801865d0 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.424+0000 7f6aae7fc640 1 -- 172.21.15.28:0/728133842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8031b10 con 0x55ca801865d0 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 -- 172.21.15.28:0/728133842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca801865d0 msgr2=0x55ca80186a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca801865d0 0x55ca80186a10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009900 tx=0x7f6aa8004170 comp rx=0 tx=0).stop 2024-09-15T22:05:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 -- 172.21.15.28:0/728133842 shutdown_connections 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80191370 0x55ca80193760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca801865d0 0x55ca80186a10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 --2- 172.21.15.28:0/728133842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80186090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 -- 172.21.15.28:0/728133842 >> 172.21.15.28:0/728133842 conn(0x55ca8014f310 msgr2=0x55ca80182c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 -- 172.21.15.28:0/728133842 shutdown_connections 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 -- 172.21.15.28:0/728133842 wait complete. 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 -- start start 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.425+0000 7f6ac094f740 1 Processor -- start 2024-09-15T22:05:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca801865d0 0x55ca80190d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80187b40 0x55ca8018dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca8018e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ac094f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca80194010 con 0x55ca80191370 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ac094f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ca800a7060 con 0x55ca801865d0 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ac094f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ca80194190 con 0x55ca80187b40 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ac094f740 10 client.0 _ll_register_callbacks cb 0x55ca7fdfa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6aaffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80187b40 0x55ca8018dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6aaffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80187b40 0x55ca8018dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:41206/0 (socket says 172.21.15.28:41206) 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6aaffff640 1 -- 172.21.15.28:0/2187644712 learned_addr learned my addr 172.21.15.28:0/2187644712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6abcb13640 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca8018e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.426+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca801865d0 0x55ca80190d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6aaffff640 1 -- 172.21.15.28:0/2187644712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca801865d0 msgr2=0x55ca80190d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6aaffff640 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca801865d0 0x55ca80190d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6aaffff640 1 -- 172.21.15.28:0/2187644712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 msgr2=0x55ca8018e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6aaffff640 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca8018e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6aaffff640 1 -- 172.21.15.28:0/2187644712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca7fe5d3c0 con 0x55ca80187b40 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6aaffff640 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80187b40 0x55ca8018dfc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0098160 tx=0x7f6aa009b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a609ab54f17abd13 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6a977fe640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6a977fe640 1 -- 172.21.15.28:0/2187644712 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa001d030 con 0x55ca80187b40 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6a977fe640 1 -- 172.21.15.28:0/2187644712 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6aa009b150 con 0x55ca80187b40 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6aa8040070 con 0x55ca80187b40 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6a977fe640 1 -- 172.21.15.28:0/2187644712 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6aa009bd00 con 0x55ca80187b40 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6ac094f740 10 client.26317 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.427+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ca801b0a70 con 0x55ca80187b40 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6a977fe640 1 -- 172.21.15.28:0/2187644712 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6aa8040070 con 0x55ca80187b40 2024-09-15T22:05:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6a977fe640 1 -- 172.21.15.28:0/2187644712 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f6aa009abc0 con 0x55ca80187b40 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6ac094f740 10 client.26317 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6ac094f740 10 client.26317 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6ac094f740 20 client.26317 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6ac094f740 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca801b1ed0 0x55ca801b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6ac094f740 4 client.26317 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca800f9860 con 0x55ca801b1ed0 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.428+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca801b1ed0 0x55ca801b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.429+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca801b1ed0 0x55ca801b42c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=58538274c3e6f605 server_cookie=a80e372dda5af539 in_seq=0 out_seq=0 2024-09-15T22:05:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.429+0000 7f6a977fe640 10 client.26317 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6a977fe640 1 -- 172.21.15.28:0/2187644712 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca800f9860 con 0x55ca801b1ed0 2024-09-15T22:05:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6a977fe640 10 client.26317 handle_command_reply: tid=0 2024-09-15T22:05:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 client.26317 shutdown 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80187b40 msgr2=0x55ca8018dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80187b40 0x55ca8018dfc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0098160 tx=0x7f6aa009b840 comp rx=0 tx=0).stop 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 shutdown_connections 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca801b1ed0 0x55ca801b42c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca8018e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca80187b40 0x55ca8018dfc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6a977fe640 0 client.26317 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 --2- 172.21.15.28:0/2187644712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca801865d0 0x55ca80190d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 >> 172.21.15.28:0/2187644712 conn(0x55ca8014f310 msgr2=0x55ca80188930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 shutdown_connections 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 1 -- 172.21.15.28:0/2187644712 wait complete. 2024-09-15T22:05:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.430+0000 7f6ac094f740 20 client.26317 trim_cache size 0 max 16384 2024-09-15T22:05:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 Processor -- start 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 -- start start 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80182e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80187b40 0x55ca801b9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80191370 0x55ca801ba080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab81a8a60 con 0x55ca80187b40 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab81aa5d0 con 0x55ca80191370 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6ac094f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab81a9130 con 0x55ca801865d0 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80187b40 0x55ca801b9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80187b40 0x55ca801b9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52148/0 (socket says 172.21.15.28:52148) 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.443+0000 7f6aaffff640 1 -- 172.21.15.28:0/4134740839 learned_addr learned my addr 172.21.15.28:0/4134740839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80182e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6abcb13640 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80191370 0x55ca801ba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aaffff640 1 -- 172.21.15.28:0/4134740839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 msgr2=0x55ca80182e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aaffff640 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80182e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aaffff640 1 -- 172.21.15.28:0/4134740839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80191370 msgr2=0x55ca801ba080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aaffff640 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80191370 0x55ca801ba080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aaffff640 1 -- 172.21.15.28:0/4134740839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca80189120 con 0x55ca80187b40 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6abcb13640 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80191370 0x55ca801ba080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aaffff640 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80187b40 0x55ca801b9b40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00986d0 tx=0x7f6aa009b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e8a942f7de908b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aadffb640 1 -- 172.21.15.28:0/4134740839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa001d040 con 0x55ca80187b40 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aadffb640 1 -- 172.21.15.28:0/4134740839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6aa00a1240 con 0x55ca80187b40 2024-09-15T22:05:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.444+0000 7f6aadffb640 1 -- 172.21.15.28:0/4134740839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa009b150 con 0x55ca80187b40 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 -- 172.21.15.28:0/4134740839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80187b40 msgr2=0x55ca801b9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80187b40 0x55ca801b9b40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00986d0 tx=0x7f6aa009b970 comp rx=0 tx=0).stop 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 -- 172.21.15.28:0/4134740839 shutdown_connections 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80191370 0x55ca801ba080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80187b40 0x55ca801b9b40 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 --2- 172.21.15.28:0/4134740839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80182e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 -- 172.21.15.28:0/4134740839 >> 172.21.15.28:0/4134740839 conn(0x55ca8014f310 msgr2=0x55ca80182700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 -- 172.21.15.28:0/4134740839 shutdown_connections 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 -- 172.21.15.28:0/4134740839 wait complete. 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 -- start start 2024-09-15T22:05:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.445+0000 7f6ac094f740 1 Processor -- start 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80199790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ac094f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca801965f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ac094f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab81a9f00 con 0x55ca80191370 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ac094f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab81a9830 con 0x55ca80187b40 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ac094f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab80739a0 con 0x55ca801865d0 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ac094f740 10 client.0 _ll_register_callbacks cb 0x7f6aa0008f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca801965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca801965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52162/0 (socket says 172.21.15.28:52162) 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 -- 172.21.15.28:0/3867004799 learned_addr learned my addr 172.21.15.28:0/3867004799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6aaffff640 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 -- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 msgr2=0x55ca80199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 -- 172.21.15.28:0/3867004799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 msgr2=0x55ca80199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 -- 172.21.15.28:0/3867004799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca80196e20 con 0x55ca80191370 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6abcb13640 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca801965f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000fa80 tx=0x7f6ab0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9047f56607628106 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6a977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.446+0000 7f6a977fe640 1 -- 172.21.15.28:0/3867004799 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0012ae0 con 0x55ca80191370 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6a977fe640 1 -- 172.21.15.28:0/3867004799 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6ab0012c80 con 0x55ca80191370 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6aa00a1240 con 0x55ca80191370 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6a977fe640 1 -- 172.21.15.28:0/3867004799 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6ab0033050 con 0x55ca80191370 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 10 client.16881 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ca8021d6f0 con 0x55ca80191370 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6a977fe640 1 -- 172.21.15.28:0/3867004799 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6aa00a1240 con 0x55ca80191370 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6a977fe640 1 -- 172.21.15.28:0/3867004799 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f6ab0027070 con 0x55ca80191370 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 10 client.16881 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 10 client.16881 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 20 client.16881 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca8021de90 0x55ca802202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 4 client.16881 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca800f3d70 con 0x55ca8021de90 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.447+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca8021de90 0x55ca802202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ab7fff640 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca8021de90 0x55ca802202b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4478426254bca328 server_cookie=10aebd553d65cdfb in_seq=0 out_seq=0 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6a977fe640 10 client.16881 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6a977fe640 1 -- 172.21.15.28:0/3867004799 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca800f3d70 con 0x55ca8021de90 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6a977fe640 10 client.16881 handle_command_reply: tid=0 2024-09-15T22:05:07.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 client.16881 shutdown 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 msgr2=0x55ca801965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca801965f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000fa80 tx=0x7f6ab0004ab0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 shutdown_connections 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x55ca8021de90 0x55ca802202b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca80191370 0x55ca801965f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ca80187b40 0x55ca80199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 --2- 172.21.15.28:0/3867004799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ca801865d0 0x55ca80199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 >> 172.21.15.28:0/3867004799 conn(0x55ca8014f310 msgr2=0x55ca801814e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6a977fe640 0 client.16881 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 shutdown_connections 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 1 -- 172.21.15.28:0/3867004799 wait complete. 2024-09-15T22:05:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.448+0000 7f6ac094f740 20 client.16881 trim_cache size 0 max 16384 2024-09-15T22:05:07.456 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6aa40a33b0 msgr2=0x7f6aa40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6aa40a33b0 0x7f6aa40a5870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009370 tx=0x7f6ab0009590 comp rx=0 tx=0).stop 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8102570 msgr2=0x7f6ab80fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8102570 0x7f6ab80fd1c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000c890 tx=0x7f6aa000cd50 comp rx=0 tx=0).stop 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 shutdown_connections 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6aa40a33b0 0x7f6aa40a5870 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ab8102ef0 0x7f6ab80fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab8102570 0x7f6ab80fd1c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 --2- 172.21.15.28:0/556040524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab8100e20 0x7f6ab80fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 >> 172.21.15.28:0/556040524 conn(0x7f6ab80fc7d0 msgr2=0x7f6ab806a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 shutdown_connections 2024-09-15T22:05:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:07.449+0000 7f6abe576640 1 -- 172.21.15.28:0/556040524 wait complete. 2024-09-15T22:05:08.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:07 smithi028 ceph-mon[30969]: pgmap v238: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-15T22:05:08.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:05:08.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:07 smithi203 ceph-mon[35546]: pgmap v238: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-15T22:05:08.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:07 smithi174 ceph-mon[35244]: pgmap v238: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-09-15T22:05:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3547398966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34101de0 msgr2=0x7f6a341021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3547398966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34101de0 0x7f6a341021c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009900 tx=0x7f6a2402a410 comp rx=0 tx=0).stop 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3547398966 shutdown_connections 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3547398966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a34106f10 0x7f6a3410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3547398966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a34102790 0x7f6a341067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3547398966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34101de0 0x7f6a341021c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3547398966 >> 172.21.15.28:0/3547398966 conn(0x7f6a340fd8c0 msgr2=0x7f6a340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3547398966 shutdown_connections 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.951+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3547398966 wait complete. 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 Processor -- start 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 -- start start 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a34101de0 0x7f6a341983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a34102790 0x7f6a34198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34106f10 0x7f6a3419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a34111f70 con 0x7f6a34106f10 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a34111df0 con 0x7f6a34101de0 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a3a97a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341120f0 con 0x7f6a34102790 2024-09-15T22:05:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a38f17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34106f10 0x7f6a3419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a38f17640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34106f10 0x7f6a3419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52170/0 (socket says 172.21.15.28:52170) 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a38f17640 1 -- 172.21.15.28:0/3978144128 learned_addr learned my addr 172.21.15.28:0/3978144128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.952+0000 7f6a33fff640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a34101de0 0x7f6a341983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a337fe640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a34102790 0x7f6a34198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a38f17640 1 -- 172.21.15.28:0/3978144128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a34102790 msgr2=0x7f6a34198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a38f17640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a34102790 0x7f6a34198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a38f17640 1 -- 172.21.15.28:0/3978144128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a34101de0 msgr2=0x7f6a341983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a38f17640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a34101de0 0x7f6a341983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a38f17640 1 -- 172.21.15.28:0/3978144128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3419d380 con 0x7f6a34106f10 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a33fff640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a34101de0 0x7f6a341983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a38f17640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34106f10 0x7f6a3419cca0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00cce0 tx=0x7f6a2c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a317fa640 1 -- 172.21.15.28:0/3978144128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00f450 con 0x7f6a34106f10 2024-09-15T22:05:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3419d670 con 0x7f6a34106f10 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a317fa640 1 -- 172.21.15.28:0/3978144128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c00fad0 con 0x7f6a34106f10 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a317fa640 1 -- 172.21.15.28:0/3978144128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c018c10 con 0x7f6a34106f10 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.953+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a341a4f50 con 0x7f6a34106f10 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.956+0000 7f6a317fa640 1 -- 172.21.15.28:0/3978144128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6a2c00f5f0 con 0x7f6a34106f10 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.956+0000 7f6a3cd53740 1 Processor -- start 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a3cd53740 1 -- start start 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cbd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbd5d0 0x555df0cbda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555df0c232f0 con 0x555df0cbeb40 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555df0bcacf0 con 0x555df0cc8370 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555df0c1f2b0 con 0x555df0cbd5d0 2024-09-15T22:05:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cbd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cbd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52186/0 (socket says 172.21.15.28:52186) 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbd5d0 0x555df0cbda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a337fe640 1 -- 172.21.15.28:0/1074878896 learned_addr learned my addr 172.21.15.28:0/1074878896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a337fe640 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a317fa640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a33fff640 1 -- 172.21.15.28:0/1074878896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 msgr2=0x555df0cca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a33fff640 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a33fff640 1 -- 172.21.15.28:0/1074878896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 msgr2=0x555df0cbd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a33fff640 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cbd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a33fff640 1 -- 172.21.15.28:0/1074878896 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df0baf3a0 con 0x555df0cbd5d0 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a317fa640 1 -- 172.21.15.28:0/3978144128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6a2c0d06a0 con 0x7f6a34106f10 2024-09-15T22:05:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.957+0000 7f6a38f17640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.958+0000 7f6a33fff640 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbd5d0 0x555df0cbda10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6a24005940 tx=0x7f6a24031630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d670b81baae3ffc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.958+0000 7f6a38f17640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0cdaa0 tx=0x7f6a2c06e350 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.958+0000 7f6a1a7fc640 1 -- 172.21.15.28:0/1074878896 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24035040 con 0x555df0cbd5d0 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.958+0000 7f6a1a7fc640 1 -- 172.21.15.28:0/1074878896 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a24005020 con 0x555df0cbd5d0 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.958+0000 7f6a1a7fc640 1 -- 172.21.15.28:0/1074878896 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24004a00 con 0x555df0cbd5d0 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1074878896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbd5d0 msgr2=0x555df0cbda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbd5d0 0x555df0cbda10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6a24005940 tx=0x7f6a24031630 comp rx=0 tx=0).stop 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1074878896 shutdown_connections 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbd5d0 0x555df0cbda10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1074878896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cbd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1074878896 >> 172.21.15.28:0/1074878896 conn(0x555df0c86310 msgr2=0x555df0cb9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1074878896 shutdown_connections 2024-09-15T22:05:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.959+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1074878896 wait complete. 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 -- start start 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 Processor -- start 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbd5d0 0x555df0cc7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 0x555df0cc4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cc5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555df0ccb010 con 0x555df0cbd5d0 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555df0bde060 con 0x555df0cc8370 2024-09-15T22:05:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555df0ccb190 con 0x555df0cbeb40 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a3cd53740 10 client.0 _ll_register_callbacks cb 0x555df09318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.960+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbd5d0 0x555df0cc7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 0x555df0cc4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbd5d0 0x555df0cc7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52194/0 (socket says 172.21.15.28:52194) 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 -- 172.21.15.28:0/97240894 learned_addr learned my addr 172.21.15.28:0/97240894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cc5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 -- 172.21.15.28:0/97240894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 msgr2=0x555df0cc4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 0x555df0cc4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 -- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 msgr2=0x555df0cc5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cc5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 -- 172.21.15.28:0/97240894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555df09943c0 con 0x555df0cbd5d0 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a337fe640 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbd5d0 0x555df0cc7c50 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800bee0 tx=0x7f6a28014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0b1e044195c7da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a137fe640 1 -- 172.21.15.28:0/97240894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28012920 con 0x555df0cbd5d0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a24005020 con 0x555df0cbd5d0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.961+0000 7f6a137fe640 1 -- 172.21.15.28:0/97240894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6a28012ac0 con 0x555df0cbd5d0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a137fe640 1 -- 172.21.15.28:0/97240894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6a28012cc0 con 0x555df0cbd5d0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 10 client.16893 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555df0cd69c0 con 0x555df0cbd5d0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a137fe640 1 -- 172.21.15.28:0/97240894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a24005020 con 0x555df0cbd5d0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a137fe640 1 -- 172.21.15.28:0/97240894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f6a2800f3f0 con 0x555df0cbd5d0 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 10 client.16893 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 10 client.16893 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 20 client.16893 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0cd7e80 0x555df0cda270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 4 client.16893 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:08.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555df0c30860 con 0x555df0cd7e80 2024-09-15T22:05:08.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.962+0000 7f6a33fff640 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0cd7e80 0x555df0cda270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.963+0000 7f6a33fff640 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0cd7e80 0x555df0cda270 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=70216e5705222569 server_cookie=760f9c0a1d5dfa3c in_seq=0 out_seq=0 2024-09-15T22:05:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.963+0000 7f6a137fe640 10 client.16893 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.963+0000 7f6a137fe640 1 -- 172.21.15.28:0/97240894 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555df0c30860 con 0x555df0cd7e80 2024-09-15T22:05:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.963+0000 7f6a137fe640 10 client.16893 handle_command_reply: tid=0 2024-09-15T22:05:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.963+0000 7f6a3cd53740 1 client.16893 shutdown 2024-09-15T22:05:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbd5d0 msgr2=0x555df0cc7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbd5d0 0x555df0cc7c50 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800bee0 tx=0x7f6a28014040 comp rx=0 tx=0).stop 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 shutdown_connections 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cc8370 0x555df0cc5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 0x555df0cc4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0cd7e80 0x555df0cda270 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/97240894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbd5d0 0x555df0cc7c50 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 >> 172.21.15.28:0/97240894 conn(0x555df0c86310 msgr2=0x555df0cbf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a137fe640 0 client.16893 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 shutdown_connections 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 1 -- 172.21.15.28:0/97240894 wait complete. 2024-09-15T22:05:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.964+0000 7f6a3cd53740 20 client.16893 trim_cache size 0 max 16384 2024-09-15T22:05:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 Processor -- start 2024-09-15T22:05:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 -- start start 2024-09-15T22:05:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 0x555df0cd42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 0x555df0cd9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cc8370 0x555df0cc9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341aef60 con 0x555df0cc8370 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341b0ad0 con 0x555df0cbd5d0 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341af630 con 0x555df0cbeb40 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.977+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cc8370 0x555df0cc9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cc8370 0x555df0cc9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52196/0 (socket says 172.21.15.28:52196) 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 -- 172.21.15.28:0/4138208553 learned_addr learned my addr 172.21.15.28:0/4138208553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 -- 172.21.15.28:0/4138208553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 msgr2=0x555df0cd9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 --2- 172.21.15.28:0/4138208553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 0x555df0cd9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 -- 172.21.15.28:0/4138208553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 msgr2=0x555df0cd42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 --2- 172.21.15.28:0/4138208553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 0x555df0cd42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 -- 172.21.15.28:0/4138208553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df0cb4f90 con 0x555df0cc8370 2024-09-15T22:05:08.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a337fe640 1 --2- 172.21.15.28:0/4138208553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cc8370 0x555df0cc9760 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6a28005fb0 tx=0x7f6a280045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636a266b8e59cbcb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:08.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a19ffb640 1 -- 172.21.15.28:0/4138208553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28012e60 con 0x555df0cc8370 2024-09-15T22:05:08.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a19ffb640 1 -- 172.21.15.28:0/4138208553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2800f3f0 con 0x555df0cc8370 2024-09-15T22:05:08.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.978+0000 7f6a19ffb640 1 -- 172.21.15.28:0/4138208553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28021da0 con 0x555df0cc8370 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 -- 172.21.15.28:0/4138208553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cc8370 msgr2=0x555df0cc9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/4138208553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cc8370 0x555df0cc9760 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6a28005fb0 tx=0x7f6a280045a0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 -- 172.21.15.28:0/4138208553 shutdown_connections 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/4138208553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cc8370 0x555df0cc9760 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/4138208553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cbeb40 0x555df0cd9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/4138208553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 0x555df0cd42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 -- 172.21.15.28:0/4138208553 >> 172.21.15.28:0/4138208553 conn(0x555df0c86310 msgr2=0x555df0cc37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 -- 172.21.15.28:0/4138208553 shutdown_connections 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 -- 172.21.15.28:0/4138208553 wait complete. 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.979+0000 7f6a3cd53740 1 -- start start 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 1 Processor -- start 2024-09-15T22:05:08.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 0x555df0cdf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cdf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cc8370 0x555df0ccf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341b0400 con 0x555df0cbeb40 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341afd30 con 0x555df0cbd5d0 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6a341acf60 con 0x555df0cc8370 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a3cd53740 10 client.0 _ll_register_callbacks cb 0x7f6a2800bc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cdf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cdf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52200/0 (socket says 172.21.15.28:52200) 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 -- 172.21.15.28:0/1429638718 learned_addr learned my addr 172.21.15.28:0/1429638718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a337fe640 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cc8370 0x555df0ccf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a337fe640 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 0x555df0cdf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 -- 172.21.15.28:0/1429638718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cc8370 msgr2=0x555df0ccf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cc8370 0x555df0ccf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 -- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 msgr2=0x555df0cdf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 0x555df0cdf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 -- 172.21.15.28:0/1429638718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555df0ccf960 con 0x555df0cbeb40 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a33fff640 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cdf630 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6a24031550 tx=0x7f6a24000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13b3cfbab57a9e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.980+0000 7f6a137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a137fe640 1 -- 172.21.15.28:0/1429638718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24056020 con 0x555df0cbeb40 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a137fe640 1 -- 172.21.15.28:0/1429638718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6a24031a80 con 0x555df0cbeb40 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a2800f3f0 con 0x555df0cbeb40 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a137fe640 1 -- 172.21.15.28:0/1429638718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6a24061050 con 0x555df0cbeb40 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 10 client.16899 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:08.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555df0cd0c60 con 0x555df0cbeb40 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a137fe640 1 -- 172.21.15.28:0/1429638718 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2800f3f0 con 0x555df0cbeb40 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a137fe640 1 -- 172.21.15.28:0/1429638718 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f6a24046450 con 0x555df0cbeb40 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 10 client.16899 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 10 client.16899 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 20 client.16899 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0d42b10 0x555df0d44f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 4 client.16899 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.981+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555df0c2ad70 con 0x555df0d42b10 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a337fe640 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0d42b10 0x555df0d44f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a337fe640 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0d42b10 0x555df0d44f00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=da8cc00c32f959c8 server_cookie=e286f5320a7f30f3 in_seq=0 out_seq=0 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a137fe640 10 client.16899 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:08.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a137fe640 1 -- 172.21.15.28:0/1429638718 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555df0c2ad70 con 0x555df0d42b10 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a137fe640 10 client.16899 handle_command_reply: tid=0 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a3cd53740 1 client.16899 shutdown 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 msgr2=0x555df0cdf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cdf630 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6a24031550 tx=0x7f6a24000bc0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 shutdown_connections 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555df0cc8370 0x555df0ccf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x555df0d42b10 0x555df0d44f00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.982+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555df0cbeb40 0x555df0cdf630 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3cd53740 1 --2- 172.21.15.28:0/1429638718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555df0cbd5d0 0x555df0cdf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 >> 172.21.15.28:0/1429638718 conn(0x555df0c86310 msgr2=0x555df0cb84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a137fe640 0 client.16899 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 shutdown_connections 2024-09-15T22:05:08.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3cd53740 1 -- 172.21.15.28:0/1429638718 wait complete. 2024-09-15T22:05:08.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3cd53740 20 client.16899 trim_cache size 0 max 16384 2024-09-15T22:05:08.991 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:08.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6a1c0a33b0 msgr2=0x7f6a1c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0cdaa0 tx=0x7f6a2c06e350 comp rx=0 tx=0).stop 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34106f10 msgr2=0x7f6a3419cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34106f10 0x7f6a3419cca0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00cce0 tx=0x7f6a2c00a620 comp rx=0 tx=0).stop 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 shutdown_connections 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6a1c0a33b0 0x7f6a1c0a5870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a34106f10 0x7f6a3419cca0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a34102790 0x7f6a34198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 --2- 172.21.15.28:0/3978144128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a34101de0 0x7f6a341983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 >> 172.21.15.28:0/3978144128 conn(0x7f6a340fd8c0 msgr2=0x7f6a340febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 shutdown_connections 2024-09-15T22:05:08.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:08.983+0000 7f6a3a97a640 1 -- 172.21.15.28:0/3978144128 wait complete. 2024-09-15T22:05:09.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:09 smithi028 ceph-mon[30969]: pgmap v239: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-15T22:05:09.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:05:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:10.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:09 smithi203 ceph-mon[35546]: pgmap v239: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-15T22:05:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:09 smithi174 ceph-mon[35244]: pgmap v239: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 -- 172.21.15.28:0/3810583905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 msgr2=0x7f941410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 --2- 172.21.15.28:0/3810583905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941410ec40 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f940c009cd0 tx=0x7f940c02a910 comp rx=0 tx=0).stop 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 -- 172.21.15.28:0/3810583905 shutdown_connections 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 --2- 172.21.15.28:0/3810583905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941410ec40 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 --2- 172.21.15.28:0/3810583905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94141036e0 0x7f9414107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 --2- 172.21.15.28:0/3810583905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9414102d30 0x7f9414103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 -- 172.21.15.28:0/3810583905 >> 172.21.15.28:0/3810583905 conn(0x7f94140fc7d0 msgr2=0x7f94140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 -- 172.21.15.28:0/3810583905 shutdown_connections 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 -- 172.21.15.28:0/3810583905 wait complete. 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.712+0000 7f941a006640 1 Processor -- start 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f941a006640 1 -- start start 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f941a006640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9414102d30 0x7f9414198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f941a006640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94141036e0 0x7f9414198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f941a006640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f941a006640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94141120e0 con 0x7f94141082a0 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f941a006640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9414111f60 con 0x7f9414102d30 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f941a006640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9414112260 con 0x7f94141036e0 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f9413fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f9413fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941419cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40344/0 (socket says 172.21.15.28:40344) 2024-09-15T22:05:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f9413fff640 1 -- 172.21.15.28:0/47066237 learned_addr learned my addr 172.21.15.28:0/47066237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f9412ffd640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94141036e0 0x7f9414198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.713+0000 7f94137fe640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9414102d30 0x7f9414198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9413fff640 1 -- 172.21.15.28:0/47066237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94141036e0 msgr2=0x7f9414198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9413fff640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94141036e0 0x7f9414198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9413fff640 1 -- 172.21.15.28:0/47066237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9414102d30 msgr2=0x7f9414198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9413fff640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9414102d30 0x7f9414198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9413fff640 1 -- 172.21.15.28:0/47066237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941419d4d0 con 0x7f94141082a0 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9413fff640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941419cdf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f940c005cc0 tx=0x7f940c031460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9410ff9640 1 -- 172.21.15.28:0/47066237 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c031850 con 0x7f94141082a0 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941419d760 con 0x7f94141082a0 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9410ff9640 1 -- 172.21.15.28:0/47066237 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f940c004520 con 0x7f94141082a0 2024-09-15T22:05:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f9410ff9640 1 -- 172.21.15.28:0/47066237 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c02fcd0 con 0x7f94141082a0 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.714+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94141a50a0 con 0x7f94141082a0 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.716+0000 7f941c3df740 1 Processor -- start 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.716+0000 7f9410ff9640 1 -- 172.21.15.28:0/47066237 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f940c0046d0 con 0x7f94141082a0 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f941c3df740 1 -- start start 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f941c3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d4193090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f941c3df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d4193a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f941c3df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 0x5645d41a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f941c3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5645d40f92f0 con 0x5645d4194b40 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f941c3df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5645d40a0cf0 con 0x5645d41935d0 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f941c3df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645d40f52b0 con 0x5645d41a2780 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d4193090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d4193090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40356/0 (socket says 172.21.15.28:40356) 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 -- 172.21.15.28:0/1687936038 learned_addr learned my addr 172.21.15.28:0/1687936038 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f94137fe640 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d4193a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 0x5645d41a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9410ff9640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93fc0a3430 0x7f93fc0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 -- 172.21.15.28:0/1687936038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 msgr2=0x5645d41a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 0x5645d41a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 -- 172.21.15.28:0/1687936038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 msgr2=0x5645d4193a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d4193a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9412ffd640 1 -- 172.21.15.28:0/1687936038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d40853a0 con 0x5645d4194b40 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.717+0000 7f9413fff640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93fc0a3430 0x7f93fc0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.718+0000 7f9410ff9640 1 -- 172.21.15.28:0/47066237 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f940c0f1600 con 0x7f94141082a0 2024-09-15T22:05:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.718+0000 7f9412ffd640 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d4193090 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9408009330 tx=0x7f9408009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ceddfc9b6477e21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.718+0000 7f93f9ffb640 1 -- 172.21.15.28:0/1687936038 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408017610 con 0x5645d4194b40 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.718+0000 7f93f9ffb640 1 -- 172.21.15.28:0/1687936038 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f940801c070 con 0x5645d4194b40 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.718+0000 7f93f9ffb640 1 -- 172.21.15.28:0/1687936038 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408015880 con 0x5645d4194b40 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 -- 172.21.15.28:0/1687936038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 msgr2=0x5645d4193090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f9413fff640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93fc0a3430 0x7f93fc0a58f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f940c0eeb60 tx=0x7f940c08f480 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d4193090 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9408009330 tx=0x7f9408009570 comp rx=0 tx=0).stop 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 -- 172.21.15.28:0/1687936038 shutdown_connections 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 0x5645d41a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d4193a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 --2- 172.21.15.28:0/1687936038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d4193090 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 -- 172.21.15.28:0/1687936038 >> 172.21.15.28:0/1687936038 conn(0x5645d415c310 msgr2=0x5645d418fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 -- 172.21.15.28:0/1687936038 shutdown_connections 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 -- 172.21.15.28:0/1687936038 wait complete. 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 -- start start 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.719+0000 7f941c3df740 1 Processor -- start 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41935d0 0x5645d41a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d4194b40 0x5645d419f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41a2780 0x5645d419f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5645d41a5420 con 0x5645d41935d0 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5645d40b4060 con 0x5645d41a2780 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645d41a55a0 con 0x5645d4194b40 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 10 client.0 _ll_register_callbacks cb 0x5645d3e078c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41935d0 0x5645d41a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41935d0 0x5645d41a20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40364/0 (socket says 172.21.15.28:40364) 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 -- 172.21.15.28:0/3373228421 learned_addr learned my addr 172.21.15.28:0/3373228421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f94137fe640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d4194b40 0x5645d419f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41a2780 0x5645d419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 -- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d4194b40 msgr2=0x5645d419f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d4194b40 0x5645d419f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 -- 172.21.15.28:0/3373228421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41a2780 msgr2=0x5645d419f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41a2780 0x5645d419f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 -- 172.21.15.28:0/3373228421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d3e6a3c0 con 0x5645d41935d0 2024-09-15T22:05:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f94137fe640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d4194b40 0x5645d419f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f9412ffd640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41935d0 0x5645d41a20f0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9408010a50 tx=0x7f9408015520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58feb81791d69f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f93f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f93f2ffd640 1 -- 172.21.15.28:0/3373228421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080176a0 con 0x5645d41935d0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f93f2ffd640 1 -- 172.21.15.28:0/3373228421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f9408017840 con 0x5645d41935d0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.720+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f940801c070 con 0x5645d41935d0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f93f2ffd640 1 -- 172.21.15.28:0/3373228421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9408017a40 con 0x5645d41935d0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 10 client.16911 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5645d41b0e30 con 0x5645d41935d0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f93f2ffd640 1 -- 172.21.15.28:0/3373228421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f940801c070 con 0x5645d41935d0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f93f2ffd640 1 -- 172.21.15.28:0/3373228421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f9408020680 con 0x5645d41935d0 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 10 client.16911 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 10 client.16911 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:05:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 20 client.16911 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d41b21e0 0x5645d41b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 4 client.16911 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645d4106860 con 0x5645d41b21e0 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.721+0000 7f94137fe640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d41b21e0 0x5645d41b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.722+0000 7f94137fe640 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d41b21e0 0x5645d41b45d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1fc61e3f10bf9f1a server_cookie=3a41350534fbe325 in_seq=0 out_seq=0 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.722+0000 7f93f2ffd640 10 client.16911 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f93f2ffd640 1 -- 172.21.15.28:0/3373228421 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5645d4106860 con 0x5645d41b21e0 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f93f2ffd640 10 client.16911 handle_command_reply: tid=0 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 client.16911 shutdown 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41935d0 msgr2=0x5645d41a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41935d0 0x5645d41a20f0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9408010a50 tx=0x7f9408015520 comp rx=0 tx=0).stop 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 shutdown_connections 2024-09-15T22:05:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d41b21e0 0x5645d41b45d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41a2780 0x5645d419f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d4194b40 0x5645d419f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f93f2ffd640 0 client.16911 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 --2- 172.21.15.28:0/3373228421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41935d0 0x5645d41a20f0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 >> 172.21.15.28:0/3373228421 conn(0x5645d415c310 msgr2=0x5645d4195930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 shutdown_connections 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 1 -- 172.21.15.28:0/3373228421 wait complete. 2024-09-15T22:05:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.723+0000 7f941c3df740 20 client.16911 trim_cache size 0 max 16384 2024-09-15T22:05:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 Processor -- start 2024-09-15T22:05:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 -- start start 2024-09-15T22:05:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d41ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d418ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 0x5645d41b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94141acde0 con 0x5645d4194b40 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ae950 con 0x5645d41935d0 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.736+0000 7f941c3df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ad4b0 con 0x5645d41a2780 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d418ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d418ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40376/0 (socket says 172.21.15.28:40376) 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 -- 172.21.15.28:0/1712538822 learned_addr learned my addr 172.21.15.28:0/1712538822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f9412ffd640 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d41ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 -- 172.21.15.28:0/1712538822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 msgr2=0x5645d41b9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 0x5645d41b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 -- 172.21.15.28:0/1712538822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 msgr2=0x5645d41ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d41ae6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 -- 172.21.15.28:0/1712538822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d4196120 con 0x5645d4194b40 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f94137fe640 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d418ec10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9404000c00 tx=0x7f9404010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ff2c320562b06b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f93f97fa640 1 -- 172.21.15.28:0/1712538822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940402a070 con 0x5645d4194b40 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f93f97fa640 1 -- 172.21.15.28:0/1712538822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f940401eaf0 con 0x5645d4194b40 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.737+0000 7f93f97fa640 1 -- 172.21.15.28:0/1712538822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9404025c50 con 0x5645d4194b40 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 -- 172.21.15.28:0/1712538822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 msgr2=0x5645d418ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d418ec10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9404000c00 tx=0x7f9404010800 comp rx=0 tx=0).stop 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 -- 172.21.15.28:0/1712538822 shutdown_connections 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41a2780 0x5645d41b9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d4194b40 0x5645d418ec10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 --2- 172.21.15.28:0/1712538822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d41935d0 0x5645d41ae6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 -- 172.21.15.28:0/1712538822 >> 172.21.15.28:0/1712538822 conn(0x5645d415c310 msgr2=0x5645d418e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 -- 172.21.15.28:0/1712538822 shutdown_connections 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 -- 172.21.15.28:0/1712538822 wait complete. 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 -- start start 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.738+0000 7f941c3df740 1 Processor -- start 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f941c3df740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41935d0 0x5645d41aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f941c3df740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d4194b40 0x5645d41aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f941c3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41a2780 0x5645d41a7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f941c3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ae280 con 0x5645d41a2780 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f941c3df740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94141adbb0 con 0x5645d4194b40 2024-09-15T22:05:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f941c3df740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94141aade0 con 0x5645d41935d0 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f941c3df740 10 client.0 _ll_register_callbacks cb 0x7f940400a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41a2780 0x5645d41a7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41a2780 0x5645d41a7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40390/0 (socket says 172.21.15.28:40390) 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 -- 172.21.15.28:0/198633938 learned_addr learned my addr 172.21.15.28:0/198633938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41935d0 0x5645d41aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f94137fe640 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d4194b40 0x5645d41aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 -- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41935d0 msgr2=0x5645d41aaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41935d0 0x5645d41aaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 -- 172.21.15.28:0/198633938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d4194b40 msgr2=0x5645d41aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d4194b40 0x5645d41aaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.739+0000 7f9412ffd640 1 -- 172.21.15.28:0/198633938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d41a7ef0 con 0x5645d41a2780 2024-09-15T22:05:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f9412ffd640 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41a2780 0x5645d41a7800 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f94080091f0 tx=0x7f9408000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c31fc8d9ab0be28 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f93f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f93f2ffd640 1 -- 172.21.15.28:0/198633938 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408020630 con 0x5645d41a2780 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f93f2ffd640 1 -- 172.21.15.28:0/198633938 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f94080207d0 con 0x5645d41a2780 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f940401eaf0 con 0x5645d41a2780 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f93f2ffd640 1 -- 172.21.15.28:0/198633938 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9408015590 con 0x5645d41a2780 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 10 client.16917 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5645d421d560 con 0x5645d41a2780 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f93f2ffd640 1 -- 172.21.15.28:0/198633938 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f940401eaf0 con 0x5645d41a2780 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f93f2ffd640 1 -- 172.21.15.28:0/198633938 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f94080226d0 con 0x5645d41a2780 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 10 client.16917 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 10 client.16917 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:05:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 20 client.16917 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d421dc20 0x5645d4220010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 4 client.16917 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.740+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645d4100d70 con 0x5645d421dc20 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.741+0000 7f94137fe640 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d421dc20 0x5645d4220010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.741+0000 7f93f2ffd640 10 client.16917 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.741+0000 7f94137fe640 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d421dc20 0x5645d4220010 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b36009bafec0463d server_cookie=b49050268abda508 in_seq=0 out_seq=0 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.741+0000 7f93f2ffd640 1 -- 172.21.15.28:0/198633938 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645d4100d70 con 0x5645d421dc20 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.741+0000 7f93f2ffd640 10 client.16917 handle_command_reply: tid=0 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 client.16917 shutdown 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41a2780 msgr2=0x5645d41a7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41a2780 0x5645d41a7800 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f94080091f0 tx=0x7f9408000bc0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 shutdown_connections 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x5645d421dc20 0x5645d4220010 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5645d41a2780 0x5645d41a7800 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f93f2ffd640 0 client.16917 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5645d4194b40 0x5645d41aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 --2- 172.21.15.28:0/198633938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645d41935d0 0x5645d41aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 >> 172.21.15.28:0/198633938 conn(0x5645d415c310 msgr2=0x5645d4195930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 shutdown_connections 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 1 -- 172.21.15.28:0/198633938 wait complete. 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.742+0000 7f941c3df740 20 client.16917 trim_cache size 0 max 16384 2024-09-15T22:05:10.751 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93fc0a3430 msgr2=0x7f93fc0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93fc0a3430 0x7f93fc0a58f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f940c0eeb60 tx=0x7f940c08f480 comp rx=0 tx=0).stop 2024-09-15T22:05:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 msgr2=0x7f941419cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941419cdf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f940c005cc0 tx=0x7f940c031460 comp rx=0 tx=0).stop 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 shutdown_connections 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f93fc0a3430 0x7f93fc0a58f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94141082a0 0x7f941419cdf0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94141036e0 0x7f9414198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 --2- 172.21.15.28:0/47066237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9414102d30 0x7f9414198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 >> 172.21.15.28:0/47066237 conn(0x7f94140fc7d0 msgr2=0x7f94140fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 shutdown_connections 2024-09-15T22:05:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:10.743+0000 7f941a006640 1 -- 172.21.15.28:0/47066237 wait complete. 2024-09-15T22:05:11.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:05:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:12.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:11 smithi028 ceph-mon[30969]: pgmap v240: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-09-15T22:05:12.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:11 smithi203 ceph-mon[35546]: pgmap v240: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-09-15T22:05:12.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:11 smithi174 ceph-mon[35244]: pgmap v240: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-09-15T22:05:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.282+0000 7fc807353640 1 -- 172.21.15.28:0/590994572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102570 msgr2=0x7fc8001029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.282+0000 7fc807353640 1 --2- 172.21.15.28:0/590994572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102570 0x7fc8001029b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009930 tx=0x7fc7f002a4d0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.282+0000 7fc807353640 1 -- 172.21.15.28:0/590994572 shutdown_connections 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.282+0000 7fc807353640 1 --2- 172.21.15.28:0/590994572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc800102ef0 0x7fc80010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.282+0000 7fc807353640 1 --2- 172.21.15.28:0/590994572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800102570 0x7fc8001029b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.282+0000 7fc807353640 1 --2- 172.21.15.28:0/590994572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800100e20 0x7fc800101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.282+0000 7fc807353640 1 -- 172.21.15.28:0/590994572 >> 172.21.15.28:0/590994572 conn(0x7fc8000fc7d0 msgr2=0x7fc8000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.283+0000 7fc807353640 1 -- 172.21.15.28:0/590994572 shutdown_connections 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.283+0000 7fc807353640 1 -- 172.21.15.28:0/590994572 wait complete. 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.283+0000 7fc807353640 1 Processor -- start 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.283+0000 7fc807353640 1 -- start start 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.283+0000 7fc807353640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800100e20 0x7fc800199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.283+0000 7fc807353640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800102570 0x7fc800199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc807353640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc800102ef0 0x7fc8001a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc807353640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc800110f80 con 0x7fc800102570 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc807353640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc800110e00 con 0x7fc800100e20 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc807353640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc800111100 con 0x7fc800102ef0 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8048ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800102570 0x7fc800199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8050ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800100e20 0x7fc800199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8048ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800102570 0x7fc800199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40410/0 (socket says 172.21.15.28:40410) 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8048ee640 1 -- 172.21.15.28:0/3034611834 learned_addr learned my addr 172.21.15.28:0/3034611834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8058f0640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc800102ef0 0x7fc8001a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8050ef640 1 -- 172.21.15.28:0/3034611834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc800102ef0 msgr2=0x7fc8001a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8050ef640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc800102ef0 0x7fc8001a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8050ef640 1 -- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800102570 msgr2=0x7fc800199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8050ef640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800102570 0x7fc800199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8050ef640 1 -- 172.21.15.28:0/3034611834 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8001a4910 con 0x7fc800100e20 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8048ee640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800102570 0x7fc800199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.284+0000 7fc8050ef640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800100e20 0x7fc800199350 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00e9f0 tx=0x7fc7fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.285+0000 7fc7f67fc640 1 -- 172.21.15.28:0/3034611834 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00ccc0 con 0x7fc800100e20 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.285+0000 7fc7f67fc640 1 -- 172.21.15.28:0/3034611834 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7fc010910 con 0x7fc800100e20 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.285+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8001a4ae0 con 0x7fc800100e20 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.285+0000 7fc7f67fc640 1 -- 172.21.15.28:0/3034611834 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc018960 con 0x7fc800100e20 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.285+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8001a4ff0 con 0x7fc800100e20 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.287+0000 7fc80972c740 1 Processor -- start 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.287+0000 7fc80972c740 1 -- start start 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.287+0000 7fc7f67fc640 1 -- 172.21.15.28:0/3034611834 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fc7fc010ac0 con 0x7fc800100e20 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc80972c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424beb40 0x557f424bd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc80972c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424bda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc80972c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424ca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc80972c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f424232f0 con 0x557f424bd5d0 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc80972c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f423cacf0 con 0x557f424c8370 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc80972c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557f4241f2b0 con 0x557f424beb40 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8050ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424bda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424beb40 0x557f424bd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424beb40 0x557f424bd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:37268/0 (socket says 172.21.15.28:37268) 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8050ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424bda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40416/0 (socket says 172.21.15.28:40416) 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 -- 172.21.15.28:0/1756579630 learned_addr learned my addr 172.21.15.28:0/1756579630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8048ee640 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 -- 172.21.15.28:0/1756579630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 msgr2=0x557f424ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 -- 172.21.15.28:0/1756579630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 msgr2=0x557f424bda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424bda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 -- 172.21.15.28:0/1756579630 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f423af3a0 con 0x557f424beb40 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8050ef640 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424bda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc8058f0640 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424beb40 0x557f424bd090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009f50 tx=0x7fc7f80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f0569f07ce19d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.288+0000 7fc7f67fc640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7ec0a33b0 0x7fc7ec0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc8050ef640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7ec0a33b0 0x7fc7ec0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc7eb7fe640 1 -- 172.21.15.28:0/1756579630 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8004d20 con 0x557f424beb40 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc7eb7fe640 1 -- 172.21.15.28:0/1756579630 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f8004050 con 0x557f424beb40 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc7eb7fe640 1 -- 172.21.15.28:0/1756579630 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8005280 con 0x557f424beb40 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc7f67fc640 1 -- 172.21.15.28:0/3034611834 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fc7fc0d0460 con 0x7fc800100e20 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc80972c740 1 -- 172.21.15.28:0/1756579630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424beb40 msgr2=0x557f424bd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc8050ef640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7ec0a33b0 0x7fc7ec0a5870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0cd8b0 tx=0x7fc7fc0d6060 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.289+0000 7fc80972c740 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424beb40 0x557f424bd090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009f50 tx=0x7fc7f80118d0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- 172.21.15.28:0/1756579630 shutdown_connections 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424bda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 --2- 172.21.15.28:0/1756579630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424beb40 0x557f424bd090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- 172.21.15.28:0/1756579630 >> 172.21.15.28:0/1756579630 conn(0x557f42486310 msgr2=0x557f424b9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- 172.21.15.28:0/1756579630 shutdown_connections 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- 172.21.15.28:0/1756579630 wait complete. 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- start start 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 Processor -- start 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424beb40 0x557f424c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f424cb070 con 0x557f424bd5d0 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f423de060 con 0x557f424beb40 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.290+0000 7fc80972c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557f424cb1f0 con 0x557f424c8370 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc80972c740 10 client.0 _ll_register_callbacks cb 0x557f421318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40424/0 (socket says 172.21.15.28:40424) 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 -- 172.21.15.28:0/44165776 learned_addr learned my addr 172.21.15.28:0/44165776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8048ee640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424beb40 0x557f424c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 -- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 msgr2=0x557f424c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 -- 172.21.15.28:0/44165776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424beb40 msgr2=0x557f424c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424beb40 0x557f424c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 -- 172.21.15.28:0/44165776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f421943c0 con 0x557f424bd5d0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8048ee640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424beb40 0x557f424c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc8058f0640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424c7d20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009250 tx=0x7fc7f8009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26145909eb728540 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc7e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/44165776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f802b070 con 0x557f424bd5d0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/44165776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fc7f8025450 con 0x557f424bd5d0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.291+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f8004050 con 0x557f424bd5d0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/44165776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc7f8003e40 con 0x557f424bd5d0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 10 client.16929 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557f424d6ac0 con 0x557f424bd5d0 2024-09-15T22:05:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/44165776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f8004050 con 0x557f424bd5d0 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/44165776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc7f800f5a0 con 0x557f424bd5d0 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 10 client.16929 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 10 client.16929 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 20 client.16929 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f424d7f20 0x557f424da310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 4 client.16929 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f42430860 con 0x557f424d7f20 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.292+0000 7fc8048ee640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f424d7f20 0x557f424da310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.293+0000 7fc8048ee640 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f424d7f20 0x557f424da310 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c2f66e7275dfd0c server_cookie=9f94c69542a256f3 in_seq=0 out_seq=0 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.293+0000 7fc7e8ff9640 10 client.16929 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.293+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/44165776 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f42430860 con 0x557f424d7f20 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.293+0000 7fc7e8ff9640 10 client.16929 handle_command_reply: tid=0 2024-09-15T22:05:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.293+0000 7fc80972c740 1 client.16929 shutdown 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 msgr2=0x557f424c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424c7d20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009250 tx=0x7fc7f8009c80 comp rx=0 tx=0).stop 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 shutdown_connections 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f424d7f20 0x557f424da310 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424beb40 0x557f424c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 --2- 172.21.15.28:0/44165776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424bd5d0 0x557f424c7d20 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc7e8ff9640 0 client.16929 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 >> 172.21.15.28:0/44165776 conn(0x557f42486310 msgr2=0x557f424bf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 shutdown_connections 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 1 -- 172.21.15.28:0/44165776 wait complete. 2024-09-15T22:05:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.294+0000 7fc80972c740 20 client.16929 trim_cache size 0 max 16384 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.307+0000 7fc80972c740 1 Processor -- start 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.307+0000 7fc80972c740 1 -- start start 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc80972c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424bd5d0 0x557f424b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc80972c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424c0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc80972c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc80972c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001af110 con 0x557f424beb40 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc80972c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001b0c80 con 0x557f424bd5d0 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc80972c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001af7e0 con 0x557f424c8370 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424c0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40434/0 (socket says 172.21.15.28:40434) 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 -- 172.21.15.28:0/1821969039 learned_addr learned my addr 172.21.15.28:0/1821969039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8058f0640 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8058f0640 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424bd5d0 0x557f424b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 -- 172.21.15.28:0/1821969039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 msgr2=0x557f424c0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 -- 172.21.15.28:0/1821969039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424bd5d0 msgr2=0x557f424b8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424bd5d0 0x557f424b8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 -- 172.21.15.28:0/1821969039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f424c46a0 con 0x557f424beb40 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8058f0640 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424bd5d0 0x557f424b8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.308+0000 7fc8048ee640 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424c0120 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009850 tx=0x7fc7f0004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f58afcb3199be32 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.309+0000 7fc7eaffd640 1 -- 172.21.15.28:0/1821969039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0035040 con 0x557f424beb40 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.309+0000 7fc7eaffd640 1 -- 172.21.15.28:0/1821969039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f0005500 con 0x557f424beb40 2024-09-15T22:05:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.309+0000 7fc7eaffd640 1 -- 172.21.15.28:0/1821969039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0003850 con 0x557f424beb40 2024-09-15T22:05:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 -- 172.21.15.28:0/1821969039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 msgr2=0x557f424c0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424c0120 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009850 tx=0x7fc7f0004310 comp rx=0 tx=0).stop 2024-09-15T22:05:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 -- 172.21.15.28:0/1821969039 shutdown_connections 2024-09-15T22:05:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424c8370 0x557f424c0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424c0120 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 --2- 172.21.15.28:0/1821969039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424bd5d0 0x557f424b8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 -- 172.21.15.28:0/1821969039 >> 172.21.15.28:0/1821969039 conn(0x557f42486310 msgr2=0x557f424c3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 -- 172.21.15.28:0/1821969039 shutdown_connections 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 -- 172.21.15.28:0/1821969039 wait complete. 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 -- start start 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.310+0000 7fc80972c740 1 Processor -- start 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc80972c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424bd5d0 0x557f424cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc80972c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc80972c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc80972c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001b05b0 con 0x557f424beb40 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc80972c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001afee0 con 0x557f424c8370 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc80972c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001ad110 con 0x557f424bd5d0 2024-09-15T22:05:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc80972c740 10 client.0 _ll_register_callbacks cb 0x7fc7fc0d3e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc8048ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc8048ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424cc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40442/0 (socket says 172.21.15.28:40442) 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc8048ee640 1 -- 172.21.15.28:0/2643256941 learned_addr learned my addr 172.21.15.28:0/2643256941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc8058f0640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.311+0000 7fc8058f0640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424bd5d0 0x557f424cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc8048ee640 1 -- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424bd5d0 msgr2=0x557f424cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc8048ee640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424bd5d0 0x557f424cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc8048ee640 1 -- 172.21.15.28:0/2643256941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 msgr2=0x557f424cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc8048ee640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc8048ee640 1 -- 172.21.15.28:0/2643256941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f42541c20 con 0x557f424beb40 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc8058f0640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc8048ee640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424cc970 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009a50 tx=0x7fc7f00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b4fd3143e49e73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc7e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/2643256941 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0035040 con 0x557f424beb40 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/2643256941 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fc7f00046e0 con 0x557f424beb40 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f0005500 con 0x557f424beb40 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/2643256941 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc7f0065050 con 0x557f424beb40 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc80972c740 10 client.16935 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.312+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557f42542e70 con 0x557f424beb40 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/2643256941 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f0057070 con 0x557f424beb40 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/2643256941 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc7f0031c20 con 0x557f424beb40 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc80972c740 10 client.16935 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc80972c740 10 client.16935 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc80972c740 20 client.16935 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc80972c740 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f42543530 0x557f42545950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc80972c740 4 client.16935 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc7f0005500 con 0x557f42543530 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.313+0000 7fc8058f0640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f42543530 0x557f42545950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc8058f0640 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f42543530 0x557f42545950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4fb575e8e5ff75c server_cookie=4a7eb6d46030e9f0 in_seq=0 out_seq=0 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc7e8ff9640 10 client.16935 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc7e8ff9640 1 -- 172.21.15.28:0/2643256941 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc7f0005500 con 0x557f42543530 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc7e8ff9640 10 client.16935 handle_command_reply: tid=0 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 client.16935 shutdown 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 msgr2=0x557f424cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424cc970 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009a50 tx=0x7fc7f00045e0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 shutdown_connections 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x557f42543530 0x557f42545950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f424c8370 0x557f424cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f424beb40 0x557f424cc970 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 --2- 172.21.15.28:0/2643256941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f424bd5d0 0x557f424cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 >> 172.21.15.28:0/2643256941 conn(0x557f42486310 msgr2=0x557f424c6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc7e8ff9640 0 client.16935 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 shutdown_connections 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 1 -- 172.21.15.28:0/2643256941 wait complete. 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.314+0000 7fc80972c740 20 client.16935 trim_cache size 0 max 16384 2024-09-15T22:05:12.331 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7ec0a33b0 msgr2=0x7fc7ec0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7ec0a33b0 0x7fc7ec0a5870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0cd8b0 tx=0x7fc7fc0d6060 comp rx=0 tx=0).stop 2024-09-15T22:05:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800100e20 msgr2=0x7fc800199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800100e20 0x7fc800199350 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00e9f0 tx=0x7fc7fc00eeb0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 shutdown_connections 2024-09-15T22:05:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc7ec0a33b0 0x7fc7ec0a5870 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc800102ef0 0x7fc8001a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc800102570 0x7fc800199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 --2- 172.21.15.28:0/3034611834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc800100e20 0x7fc800199350 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 >> 172.21.15.28:0/3034611834 conn(0x7fc8000fc7d0 msgr2=0x7fc8001048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 shutdown_connections 2024-09-15T22:05:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:12.315+0000 7fc807353640 1 -- 172.21.15.28:0/3034611834 wait complete. 2024-09-15T22:05:13.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:05:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:13.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.799+0000 7f651c6a3640 1 -- 172.21.15.28:0/2515625831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514101de0 msgr2=0x7f65141021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.799+0000 7f651c6a3640 1 --2- 172.21.15.28:0/2515625831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514101de0 0x7f65141021c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f6510009900 tx=0x7f651002a410 comp rx=0 tx=0).stop 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- 172.21.15.28:0/2515625831 shutdown_connections 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 --2- 172.21.15.28:0/2515625831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6514106f10 0x7f651410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 --2- 172.21.15.28:0/2515625831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6514102790 0x7f65141067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 --2- 172.21.15.28:0/2515625831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514101de0 0x7f65141021c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- 172.21.15.28:0/2515625831 >> 172.21.15.28:0/2515625831 conn(0x7f65140fd8c0 msgr2=0x7f65140ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- 172.21.15.28:0/2515625831 shutdown_connections 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- 172.21.15.28:0/2515625831 wait complete. 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 Processor -- start 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- start start 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6514101de0 0x7f65140744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514102790 0x7f6514074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6514106f10 0x7f6514074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6514111f90 con 0x7f6514102790 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6514111e10 con 0x7f6514101de0 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.800+0000 7f651c6a3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6514112110 con 0x7f6514106f10 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514102790 0x7f6514074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514102790 0x7f6514074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40464/0 (socket says 172.21.15.28:40464) 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 -- 172.21.15.28:0/522363943 learned_addr learned my addr 172.21.15.28:0/522363943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f651a43f640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6514101de0 0x7f65140744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f651ac40640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6514106f10 0x7f6514074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 -- 172.21.15.28:0/522363943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6514106f10 msgr2=0x7f6514074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6514106f10 0x7f6514074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 -- 172.21.15.28:0/522363943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6514101de0 msgr2=0x7f65140744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6514101de0 0x7f65140744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 -- 172.21.15.28:0/522363943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65141a6920 con 0x7f6514102790 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f6519c3e640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514102790 0x7f6514074a30 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f650400b4a0 tx=0x7f650400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f650b7fe640 1 -- 172.21.15.28:0/522363943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504014070 con 0x7f6514102790 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65141a6c10 con 0x7f6514102790 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f650b7fe640 1 -- 172.21.15.28:0/522363943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6504010af0 con 0x7f6514102790 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.801+0000 7f650b7fe640 1 -- 172.21.15.28:0/522363943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504017d10 con 0x7f6514102790 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.802+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65141a7100 con 0x7f6514102790 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.804+0000 7f650b7fe640 1 -- 172.21.15.28:0/522363943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6504017070 con 0x7f6514102790 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.804+0000 7f651ea7c740 1 Processor -- start 2024-09-15T22:05:13.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.804+0000 7f651ea7c740 1 -- start start 2024-09-15T22:05:13.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.804+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ebab40 0x55e207eb9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.804+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207eb95d0 0x55e207eb9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ec4370 0x55e207ec6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ea7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e207e1f2f0 con 0x55e207ec4370 2024-09-15T22:05:13.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ea7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e207dc6cf0 con 0x55e207eb95d0 2024-09-15T22:05:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ea7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e207e1b2b0 con 0x55e207ebab40 2024-09-15T22:05:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651a43f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207eb95d0 0x55e207eb9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ac40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ebab40 0x55e207eb9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f6519c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ec4370 0x55e207ec6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651a43f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207eb95d0 0x55e207eb9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:54646/0 (socket says 172.21.15.28:54646) 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651a43f640 1 -- 172.21.15.28:0/94221790 learned_addr learned my addr 172.21.15.28:0/94221790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f650b7fe640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65000a33b0 0x7f65000a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ac40640 1 -- 172.21.15.28:0/94221790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207eb95d0 msgr2=0x55e207eb9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ac40640 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207eb95d0 0x55e207eb9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ac40640 1 -- 172.21.15.28:0/94221790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ec4370 msgr2=0x55e207ec6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ac40640 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ec4370 0x55e207ec6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ac40640 1 -- 172.21.15.28:0/94221790 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e207dab3a0 con 0x55e207ebab40 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651a43f640 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207eb95d0 0x55e207eb9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f650b7fe640 1 -- 172.21.15.28:0/522363943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f65040cf2e0 con 0x7f6514102790 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.805+0000 7f651ac40640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65000a33b0 0x7f65000a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.806+0000 7f651ac40640 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ebab40 0x55e207eb9090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f650c007e40 tx=0x7f650c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc3c53d6685a617d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.806+0000 7f6508ff9640 1 -- 172.21.15.28:0/94221790 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c010a40 con 0x55e207ebab40 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.806+0000 7f6508ff9640 1 -- 172.21.15.28:0/94221790 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f650c01ae00 con 0x55e207ebab40 2024-09-15T22:05:13.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.806+0000 7f6508ff9640 1 -- 172.21.15.28:0/94221790 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c019c30 con 0x55e207ebab40 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 -- 172.21.15.28:0/94221790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ebab40 msgr2=0x55e207eb9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ebab40 0x55e207eb9090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f650c007e40 tx=0x7f650c00f790 comp rx=0 tx=0).stop 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ac40640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65000a33b0 0x7f65000a5870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f650c01e000 tx=0x7f650c01f000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 -- 172.21.15.28:0/94221790 shutdown_connections 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ec4370 0x55e207ec6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207eb95d0 0x55e207eb9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 --2- 172.21.15.28:0/94221790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ebab40 0x55e207eb9090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 -- 172.21.15.28:0/94221790 >> 172.21.15.28:0/94221790 conn(0x55e207e82310 msgr2=0x55e207eb5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 -- 172.21.15.28:0/94221790 shutdown_connections 2024-09-15T22:05:13.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 -- 172.21.15.28:0/94221790 wait complete. 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 -- start start 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.807+0000 7f651ea7c740 1 Processor -- start 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207ec3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ec0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 0x55e207ec1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ea7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e207ec7070 con 0x55e207eb95d0 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ea7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e207dda060 con 0x55e207ebab40 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ea7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e207ec71f0 con 0x55e207ec4370 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ea7c740 10 client.0 _ll_register_callbacks cb 0x55e207b2d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207ec3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207ec3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40494/0 (socket says 172.21.15.28:40494) 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 -- 172.21.15.28:0/2604288113 learned_addr learned my addr 172.21.15.28:0/2604288113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651ac40640 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 0x55e207ec1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f6519c3e640 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ec0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 -- 172.21.15.28:0/2604288113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 msgr2=0x55e207ec1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 0x55e207ec1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 -- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 msgr2=0x55e207ec0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ec0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 -- 172.21.15.28:0/2604288113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e207b903c0 con 0x55e207eb95d0 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f651a43f640 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207ec3cd0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6510005940 tx=0x7f6510031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcad9ec7e2741322 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f64fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.808+0000 7f64fdffb640 1 -- 172.21.15.28:0/2604288113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510035040 con 0x55e207eb95d0 2024-09-15T22:05:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f650c01ae00 con 0x55e207eb95d0 2024-09-15T22:05:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f64fdffb640 1 -- 172.21.15.28:0/2604288113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f6510004400 con 0x55e207eb95d0 2024-09-15T22:05:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f64fdffb640 1 -- 172.21.15.28:0/2604288113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6510005020 con 0x55e207eb95d0 2024-09-15T22:05:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 10 client.16947 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e207ed2ac0 con 0x55e207eb95d0 2024-09-15T22:05:13.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f64fdffb640 1 -- 172.21.15.28:0/2604288113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f650c01ae00 con 0x55e207eb95d0 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f64fdffb640 1 -- 172.21.15.28:0/2604288113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f65100035f0 con 0x55e207eb95d0 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 10 client.16947 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 20 client.16947 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207ed3f80 0x55e207ed63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 4 client.16947 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.809+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e207e2c860 con 0x55e207ed3f80 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.810+0000 7f6519c3e640 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207ed3f80 0x55e207ed63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.810+0000 7f6519c3e640 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207ed3f80 0x55e207ed63b0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71e43cd4b71c0ffe server_cookie=af2de2959e38c098 in_seq=0 out_seq=0 2024-09-15T22:05:13.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.810+0000 7f64fdffb640 10 client.16947 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f64fdffb640 1 -- 172.21.15.28:0/2604288113 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e207e2c860 con 0x55e207ed3f80 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f64fdffb640 10 client.16947 handle_command_reply: tid=0 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 client.16947 shutdown 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 msgr2=0x55e207ec3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207ec3cd0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6510005940 tx=0x7f6510031a60 comp rx=0 tx=0).stop 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 shutdown_connections 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207ed3f80 0x55e207ed63b0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 0x55e207ec1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ec0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f64fdffb640 0 client.16947 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 --2- 172.21.15.28:0/2604288113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207ec3cd0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 >> 172.21.15.28:0/2604288113 conn(0x55e207e82310 msgr2=0x55e207ebb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:13.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 shutdown_connections 2024-09-15T22:05:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 1 -- 172.21.15.28:0/2604288113 wait complete. 2024-09-15T22:05:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.811+0000 7f651ea7c740 20 client.16947 trim_cache size 0 max 16384 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 Processor -- start 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 -- start start 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207eb4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ebc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 0x55e207ebc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b11c0 con 0x55e207eb95d0 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b2d30 con 0x55e207ebab40 2024-09-15T22:05:13.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.824+0000 7f651ea7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b1890 con 0x55e207ec4370 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207eb4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207eb4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40510/0 (socket says 172.21.15.28:40510) 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 -- 172.21.15.28:0/811706454 learned_addr learned my addr 172.21.15.28:0/811706454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f6519c3e640 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ebc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 -- 172.21.15.28:0/811706454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 msgr2=0x55e207ebc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 0x55e207ebc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 -- 172.21.15.28:0/811706454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 msgr2=0x55e207ebc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ebc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 -- 172.21.15.28:0/811706454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e207ebcba0 con 0x55e207eb95d0 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f651a43f640 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207eb4c10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f651002a930 tx=0x7f651002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb03924842e6e403 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f64fffff640 1 -- 172.21.15.28:0/811706454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510058070 con 0x55e207eb95d0 2024-09-15T22:05:13.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f64fffff640 1 -- 172.21.15.28:0/811706454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6510003f90 con 0x55e207eb95d0 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.825+0000 7f64fffff640 1 -- 172.21.15.28:0/811706454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510035030 con 0x55e207eb95d0 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- 172.21.15.28:0/811706454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 msgr2=0x55e207eb4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207eb4c10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f651002a930 tx=0x7f651002a410 comp rx=0 tx=0).stop 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- 172.21.15.28:0/811706454 shutdown_connections 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207ec4370 0x55e207ebc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ebab40 0x55e207ebc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 --2- 172.21.15.28:0/811706454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207eb95d0 0x55e207eb4c10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- 172.21.15.28:0/811706454 >> 172.21.15.28:0/811706454 conn(0x55e207e82310 msgr2=0x55e207eb4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- 172.21.15.28:0/811706454 shutdown_connections 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- 172.21.15.28:0/811706454 wait complete. 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- start start 2024-09-15T22:05:13.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 Processor -- start 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207eb95d0 0x55e207ec83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ebab40 0x55e207ec88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ec4370 0x55e207ec8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b2660 con 0x55e207ebab40 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65141b1f90 con 0x55e207ec4370 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.826+0000 7f651ea7c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65141af1c0 con 0x55e207eb95d0 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f651ea7c740 10 client.0 _ll_register_callbacks cb 0x7f6510009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ebab40 0x55e207ec88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ebab40 0x55e207ec88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40520/0 (socket says 172.21.15.28:40520) 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 -- 172.21.15.28:0/1937051772 learned_addr learned my addr 172.21.15.28:0/1937051772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f651ac40640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ec4370 0x55e207ec8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f651a43f640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207eb95d0 0x55e207ec83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 -- 172.21.15.28:0/1937051772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207eb95d0 msgr2=0x55e207ec83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207eb95d0 0x55e207ec83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 -- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ec4370 msgr2=0x55e207ec8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ec4370 0x55e207ec8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 -- 172.21.15.28:0/1937051772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e207f3de50 con 0x55e207ebab40 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f651ac40640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ec4370 0x55e207ec8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f6519c3e640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ebab40 0x55e207ec88e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f650409b0d0 tx=0x7f65040d2e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3c106491c3fd5d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f64fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f64fdffb640 1 -- 172.21.15.28:0/1937051772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650401e040 con 0x55e207ebab40 2024-09-15T22:05:13.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f64fdffb640 1 -- 172.21.15.28:0/1937051772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f65040cc900 con 0x55e207ebab40 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6510003f90 con 0x55e207ebab40 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f64fdffb640 1 -- 172.21.15.28:0/1937051772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f650409b3d0 con 0x55e207ebab40 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f651ea7c740 10 client.16953 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.827+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e207f3f150 con 0x55e207ebab40 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f64fdffb640 1 -- 172.21.15.28:0/1937051772 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6510003f90 con 0x55e207ebab40 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f64fdffb640 1 -- 172.21.15.28:0/1937051772 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f65040b4190 con 0x55e207ebab40 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f651ea7c740 10 client.16953 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f651ea7c740 10 client.16953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f651ea7c740 20 client.16953 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:13.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f651ea7c740 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207f3f8f0 0x55e207f41d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f651ea7c740 4 client.16953 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e207e26d70 con 0x55e207f3f8f0 2024-09-15T22:05:13.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.828+0000 7f651a43f640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207f3f8f0 0x55e207f41d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651a43f640 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207f3f8f0 0x55e207f41d10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=734d7fdffcd85544 server_cookie=e23c80335b1ae1f0 in_seq=0 out_seq=0 2024-09-15T22:05:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f64fdffb640 10 client.16953 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f64fdffb640 1 -- 172.21.15.28:0/1937051772 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e207e26d70 con 0x55e207f3f8f0 2024-09-15T22:05:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f64fdffb640 10 client.16953 handle_command_reply: tid=0 2024-09-15T22:05:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 client.16953 shutdown 2024-09-15T22:05:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ebab40 msgr2=0x55e207ec88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ebab40 0x55e207ec88e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f650409b0d0 tx=0x7f65040d2e30 comp rx=0 tx=0).stop 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 shutdown_connections 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55e207f3f8f0 0x55e207f41d10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e207ec4370 0x55e207ec8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e207ebab40 0x55e207ec88e0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 --2- 172.21.15.28:0/1937051772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e207eb95d0 0x55e207ec83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 >> 172.21.15.28:0/1937051772 conn(0x55e207e82310 msgr2=0x55e207eb4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f64fdffb640 0 client.16953 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 shutdown_connections 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 1 -- 172.21.15.28:0/1937051772 wait complete. 2024-09-15T22:05:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.829+0000 7f651ea7c740 20 client.16953 trim_cache size 0 max 16384 2024-09-15T22:05:13.840 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65000a33b0 msgr2=0x7f65000a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65000a33b0 0x7f65000a5870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f650c01e000 tx=0x7f650c01f000 comp rx=0 tx=0).stop 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514102790 msgr2=0x7f6514074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514102790 0x7f6514074a30 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f650400b4a0 tx=0x7f650400b960 comp rx=0 tx=0).stop 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 shutdown_connections 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f65000a33b0 0x7f65000a5870 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6514106f10 0x7f6514074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6514102790 0x7f6514074a30 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 --2- 172.21.15.28:0/522363943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6514101de0 0x7f65140744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 >> 172.21.15.28:0/522363943 conn(0x7f65140fd8c0 msgr2=0x7f65140feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 shutdown_connections 2024-09-15T22:05:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:13.830+0000 7f651c6a3640 1 -- 172.21.15.28:0/522363943 wait complete. 2024-09-15T22:05:14.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:13 smithi028 ceph-mon[30969]: pgmap v241: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-15T22:05:14.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:13 smithi203 ceph-mon[35546]: pgmap v241: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-15T22:05:14.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:13 smithi174 ceph-mon[35244]: pgmap v241: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-09-15T22:05:15.622 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T22:05:15.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T22:05:15.895 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:15.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:15 smithi028 ceph-mon[30969]: pgmap v242: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-15T22:05:16.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:15 smithi203 ceph-mon[35546]: pgmap v242: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-15T22:05:16.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:15 smithi174 ceph-mon[35244]: pgmap v242: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-15T22:05:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 -- 172.21.15.28:0/1022802667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098106f10 msgr2=0x7fd09810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/1022802667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098106f10 0x7fd09810ea90 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd090009ca0 tx=0x7fd09002a8c0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 -- 172.21.15.28:0/1022802667 shutdown_connections 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/1022802667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098106f10 0x7fd09810ea90 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/1022802667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd098102790 0x7fd0981067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/1022802667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd098101de0 0x7fd0981021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 -- 172.21.15.28:0/1022802667 >> 172.21.15.28:0/1022802667 conn(0x7fd0980fd8c0 msgr2=0x7fd0980ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 -- 172.21.15.28:0/1022802667 shutdown_connections 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.378+0000 7fd09d9c4640 1 -- 172.21.15.28:0/1022802667 wait complete. 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 Processor -- start 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 -- start start 2024-09-15T22:05:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101de0 0x7fd098198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd098102790 0x7fd098198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd098106f10 0x7fd09819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd098111f70 con 0x7fd098101de0 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd098111df0 con 0x7fd098102790 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd09d9c4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981120f0 con 0x7fd098106f10 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd096ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101de0 0x7fd098198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd096ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101de0 0x7fd098198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40526/0 (socket says 172.21.15.28:40526) 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.379+0000 7fd096ffd640 1 -- 172.21.15.28:0/920874785 learned_addr learned my addr 172.21.15.28:0/920874785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd08ffff640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd098102790 0x7fd098198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd0977fe640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd098106f10 0x7fd09819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd096ffd640 1 -- 172.21.15.28:0/920874785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd098106f10 msgr2=0x7fd09819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd096ffd640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd098106f10 0x7fd09819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd096ffd640 1 -- 172.21.15.28:0/920874785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd098102790 msgr2=0x7fd098198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd096ffd640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd098102790 0x7fd098198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd096ffd640 1 -- 172.21.15.28:0/920874785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09819d3c0 con 0x7fd098101de0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd0977fe640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd098106f10 0x7fd09819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd096ffd640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101de0 0x7fd098198410 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c890 tx=0x7fd08800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd09c9c2640 1 -- 172.21.15.28:0/920874785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088004220 con 0x7fd098101de0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd09c9c2640 1 -- 172.21.15.28:0/920874785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0880043c0 con 0x7fd098101de0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09819d6b0 con 0x7fd098101de0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd09c9c2640 1 -- 172.21.15.28:0/920874785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088004a00 con 0x7fd098101de0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.380+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0981a4f90 con 0x7fd098101de0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09c9c2640 1 -- 172.21.15.28:0/920874785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd0880079b0 con 0x7fd098101de0 2024-09-15T22:05:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 Processor -- start 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 -- start start 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a4363af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363b480 0x557a43645800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a43645d40 0x557a43648130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a435a0280 con 0x557a43645d40 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557a43547b10 con 0x557a4363b480 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.383+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557a4359c240 con 0x557a4363ab60 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a43645d40 0x557a43648130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd096ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363b480 0x557a43645800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a43645d40 0x557a43648130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40540/0 (socket says 172.21.15.28:40540) 2024-09-15T22:05:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 -- 172.21.15.28:0/3843059786 learned_addr learned my addr 172.21.15.28:0/3843059786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd0977fe640 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a4363af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 -- 172.21.15.28:0/3843059786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 msgr2=0x557a4363af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a4363af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 -- 172.21.15.28:0/3843059786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363b480 msgr2=0x557a43645800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363b480 0x557a43645800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 -- 172.21.15.28:0/3843059786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a435541f0 con 0x557a43645d40 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd096ffd640 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363b480 0x557a43645800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd09c9c2640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd0840a33b0 0x7fd0840a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08ffff640 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a43645d40 0x557a43648130 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd08000bc00 tx=0x7fd08000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451e3bca73573588 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd0977fe640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd0840a33b0 0x7fd0840a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd09c9c2640 1 -- 172.21.15.28:0/920874785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fd0880cfee0 con 0x7fd098101de0 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.384+0000 7fd08dffb640 1 -- 172.21.15.28:0/3843059786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080015070 con 0x557a43645d40 2024-09-15T22:05:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd08dffb640 1 -- 172.21.15.28:0/3843059786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd080005500 con 0x557a43645d40 2024-09-15T22:05:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd08dffb640 1 -- 172.21.15.28:0/3843059786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0800116a0 con 0x557a43645d40 2024-09-15T22:05:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd0977fe640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd0840a33b0 0x7fd0840a5870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd090004af0 tx=0x7fd090005d80 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3843059786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a43645d40 msgr2=0x557a43648130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a43645d40 0x557a43648130 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd08000bc00 tx=0x7fd08000f730 comp rx=0 tx=0).stop 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3843059786 shutdown_connections 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a43645d40 0x557a43648130 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363b480 0x557a43645800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3843059786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a4363af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3843059786 >> 172.21.15.28:0/3843059786 conn(0x557a436370f0 msgr2=0x557a436374d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3843059786 shutdown_connections 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.385+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3843059786 wait complete. 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 -- start start 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 Processor -- start 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a43642c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43643150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43643690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557a4364cd20 con 0x557a4363b480 2024-09-15T22:05:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557a4355af10 con 0x557a43645d40 2024-09-15T22:05:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557a4364cea0 con 0x557a4363ab60 2024-09-15T22:05:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd09fd9d740 10 client.0 _ll_register_callbacks cb 0x557a43523a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43643150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43643150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40546/0 (socket says 172.21.15.28:40546) 2024-09-15T22:05:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.386+0000 7fd08ffff640 1 -- 172.21.15.28:0/1654556421 learned_addr learned my addr 172.21.15.28:0/1654556421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd096ffd640 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a43642c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd0977fe640 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43643690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd08ffff640 1 -- 172.21.15.28:0/1654556421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 msgr2=0x557a43642c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd08ffff640 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a43642c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd08ffff640 1 -- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 msgr2=0x557a43643690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd08ffff640 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43643690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd08ffff640 1 -- 172.21.15.28:0/1654556421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a433116a0 con 0x557a4363b480 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd08ffff640 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43643150 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd0800057c0 tx=0x7fd08000b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2223db293783fb2b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd076ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd076ffd640 1 -- 172.21.15.28:0/1654556421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080015070 con 0x557a4363b480 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd076ffd640 1 -- 172.21.15.28:0/1654556421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fd080004e50 con 0x557a4363b480 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd080005500 con 0x557a4363b480 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd076ffd640 1 -- 172.21.15.28:0/1654556421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd08002c050 con 0x557a4363b480 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd09fd9d740 10 client.16965 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.387+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557a436589b0 con 0x557a4363b480 2024-09-15T22:05:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd076ffd640 1 -- 172.21.15.28:0/1654556421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd080005500 con 0x557a4363b480 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd076ffd640 1 -- 172.21.15.28:0/1654556421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd080011a40 con 0x557a4363b480 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd09fd9d740 10 client.16965 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd09fd9d740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd09fd9d740 20 client.16965 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a43659dc0 0x557a4365c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd09fd9d740 4 client.16965 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a435ad7f0 con 0x557a43659dc0 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd096ffd640 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a43659dc0 0x557a4365c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd096ffd640 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a43659dc0 0x557a4365c1b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaa3ad2f14f1d8cf server_cookie=2ec42a0ee63a889b in_seq=0 out_seq=0 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.388+0000 7fd076ffd640 10 client.16965 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd076ffd640 1 -- 172.21.15.28:0/1654556421 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557a435ad7f0 con 0x557a43659dc0 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd076ffd640 10 client.16965 handle_command_reply: tid=0 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd09fd9d740 1 client.16965 shutdown 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 msgr2=0x557a43643150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43643150 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd0800057c0 tx=0x7fd08000b970 comp rx=0 tx=0).stop 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 shutdown_connections 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a43659dc0 0x557a4365c1b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.389+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43643690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.390+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43643150 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.390+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/1654556421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a43642c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.390+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 >> 172.21.15.28:0/1654556421 conn(0x557a436370f0 msgr2=0x557a4363cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.390+0000 7fd076ffd640 0 client.16965 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.390+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 shutdown_connections 2024-09-15T22:05:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.390+0000 7fd09fd9d740 1 -- 172.21.15.28:0/1654556421 wait complete. 2024-09-15T22:05:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.390+0000 7fd09fd9d740 20 client.16965 trim_cache size 0 max 16384 2024-09-15T22:05:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 Processor -- start 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 -- start start 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a436619c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43661f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43635a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981acf00 con 0x557a4363b480 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981aea70 con 0x557a43645d40 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981ad5d0 con 0x557a4363ab60 2024-09-15T22:05:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43661f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43661f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40556/0 (socket says 172.21.15.28:40556) 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 -- 172.21.15.28:0/2344043 learned_addr learned my addr 172.21.15.28:0/2344043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd096ffd640 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a436619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 -- 172.21.15.28:0/2344043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 msgr2=0x557a436619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd0977fe640 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43635a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a436619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 -- 172.21.15.28:0/2344043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 msgr2=0x557a43635a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43635a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd08ffff640 1 -- 172.21.15.28:0/2344043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a4363b2b0 con 0x557a4363b480 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.407+0000 7fd0977fe640 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43635a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd08ffff640 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43661f00 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd080014040 tx=0x7fd080004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71aa8c492141f4f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd08d7fa640 1 -- 172.21.15.28:0/2344043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080029070 con 0x557a4363b480 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd08d7fa640 1 -- 172.21.15.28:0/2344043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd08000ba80 con 0x557a4363b480 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd08d7fa640 1 -- 172.21.15.28:0/2344043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080015050 con 0x557a4363b480 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 -- 172.21.15.28:0/2344043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 msgr2=0x557a43661f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43661f00 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd080014040 tx=0x7fd080004b60 comp rx=0 tx=0).stop 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 -- 172.21.15.28:0/2344043 shutdown_connections 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a43645d40 0x557a43635a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a43661f00 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/2344043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a4363ab60 0x557a436619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 -- 172.21.15.28:0/2344043 >> 172.21.15.28:0/2344043 conn(0x557a436370f0 msgr2=0x557a4363cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.408+0000 7fd09fd9d740 1 -- 172.21.15.28:0/2344043 shutdown_connections 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 -- 172.21.15.28:0/2344043 wait complete. 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 -- start start 2024-09-15T22:05:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 Processor -- start 2024-09-15T22:05:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363ab60 0x557a4364e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a4364e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a43645d40 0x557a4364ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981ae3a0 con 0x557a4363b480 2024-09-15T22:05:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981adcd0 con 0x557a4363ab60 2024-09-15T22:05:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd0981aaf00 con 0x557a43645d40 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd09fd9d740 10 client.0 _ll_register_callbacks cb 0x7fd0880d3860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a4364e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a4364e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40558/0 (socket says 172.21.15.28:40558) 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.409+0000 7fd08ffff640 1 -- 172.21.15.28:0/3458841725 learned_addr learned my addr 172.21.15.28:0/3458841725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd096ffd640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363ab60 0x557a4364e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd0977fe640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a43645d40 0x557a4364ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd08ffff640 1 -- 172.21.15.28:0/3458841725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a43645d40 msgr2=0x557a4364ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd08ffff640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a43645d40 0x557a4364ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd08ffff640 1 -- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363ab60 msgr2=0x557a4364e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd08ffff640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363ab60 0x557a4364e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd08ffff640 1 -- 172.21.15.28:0/3458841725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a436c3c60 con 0x557a4363b480 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd0977fe640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a43645d40 0x557a4364ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:16.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd096ffd640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363ab60 0x557a4364e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd08ffff640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a4364e6a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd0800058c0 tx=0x7fd0800048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0a90d0812a47ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd076ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd076ffd640 1 -- 172.21.15.28:0/3458841725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08003e060 con 0x557a4363b480 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd076ffd640 1 -- 172.21.15.28:0/3458841725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fd080024620 con 0x557a4363b480 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd08000ba80 con 0x557a4363b480 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd076ffd640 1 -- 172.21.15.28:0/3458841725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd080045050 con 0x557a4363b480 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd09fd9d740 10 client.16971 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.410+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557a436c4f10 con 0x557a4363b480 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd076ffd640 1 -- 172.21.15.28:0/3458841725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd08000ba80 con 0x557a4363b480 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd076ffd640 1 -- 172.21.15.28:0/3458841725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd080004d70 con 0x557a4363b480 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd09fd9d740 10 client.16971 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd09fd9d740 10 client.16971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:16.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd09fd9d740 20 client.16971 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a436c5620 0x557a436c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd09fd9d740 4 client.16971 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557a435a7d00 con 0x557a436c5620 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.411+0000 7fd096ffd640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a436c5620 0x557a436c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.412+0000 7fd096ffd640 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a436c5620 0x557a436c7a40 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d80b290c293dec5 server_cookie=e550a2b00c92d43b in_seq=0 out_seq=0 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.412+0000 7fd076ffd640 10 client.16971 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd076ffd640 1 -- 172.21.15.28:0/3458841725 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557a435a7d00 con 0x557a436c5620 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd076ffd640 10 client.16971 handle_command_reply: tid=0 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 client.16971 shutdown 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 msgr2=0x557a4364e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a4364e6a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fd0800058c0 tx=0x7fd0800048d0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 shutdown_connections 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x557a436c5620 0x557a436c7a40 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557a43645d40 0x557a4364ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a4363b480 0x557a4364e6a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd076ffd640 0 client.16971 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 --2- 172.21.15.28:0/3458841725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a4363ab60 0x557a4364e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 >> 172.21.15.28:0/3458841725 conn(0x557a436370f0 msgr2=0x557a43635a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 shutdown_connections 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.413+0000 7fd09fd9d740 1 -- 172.21.15.28:0/3458841725 wait complete. 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09fd9d740 20 client.16971 trim_cache size 0 max 16384 2024-09-15T22:05:16.422 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:05:16.423 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T22:05:16.423 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "eed175ba-9b9e-48b7-9a74-65e2e6268c70", 2024-09-15T22:05:16.423 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T22:05:16.423 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd0840a33b0 msgr2=0x7fd0840a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd0840a33b0 0x7fd0840a5870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd090004af0 tx=0x7fd090005d80 comp rx=0 tx=0).stop 2024-09-15T22:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101de0 msgr2=0x7fd098198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101de0 0x7fd098198410 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c890 tx=0x7fd08800cd50 comp rx=0 tx=0).stop 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 shutdown_connections 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd0840a33b0 0x7fd0840a5870 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd098106f10 0x7fd09819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd098102790 0x7fd098198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 --2- 172.21.15.28:0/920874785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101de0 0x7fd098198410 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 >> 172.21.15.28:0/920874785 conn(0x7fd0980fd8c0 msgr2=0x7fd0980feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 shutdown_connections 2024-09-15T22:05:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:16.414+0000 7fd09d9c4640 1 -- 172.21.15.28:0/920874785 wait complete. 2024-09-15T22:05:17.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:05:17.412 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 -- 172.21.15.28:0/414526780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c100910 msgr2=0x7f0d5c100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 --2- 172.21.15.28:0/414526780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c100910 0x7f0d5c100cf0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009900 tx=0x7f0d4c02a3f0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 -- 172.21.15.28:0/414526780 shutdown_connections 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 --2- 172.21.15.28:0/414526780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d5c10e4c0 0x7f0d5c1108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 --2- 172.21.15.28:0/414526780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d5c1012c0 0x7f0d5c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 --2- 172.21.15.28:0/414526780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c100910 0x7f0d5c100cf0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 -- 172.21.15.28:0/414526780 >> 172.21.15.28:0/414526780 conn(0x7f0d5c0fc780 msgr2=0x7f0d5c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 -- 172.21.15.28:0/414526780 shutdown_connections 2024-09-15T22:05:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.912+0000 7f0d6391d640 1 -- 172.21.15.28:0/414526780 wait complete. 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 Processor -- start 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 -- start start 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d5c100910 0x7f0d5c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1012c0 0x7f0d5c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d5c10e4c0 0x7f0d5c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c114150 con 0x7f0d5c1012c0 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c113fd0 con 0x7f0d5c100910 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d6391d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c1142d0 con 0x7f0d5c10e4c0 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d60eb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1012c0 0x7f0d5c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d60eb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1012c0 0x7f0d5c19ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40574/0 (socket says 172.21.15.28:40574) 2024-09-15T22:05:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d60eb8640 1 -- 172.21.15.28:0/1610883004 learned_addr learned my addr 172.21.15.28:0/1610883004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d616b9640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d5c100910 0x7f0d5c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d61eba640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d5c10e4c0 0x7f0d5c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d60eb8640 1 -- 172.21.15.28:0/1610883004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d5c10e4c0 msgr2=0x7f0d5c19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d5c10e4c0 0x7f0d5c19ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d60eb8640 1 -- 172.21.15.28:0/1610883004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d5c100910 msgr2=0x7f0d5c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.913+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d5c100910 0x7f0d5c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.914+0000 7f0d60eb8640 1 -- 172.21.15.28:0/1610883004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c19f5e0 con 0x7f0d5c1012c0 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.914+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1012c0 0x7f0d5c19ab70 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c920 tx=0x7f0d5000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.914+0000 7f0d5a7fc640 1 -- 172.21.15.28:0/1610883004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50007c70 con 0x7f0d5c1012c0 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.914+0000 7f0d5a7fc640 1 -- 172.21.15.28:0/1610883004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d50007e10 con 0x7f0d5c1012c0 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.914+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5c19f8d0 con 0x7f0d5c1012c0 2024-09-15T22:05:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.914+0000 7f0d5a7fc640 1 -- 172.21.15.28:0/1610883004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50018890 con 0x7f0d5c1012c0 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.914+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d5c1a71b0 con 0x7f0d5c1012c0 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.917+0000 7f0d5a7fc640 1 -- 172.21.15.28:0/1610883004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f0d50018a70 con 0x7f0d5c1012c0 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.917+0000 7f0d65cf6740 1 Processor -- start 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.917+0000 7f0d65cf6740 1 -- start start 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d2896880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d2896dc0 0x5635d28a1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635d27fc2e0 con 0x5635d28a1680 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5635d27a3ce0 con 0x5635d28964a0 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5635d27f82a0 con 0x5635d2896dc0 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d5a7fc640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0d440a33b0 0x7f0d440a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d60eb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d2896880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d2896880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:54762/0 (socket says 172.21.15.28:54762) 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 -- 172.21.15.28:0/1047972008 learned_addr learned my addr 172.21.15.28:0/1047972008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0d440a33b0 0x7f0d440a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d616b9640 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d2896dc0 0x5635d28a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d5a7fc640 1 -- 172.21.15.28:0/1610883004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f0d500d0220 con 0x7f0d5c1012c0 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 -- 172.21.15.28:0/1047972008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d2896dc0 msgr2=0x5635d28a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d2896dc0 0x5635d28a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 -- 172.21.15.28:0/1047972008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 msgr2=0x5635d28a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.918+0000 7f0d61eba640 1 -- 172.21.15.28:0/1047972008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d2788440 con 0x5635d28964a0 2024-09-15T22:05:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.919+0000 7f0d61eba640 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d2896880 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0d54013cf0 tx=0x7f0d54010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0a375d5348ed0f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.919+0000 7f0d61eba640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0d440a33b0 0x7f0d440a5870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009fd0 tx=0x7f0d54015000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.919+0000 7f0d437fe640 1 -- 172.21.15.28:0/1047972008 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5400bb30 con 0x5635d28964a0 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.919+0000 7f0d437fe640 1 -- 172.21.15.28:0/1047972008 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54009af0 con 0x5635d28964a0 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.919+0000 7f0d437fe640 1 -- 172.21.15.28:0/1047972008 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54005770 con 0x5635d28964a0 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 -- 172.21.15.28:0/1047972008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 msgr2=0x5635d2896880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d2896880 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0d54013cf0 tx=0x7f0d54010740 comp rx=0 tx=0).stop 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 -- 172.21.15.28:0/1047972008 shutdown_connections 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d2896dc0 0x5635d28a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/1047972008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d2896880 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 -- 172.21.15.28:0/1047972008 >> 172.21.15.28:0/1047972008 conn(0x5635d2892a30 msgr2=0x5635d2892e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 -- 172.21.15.28:0/1047972008 shutdown_connections 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 -- 172.21.15.28:0/1047972008 wait complete. 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 -- start start 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 Processor -- start 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d28a1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d2896dc0 0x5635d289e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28a1680 0x5635d289e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.920+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635d28a4250 con 0x5635d2896dc0 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5635d27b7050 con 0x5635d28964a0 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5635d28a43d0 con 0x5635d28a1680 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d65cf6740 10 client.0 _ll_register_callbacks cb 0x5635d27f7160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d2896dc0 0x5635d289e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d2896dc0 0x5635d289e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40588/0 (socket says 172.21.15.28:40588) 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 -- 172.21.15.28:0/906530385 learned_addr learned my addr 172.21.15.28:0/906530385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d616b9640 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d28a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d61eba640 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28a1680 0x5635d289e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 -- 172.21.15.28:0/906530385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28a1680 msgr2=0x5635d289e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28a1680 0x5635d289e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 -- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 msgr2=0x5635d28a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d28a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 -- 172.21.15.28:0/906530385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d256d3c0 con 0x5635d2896dc0 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d2896dc0 0x5635d289e320 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0d500cd670 tx=0x7f0d50000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e491210cd5e0afb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d40ff9640 1 -- 172.21.15.28:0/906530385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d500b4030 con 0x5635d2896dc0 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d40ff9640 1 -- 172.21.15.28:0/906530385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f0d5000fb30 con 0x5635d2896dc0 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d54009af0 con 0x5635d2896dc0 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d40ff9640 1 -- 172.21.15.28:0/906530385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0d5009c850 con 0x5635d2896dc0 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d65cf6740 10 client.16983 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.921+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5635d28afd60 con 0x5635d2896dc0 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d40ff9640 1 -- 172.21.15.28:0/906530385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54009af0 con 0x5635d2896dc0 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d40ff9640 1 -- 172.21.15.28:0/906530385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f0d5009c440 con 0x5635d2896dc0 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d65cf6740 10 client.16983 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d65cf6740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d65cf6740 20 client.16983 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d28b1220 0x5635d28b3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d65cf6740 4 client.16983 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635d2809850 con 0x5635d28b1220 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.922+0000 7f0d616b9640 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d28b1220 0x5635d28b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d616b9640 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d28b1220 0x5635d28b3610 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94e6bbb89cc4111c server_cookie=9bafe6d003f7935d in_seq=0 out_seq=0 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d40ff9640 10 client.16983 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d40ff9640 1 -- 172.21.15.28:0/906530385 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5635d2809850 con 0x5635d28b1220 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d40ff9640 10 client.16983 handle_command_reply: tid=0 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 client.16983 shutdown 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d2896dc0 msgr2=0x5635d289e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d2896dc0 0x5635d289e320 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0d500cd670 tx=0x7f0d50000bc0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 shutdown_connections 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d28b1220 0x5635d28b3610 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28a1680 0x5635d289e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d2896dc0 0x5635d289e320 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d40ff9640 0 client.16983 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/906530385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d28964a0 0x5635d28a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.923+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 >> 172.21.15.28:0/906530385 conn(0x5635d2892a30 msgr2=0x5635d2898760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.924+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 shutdown_connections 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.924+0000 7f0d65cf6740 1 -- 172.21.15.28:0/906530385 wait complete. 2024-09-15T22:05:17.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.924+0000 7f0d65cf6740 20 client.16983 trim_cache size 0 max 16384 2024-09-15T22:05:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 Processor -- start 2024-09-15T22:05:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 -- start start 2024-09-15T22:05:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d289cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c1b1130 con 0x5635d28a1680 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c1b2ca0 con 0x5635d2896dc0 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.941+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c1b1800 con 0x5635d28964a0 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28b8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40594/0 (socket says 172.21.15.28:40594) 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 -- 172.21.15.28:0/4041364941 learned_addr learned my addr 172.21.15.28:0/4041364941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d616b9640 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d289cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 -- 172.21.15.28:0/4041364941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 msgr2=0x5635d28ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28ad4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 -- 172.21.15.28:0/4041364941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 msgr2=0x5635d289cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d289cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 -- 172.21.15.28:0/4041364941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d289d2d0 con 0x5635d28a1680 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d61eba640 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28b8da0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0d54006010 tx=0x7f0d54013ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5a73b3475c6433 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d42ffd640 1 -- 172.21.15.28:0/4041364941 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54034070 con 0x5635d28a1680 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d42ffd640 1 -- 172.21.15.28:0/4041364941 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54009bf0 con 0x5635d28a1680 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.942+0000 7f0d42ffd640 1 -- 172.21.15.28:0/4041364941 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5402fa10 con 0x5635d28a1680 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 -- 172.21.15.28:0/4041364941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 msgr2=0x5635d28b8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28b8da0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0d54006010 tx=0x7f0d54013ba0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 -- 172.21.15.28:0/4041364941 shutdown_connections 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d28b8da0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d289cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/4041364941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 -- 172.21.15.28:0/4041364941 >> 172.21.15.28:0/4041364941 conn(0x5635d2892a30 msgr2=0x5635d28a03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 -- 172.21.15.28:0/4041364941 shutdown_connections 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 -- 172.21.15.28:0/4041364941 wait complete. 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.943+0000 7f0d65cf6740 1 -- start start 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 1 Processor -- start 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d28a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d291ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c1b25d0 con 0x5635d28a1680 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c1b1f00 con 0x5635d2896dc0 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c1af130 con 0x5635d28964a0 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d65cf6740 10 client.0 _ll_register_callbacks cb 0x7f0d5009bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d61eba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d291ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d61eba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d291ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40604/0 (socket says 172.21.15.28:40604) 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d61eba640 1 -- 172.21.15.28:0/3149689487 learned_addr learned my addr 172.21.15.28:0/3149689487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d616b9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d60eb8640 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d28a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d61eba640 1 -- 172.21.15.28:0/3149689487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 msgr2=0x5635d28a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.944+0000 7f0d61eba640 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d61eba640 1 -- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 msgr2=0x5635d28a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d61eba640 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d28a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d61eba640 1 -- 172.21.15.28:0/3149689487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d291b2b0 con 0x5635d28a1680 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d61eba640 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d291ad70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009900 tx=0x7f0d54005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c60198d20f3245 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d40ff9640 1 -- 172.21.15.28:0/3149689487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54034030 con 0x5635d28a1680 2024-09-15T22:05:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d54009bf0 con 0x5635d28a1680 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d40ff9640 1 -- 172.21.15.28:0/3149689487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f0d5402c040 con 0x5635d28a1680 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d40ff9640 1 -- 172.21.15.28:0/3149689487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0d5400bd40 con 0x5635d28a1680 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d65cf6740 10 client.16989 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5635d291c480 con 0x5635d28a1680 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d40ff9640 1 -- 172.21.15.28:0/3149689487 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54009bf0 con 0x5635d28a1680 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.945+0000 7f0d40ff9640 1 -- 172.21.15.28:0/3149689487 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f0d5400f040 con 0x5635d28a1680 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 10 client.16989 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 20 client.16989 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d291c950 0x5635d291ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 4 client.16989 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:05:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5635d2803d60 con 0x5635d291c950 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d616b9640 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d291c950 0x5635d291ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d40ff9640 10 client.16989 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d616b9640 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d291c950 0x5635d291ed40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0afd462ec8584a server_cookie=ee049818272c0538 in_seq=0 out_seq=0 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d40ff9640 1 -- 172.21.15.28:0/3149689487 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5635d2803d60 con 0x5635d291c950 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d40ff9640 10 client.16989 handle_command_reply: tid=0 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 1 client.16989 shutdown 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 msgr2=0x5635d291ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.946+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d291ad70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009900 tx=0x7f0d54005c70 comp rx=0 tx=0).stop 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 shutdown_connections 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5635d291c950 0x5635d291ed40 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635d28a1680 0x5635d291ad70 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5635d2896dc0 0x5635d28a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 --2- 172.21.15.28:0/3149689487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5635d28964a0 0x5635d28a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 >> 172.21.15.28:0/3149689487 conn(0x5635d2892a30 msgr2=0x5635d2898760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d40ff9640 0 client.16989 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 shutdown_connections 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 1 -- 172.21.15.28:0/3149689487 wait complete. 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d65cf6740 20 client.16989 trim_cache size 0 max 16384 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T22:05:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-15T22:05:17.956 INFO:teuthology.orchestra.run.smithi028.stdout: "eed175ba-9b9e-48b7-9a74-65e2e6268c70": { 2024-09-15T22:05:17.956 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-15T22:05:17.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "eed175ba-9b9e-48b7-9a74-65e2e6268c70", 2024-09-15T22:05:17.956 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-15T22:05:17.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:05:17.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:05:17.956 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0d440a33b0 msgr2=0x7f0d440a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0d440a33b0 0x7f0d440a5870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009fd0 tx=0x7f0d54015000 comp rx=0 tx=0).stop 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1012c0 msgr2=0x7f0d5c19ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1012c0 0x7f0d5c19ab70 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c920 tx=0x7f0d5000cde0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 shutdown_connections 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0d440a33b0 0x7f0d440a5870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d5c10e4c0 0x7f0d5c19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1012c0 0x7f0d5c19ab70 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 --2- 172.21.15.28:0/1610883004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d5c100910 0x7f0d5c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.947+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 >> 172.21.15.28:0/1610883004 conn(0x7f0d5c0fc780 msgr2=0x7f0d5c10f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.948+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 shutdown_connections 2024-09-15T22:05:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:17.948+0000 7f0d6391d640 1 -- 172.21.15.28:0/1610883004 wait complete. 2024-09-15T22:05:18.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:17 smithi174 ceph-mon[35244]: pgmap v243: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-15T22:05:18.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:17 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T22:05:18.062 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:17 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:05:18.063 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:17 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:05:18.063 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:17 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:05:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:17 smithi028 ceph-mon[30969]: pgmap v243: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-15T22:05:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:17 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T22:05:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:17 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:05:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:17 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:05:18.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:17 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:05:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:17 smithi203 ceph-mon[35546]: pgmap v243: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-09-15T22:05:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:17 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T22:05:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:17 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:05:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:17 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:05:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:17 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:05:18.648 INFO:tasks.cephfs.filesystem:scrub status for tag:eed175ba-9b9e-48b7-9a74-65e2e6268c70 - {'path': '/', 'tag': 'eed175ba-9b9e-48b7-9a74-65e2e6268c70', 'options': 'recursive,force'} 2024-09-15T22:05:18.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:18 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:05:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:18 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:05:19.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:18 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:05:20.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:19 smithi028 ceph-mon[30969]: pgmap v244: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 170 B/s wr, 13 op/s 2024-09-15T22:05:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:19 smithi203 ceph-mon[35546]: pgmap v244: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 170 B/s wr, 13 op/s 2024-09-15T22:05:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:19 smithi174 ceph-mon[35244]: pgmap v244: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 170 B/s wr, 13 op/s 2024-09-15T22:05:22.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:21 smithi028 ceph-mon[30969]: pgmap v245: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T22:05:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:21 smithi203 ceph-mon[35546]: pgmap v245: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T22:05:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:21 smithi174 ceph-mon[35244]: pgmap v245: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T22:05:24.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:23 smithi028 ceph-mon[30969]: pgmap v246: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 85 B/s wr, 13 op/s 2024-09-15T22:05:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:23 smithi203 ceph-mon[35546]: pgmap v246: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 85 B/s wr, 13 op/s 2024-09-15T22:05:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:23 smithi174 ceph-mon[35244]: pgmap v246: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 85 B/s wr, 13 op/s 2024-09-15T22:05:26.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:25 smithi028 ceph-mon[30969]: pgmap v247: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:05:26.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:25 smithi203 ceph-mon[35546]: pgmap v247: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:05:26.274 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:25 smithi174 ceph-mon[35244]: pgmap v247: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:05:28.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:27 smithi028 ceph-mon[30969]: pgmap v248: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:05:28.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:27 smithi203 ceph-mon[35546]: pgmap v248: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:05:28.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:27 smithi174 ceph-mon[35244]: pgmap v248: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:05:30.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:29 smithi028 ceph-mon[30969]: pgmap v249: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:05:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:29 smithi203 ceph-mon[35546]: pgmap v249: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:05:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:29 smithi174 ceph-mon[35244]: pgmap v249: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:05:32.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:31 smithi028 ceph-mon[30969]: pgmap v250: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-15T22:05:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:31 smithi203 ceph-mon[35546]: pgmap v250: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-15T22:05:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:31 smithi174 ceph-mon[35244]: pgmap v250: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-15T22:05:34.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:33 smithi028 ceph-mon[30969]: pgmap v251: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:05:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:33 smithi203 ceph-mon[35546]: pgmap v251: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:05:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:33 smithi174 ceph-mon[35244]: pgmap v251: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:05:36.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:35 smithi028 ceph-mon[30969]: pgmap v252: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:36.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:35 smithi203 ceph-mon[35546]: pgmap v252: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:36.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:35 smithi174 ceph-mon[35244]: pgmap v252: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:37 smithi028 ceph-mon[30969]: pgmap v253: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:37 smithi203 ceph-mon[35546]: pgmap v253: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:37 smithi174 ceph-mon[35244]: pgmap v253: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:40.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:39 smithi028 ceph-mon[30969]: pgmap v254: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:40.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:39 smithi203 ceph-mon[35546]: pgmap v254: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:40.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:39 smithi174 ceph-mon[35244]: pgmap v254: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:42.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:41 smithi028 ceph-mon[30969]: pgmap v255: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:42.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:41 smithi203 ceph-mon[35546]: pgmap v255: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:41 smithi174 ceph-mon[35244]: pgmap v255: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:44.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:43 smithi028 ceph-mon[30969]: pgmap v256: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:44.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:43 smithi203 ceph-mon[35546]: pgmap v256: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:44.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:43 smithi174 ceph-mon[35244]: pgmap v256: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:45 smithi028 ceph-mon[30969]: pgmap v257: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:45 smithi203 ceph-mon[35546]: pgmap v257: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:45 smithi174 ceph-mon[35244]: pgmap v257: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:48.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:47 smithi028 ceph-mon[30969]: pgmap v258: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:48.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:47 smithi203 ceph-mon[35546]: pgmap v258: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:47 smithi174 ceph-mon[35244]: pgmap v258: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-15T22:05:48.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:05:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:49.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 -- 172.21.15.28:0/880851147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 msgr2=0x7f47941021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 --2- 172.21.15.28:0/880851147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f47941021d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4788009900 tx=0x7f478802a410 comp rx=0 tx=0).stop 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 -- 172.21.15.28:0/880851147 shutdown_connections 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 --2- 172.21.15.28:0/880851147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4794106f20 0x7f479410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 --2- 172.21.15.28:0/880851147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941027a0 0x7f47941067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 --2- 172.21.15.28:0/880851147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f47941021d0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 -- 172.21.15.28:0/880851147 >> 172.21.15.28:0/880851147 conn(0x7f47940fd8b0 msgr2=0x7f47940ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 -- 172.21.15.28:0/880851147 shutdown_connections 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 -- 172.21.15.28:0/880851147 wait complete. 2024-09-15T22:05:49.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 Processor -- start 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.600+0000 7f479b25e640 1 -- start start 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f479b25e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f4794198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f479b25e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941027a0 0x7f4794198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f479b25e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4794106f20 0x7f479419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f479b25e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4794111fd0 con 0x7f4794101df0 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f479b25e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4794111e50 con 0x7f4794106f20 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f479b25e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4794112150 con 0x7f47941027a0 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f4794198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f4794198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35708/0 (socket says 172.21.15.28:35708) 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 -- 172.21.15.28:0/432004229 learned_addr learned my addr 172.21.15.28:0/432004229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4793fff640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941027a0 0x7f4794198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f47997fb640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4794106f20 0x7f479419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 -- 172.21.15.28:0/432004229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941027a0 msgr2=0x7f4794198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941027a0 0x7f4794198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 -- 172.21.15.28:0/432004229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4794106f20 msgr2=0x7f479419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4794106f20 0x7f479419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.601+0000 7f4798ffa640 1 -- 172.21.15.28:0/432004229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479419d3e0 con 0x7f4794101df0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.602+0000 7f4798ffa640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f4794198430 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f4788004770 tx=0x7f4788003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.602+0000 7f4791ffb640 1 -- 172.21.15.28:0/432004229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478803f070 con 0x7f4794101df0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.602+0000 7f4791ffb640 1 -- 172.21.15.28:0/432004229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4788031ca0 con 0x7f4794101df0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.602+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479419d670 con 0x7f4794101df0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.602+0000 7f4791ffb640 1 -- 172.21.15.28:0/432004229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788035030 con 0x7f4794101df0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.602+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47941a4fb0 con 0x7f4794101df0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.604+0000 7f4791ffb640 1 -- 172.21.15.28:0/432004229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f4788042070 con 0x7f4794101df0 2024-09-15T22:05:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 Processor -- start 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 -- start start 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea928760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea937550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea939e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560dea88e2e0 con 0x560dea928380 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560dea835ce0 con 0x560dea928ca0 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f479d637740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560dea88a2a0 con 0x560dea937a90 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f47997fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea928760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f47997fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea928760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35724/0 (socket says 172.21.15.28:35724) 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f47997fb640 1 -- 172.21.15.28:0/2125287841 learned_addr learned my addr 172.21.15.28:0/2125287841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f4798ffa640 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea937550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f4793fff640 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea939e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.605+0000 7f4791ffb640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f47840a33b0 0x7f47840a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f47997fb640 1 -- 172.21.15.28:0/2125287841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 msgr2=0x560dea939e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f47997fb640 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea939e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f47997fb640 1 -- 172.21.15.28:0/2125287841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 msgr2=0x560dea937550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f47997fb640 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea937550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f47997fb640 1 -- 172.21.15.28:0/2125287841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dea81a440 con 0x560dea928380 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f4791ffb640 1 -- 172.21.15.28:0/432004229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f47880f0790 con 0x7f4794101df0 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f47997fb640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f47840a33b0 0x7f47840a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f47997fb640 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea928760 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f478c009fd0 tx=0x7f478c00f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225ed6ea7764c5d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f477effd640 1 -- 172.21.15.28:0/2125287841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c01c910 con 0x560dea928380 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f477effd640 1 -- 172.21.15.28:0/2125287841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c01b440 con 0x560dea928380 2024-09-15T22:05:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.606+0000 7f477effd640 1 -- 172.21.15.28:0/2125287841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c010630 con 0x560dea928380 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 -- 172.21.15.28:0/2125287841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 msgr2=0x560dea928760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea928760 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f478c009fd0 tx=0x7f478c00f870 comp rx=0 tx=0).stop 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f47997fb640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f47840a33b0 0x7f47840a5870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f478c020000 tx=0x7f478c01d000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 -- 172.21.15.28:0/2125287841 shutdown_connections 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea939e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea937550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 --2- 172.21.15.28:0/2125287841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea928760 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 -- 172.21.15.28:0/2125287841 >> 172.21.15.28:0/2125287841 conn(0x560dea924a30 msgr2=0x560dea924e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 -- 172.21.15.28:0/2125287841 shutdown_connections 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 -- 172.21.15.28:0/2125287841 wait complete. 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.607+0000 7f479d637740 1 -- start start 2024-09-15T22:05:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 1 Processor -- start 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928380 0x560dea937400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928ca0 0x560dea9346b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea934c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560dea93a660 con 0x560dea928ca0 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560dea849050 con 0x560dea928380 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560dea93a7e0 con 0x560dea937a90 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f479d637740 10 client.0 _ll_register_callbacks cb 0x560dea889160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928ca0 0x560dea9346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928ca0 0x560dea9346b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35726/0 (socket says 172.21.15.28:35726) 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 -- 172.21.15.28:0/3364579226 learned_addr learned my addr 172.21.15.28:0/3364579226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4798ffa640 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928380 0x560dea937400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f47997fb640 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea934c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 -- 172.21.15.28:0/3364579226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 msgr2=0x560dea934c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea934c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 -- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928380 msgr2=0x560dea937400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928380 0x560dea937400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.608+0000 7f4793fff640 1 -- 172.21.15.28:0/3364579226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dea5ff3c0 con 0x560dea928ca0 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f4793fff640 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928ca0 0x560dea9346b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f478000f650 tx=0x7f478000fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee59b0144b0b2a0d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f4773fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f4773fff640 1 -- 172.21.15.28:0/3364579226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478000fe20 con 0x560dea928ca0 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f478c01b440 con 0x560dea928ca0 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f4773fff640 1 -- 172.21.15.28:0/3364579226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f478000eb20 con 0x560dea928ca0 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f4773fff640 1 -- 172.21.15.28:0/3364579226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f478002d050 con 0x560dea928ca0 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f479d637740 10 client.17001 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560dea92b190 con 0x560dea928ca0 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.609+0000 7f4773fff640 1 -- 172.21.15.28:0/3364579226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c01b440 con 0x560dea928ca0 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f4773fff640 1 -- 172.21.15.28:0/3364579226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f478001a0d0 con 0x560dea928ca0 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f479d637740 10 client.17001 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f479d637740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f479d637740 20 client.17001 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f479d637740 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea92c5f0 0x560dea92ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f479d637740 4 client.17001 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560dea89b850 con 0x560dea92c5f0 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f4798ffa640 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea92c5f0 0x560dea92ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f4773fff640 10 client.17001 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.610+0000 7f4798ffa640 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea92c5f0 0x560dea92ea10 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58171e87cb4fcc45 server_cookie=d986b94351c287a1 in_seq=0 out_seq=0 2024-09-15T22:05:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.611+0000 7f4773fff640 1 -- 172.21.15.28:0/3364579226 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560dea89b850 con 0x560dea92c5f0 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.611+0000 7f4773fff640 10 client.17001 handle_command_reply: tid=0 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.611+0000 7f479d637740 1 client.17001 shutdown 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.611+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928ca0 msgr2=0x560dea9346b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.611+0000 7f479d637740 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928ca0 0x560dea9346b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f478000f650 tx=0x7f478000fb10 comp rx=0 tx=0).stop 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 shutdown_connections 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea92c5f0 0x560dea92ea10 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea937a90 0x560dea934c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928ca0 0x560dea9346b0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 --2- 172.21.15.28:0/3364579226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928380 0x560dea937400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 >> 172.21.15.28:0/3364579226 conn(0x560dea924a30 msgr2=0x560dea92a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f4773fff640 0 client.17001 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 shutdown_connections 2024-09-15T22:05:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 1 -- 172.21.15.28:0/3364579226 wait complete. 2024-09-15T22:05:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.612+0000 7f479d637740 20 client.17001 trim_cache size 0 max 16384 2024-09-15T22:05:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 Processor -- start 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 -- start start 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928380 0x560dea931910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea931e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea937a90 0x560dea923350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47941ace50 con 0x560dea937a90 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f47941ae9c0 con 0x560dea928ca0 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.629+0000 7f479d637740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f47941ad520 con 0x560dea928380 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea937a90 0x560dea923350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea937a90 0x560dea923350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35742/0 (socket says 172.21.15.28:35742) 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 -- 172.21.15.28:0/3963459288 learned_addr learned my addr 172.21.15.28:0/3963459288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f4793fff640 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea931e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f4798ffa640 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928380 0x560dea931910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 -- 172.21.15.28:0/3963459288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928380 msgr2=0x560dea931910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928380 0x560dea931910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 -- 172.21.15.28:0/3963459288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 msgr2=0x560dea931e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea931e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 -- 172.21.15.28:0/3963459288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dea9244d0 con 0x560dea937a90 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f4793fff640 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea931e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f47997fb640 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea937a90 0x560dea923350 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f478c009fd0 tx=0x7f478c00ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a171fa6ffe249bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f477e7fc640 1 -- 172.21.15.28:0/3963459288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c036070 con 0x560dea937a90 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f477e7fc640 1 -- 172.21.15.28:0/3963459288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c009bd0 con 0x560dea937a90 2024-09-15T22:05:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.630+0000 7f477e7fc640 1 -- 172.21.15.28:0/3963459288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c01c890 con 0x560dea937a90 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 -- 172.21.15.28:0/3963459288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea937a90 msgr2=0x560dea923350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea937a90 0x560dea923350 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f478c009fd0 tx=0x7f478c00ffd0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 -- 172.21.15.28:0/3963459288 shutdown_connections 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea937a90 0x560dea923350 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea928ca0 0x560dea931e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 --2- 172.21.15.28:0/3963459288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928380 0x560dea931910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 -- 172.21.15.28:0/3963459288 >> 172.21.15.28:0/3963459288 conn(0x560dea924a30 msgr2=0x560dea9337a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 -- 172.21.15.28:0/3963459288 shutdown_connections 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 -- 172.21.15.28:0/3963459288 wait complete. 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.631+0000 7f479d637740 1 -- start start 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 1 Processor -- start 2024-09-15T22:05:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea948640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928ca0 0x560dea948b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea937a90 0x560dea9ac070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47941ae2f0 con 0x560dea928380 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f47941adc20 con 0x560dea937a90 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f47941aae50 con 0x560dea928ca0 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f479d637740 10 client.0 _ll_register_callbacks cb 0x7f4780009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4798ffa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea948640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4798ffa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea948640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35754/0 (socket says 172.21.15.28:35754) 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4798ffa640 1 -- 172.21.15.28:0/9617406 learned_addr learned my addr 172.21.15.28:0/9617406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4793fff640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928ca0 0x560dea948b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f47997fb640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea937a90 0x560dea9ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4798ffa640 1 -- 172.21.15.28:0/9617406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928ca0 msgr2=0x560dea948b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4798ffa640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928ca0 0x560dea948b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4798ffa640 1 -- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea937a90 msgr2=0x560dea9ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.632+0000 7f4798ffa640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea937a90 0x560dea9ac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4798ffa640 1 -- 172.21.15.28:0/9617406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dea9ac6b0 con 0x560dea928380 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f47997fb640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea937a90 0x560dea9ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4798ffa640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea948640 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f47880b8b20 tx=0x7f47880b8a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e497261dec35d9a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4773fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4773fff640 1 -- 172.21.15.28:0/9617406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788049040 con 0x560dea928380 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4773fff640 1 -- 172.21.15.28:0/9617406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f47880ba760 con 0x560dea928380 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f478c009bd0 con 0x560dea928380 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4773fff640 1 -- 172.21.15.28:0/9617406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f47880bc020 con 0x560dea928380 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 10 client.17007 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560dea9ad9e0 con 0x560dea928380 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4773fff640 1 -- 172.21.15.28:0/9617406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c009bd0 con 0x560dea928380 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f4773fff640 1 -- 172.21.15.28:0/9617406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f47880f0040 con 0x560dea928380 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 10 client.17007 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 20 client.17007 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea9ae100 0x560dea9b0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 4 client.17007 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.633+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560dea895d60 con 0x560dea9ae100 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.634+0000 7f4793fff640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea9ae100 0x560dea9b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.634+0000 7f4793fff640 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea9ae100 0x560dea9b0520 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d41a6f5ad6b5a3 server_cookie=6ee073c368203afd in_seq=0 out_seq=0 2024-09-15T22:05:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.634+0000 7f4773fff640 10 client.17007 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.634+0000 7f4773fff640 1 -- 172.21.15.28:0/9617406 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560dea895d60 con 0x560dea9ae100 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.634+0000 7f4773fff640 10 client.17007 handle_command_reply: tid=0 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.634+0000 7f479d637740 1 client.17007 shutdown 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 msgr2=0x560dea948640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea948640 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f47880b8b20 tx=0x7f47880b8a90 comp rx=0 tx=0).stop 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 shutdown_connections 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x560dea9ae100 0x560dea9b0520 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560dea937a90 0x560dea9ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560dea928ca0 0x560dea948b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 --2- 172.21.15.28:0/9617406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560dea928380 0x560dea948640 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 >> 172.21.15.28:0/9617406 conn(0x560dea924a30 msgr2=0x560dea935f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f4773fff640 0 client.17007 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 shutdown_connections 2024-09-15T22:05:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 1 -- 172.21.15.28:0/9617406 wait complete. 2024-09-15T22:05:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.635+0000 7f479d637740 20 client.17007 trim_cache size 0 max 16384 2024-09-15T22:05:49.650 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:05:49.651 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:05:49.651 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:05:49.651 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:05:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f47840a33b0 msgr2=0x7f47840a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f47840a33b0 0x7f47840a5870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f478c020000 tx=0x7f478c01d000 comp rx=0 tx=0).stop 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 msgr2=0x7f4794198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f4794198430 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f4788004770 tx=0x7f4788003c10 comp rx=0 tx=0).stop 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 shutdown_connections 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f47840a33b0 0x7f47840a5870 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4794106f20 0x7f479419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941027a0 0x7f4794198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 --2- 172.21.15.28:0/432004229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4794101df0 0x7f4794198430 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 >> 172.21.15.28:0/432004229 conn(0x7f47940fd8b0 msgr2=0x7f47940feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 shutdown_connections 2024-09-15T22:05:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:49.636+0000 7f479b25e640 1 -- 172.21.15.28:0/432004229 wait complete. 2024-09-15T22:05:50.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:49 smithi028 ceph-mon[30969]: pgmap v259: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:49 smithi203 ceph-mon[35546]: pgmap v259: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:49 smithi174 ceph-mon[35244]: pgmap v259: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-15T22:05:50.518 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:05:50.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:05:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 -- 172.21.15.28:0/3919689510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740103660 msgr2=0x7fd74010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 --2- 172.21.15.28:0/3919689510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740103660 0x7fd74010bb50 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fd730009900 tx=0x7fd73002a480 comp rx=0 tx=0).stop 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 -- 172.21.15.28:0/3919689510 shutdown_connections 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 --2- 172.21.15.28:0/3919689510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd74010c170 0x7fd74010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 --2- 172.21.15.28:0/3919689510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740103660 0x7fd74010bb50 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 --2- 172.21.15.28:0/3919689510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd740102d40 0x7fd740103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 -- 172.21.15.28:0/3919689510 >> 172.21.15.28:0/3919689510 conn(0x7fd7400fc7a0 msgr2=0x7fd7400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 -- 172.21.15.28:0/3919689510 shutdown_connections 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 -- 172.21.15.28:0/3919689510 wait complete. 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 Processor -- start 2024-09-15T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.267+0000 7fd7484b7640 1 -- start start 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd7484b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740102d40 0x7fd740198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd7484b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd740103660 0x7fd740198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd7484b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd74010c170 0x7fd74019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd7484b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd740111fc0 con 0x7fd740102d40 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd7484b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd740111e40 con 0x7fd74010c170 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd7484b7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd740112140 con 0x7fd740103660 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740102d40 0x7fd740198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740102d40 0x7fd740198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59584/0 (socket says 172.21.15.28:59584) 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 -- 172.21.15.28:0/39420843 learned_addr learned my addr 172.21.15.28:0/39420843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd745a52640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd740103660 0x7fd740198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746a54640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd74010c170 0x7fd74019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 -- 172.21.15.28:0/39420843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd740103660 msgr2=0x7fd740198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd740103660 0x7fd740198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 -- 172.21.15.28:0/39420843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd74010c170 msgr2=0x7fd74019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd74010c170 0x7fd74019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 -- 172.21.15.28:0/39420843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74019d400 con 0x7fd740102d40 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746a54640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd74010c170 0x7fd74019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd746253640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740102d40 0x7fd740198450 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00e9f0 tx=0x7fd73c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.268+0000 7fd7377fe640 1 -- 172.21.15.28:0/39420843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c00cd80 con 0x7fd740102d40 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.269+0000 7fd7377fe640 1 -- 172.21.15.28:0/39420843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd73c010910 con 0x7fd740102d40 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.269+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd74019d6f0 con 0x7fd740102d40 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.269+0000 7fd7377fe640 1 -- 172.21.15.28:0/39420843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c018990 con 0x7fd740102d40 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.269+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7401a4fd0 con 0x7fd740102d40 2024-09-15T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.270+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd740103120 con 0x7fd740102d40 2024-09-15T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.271+0000 7fd7377fe640 1 -- 172.21.15.28:0/39420843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd73c010ac0 con 0x7fd740102d40 2024-09-15T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.271+0000 7fd7377fe640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd72c0a3430 0x7fd72c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.272+0000 7fd745a52640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd72c0a3430 0x7fd72c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.272+0000 7fd7377fe640 1 -- 172.21.15.28:0/39420843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fd73c0d03f0 con 0x7fd740102d40 2024-09-15T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.272+0000 7fd745a52640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd72c0a3430 0x7fd72c0a58f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd730000c00 tx=0x7fd730005b50 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.276+0000 7fd7377fe640 1 -- 172.21.15.28:0/39420843 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd73c09ca20 con 0x7fd740102d40 2024-09-15T22:05:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.489+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd74002d7c0 con 0x7fd740102d40 2024-09-15T22:05:51.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.489+0000 7fd7377fe640 1 -- 172.21.15.28:0/39420843 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11988 (secure 0 0 0) 0x7fd73c09c320 con 0x7fd740102d40 2024-09-15T22:05:51.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:05:51.493 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"btime":"2024-09-15T22:04:37:022811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:04:36.713632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:05:51.493 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-09-15T22:05:51.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.492+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd72c0a3430 msgr2=0x7fd72c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:51.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.492+0000 7fd7484b7640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd72c0a3430 0x7fd72c0a58f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd730000c00 tx=0x7fd730005b50 comp rx=0 tx=0).stop 2024-09-15T22:05:51.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.492+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740102d40 msgr2=0x7fd740198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.492+0000 7fd7484b7640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740102d40 0x7fd740198450 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00e9f0 tx=0x7fd73c00eeb0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 shutdown_connections 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd72c0a3430 0x7fd72c0a58f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd74010c170 0x7fd74019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd740103660 0x7fd740198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 --2- 172.21.15.28:0/39420843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd740102d40 0x7fd740198450 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 >> 172.21.15.28:0/39420843 conn(0x7fd7400fc7a0 msgr2=0x7fd7400fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 shutdown_connections 2024-09-15T22:05:51.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:51.493+0000 7fd7484b7640 1 -- 172.21.15.28:0/39420843 wait complete. 2024-09-15T22:05:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:51 smithi028 ceph-mon[30969]: pgmap v260: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:52.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:51 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/39420843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:05:52.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:05:52.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:51 smithi203 ceph-mon[35546]: pgmap v260: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:52.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:51 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/39420843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:05:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:51 smithi174 ceph-mon[35244]: pgmap v260: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:51 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/39420843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:05:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.979+0000 7f56935d9640 1 -- 172.21.15.28:0/3700547471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 msgr2=0x7f568c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.979+0000 7f56935d9640 1 --2- 172.21.15.28:0/3700547471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c1067e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f567c009900 tx=0x7f567c02a3f0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.979+0000 7f56935d9640 1 -- 172.21.15.28:0/3700547471 shutdown_connections 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.979+0000 7f56935d9640 1 --2- 172.21.15.28:0/3700547471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f568c106f10 0x7f568c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.979+0000 7f56935d9640 1 --2- 172.21.15.28:0/3700547471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c1067e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.979+0000 7f56935d9640 1 --2- 172.21.15.28:0/3700547471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f568c101de0 0x7f568c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.979+0000 7f56935d9640 1 -- 172.21.15.28:0/3700547471 >> 172.21.15.28:0/3700547471 conn(0x7f568c0fd8c0 msgr2=0x7f568c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.980+0000 7f56935d9640 1 -- 172.21.15.28:0/3700547471 shutdown_connections 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.980+0000 7f56935d9640 1 -- 172.21.15.28:0/3700547471 wait complete. 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.980+0000 7f56935d9640 1 Processor -- start 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.980+0000 7f56935d9640 1 -- start start 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56935d9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f568c101de0 0x7f568c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56935d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56935d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f568c106f10 0x7f568c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56935d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f568c111f90 con 0x7f568c102790 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56935d9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f568c111e10 con 0x7f568c101de0 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56935d9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f568c112110 con 0x7f568c106f10 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59598/0 (socket says 172.21.15.28:59598) 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 -- 172.21.15.28:0/1790428248 learned_addr learned my addr 172.21.15.28:0/1790428248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5691b76640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f568c106f10 0x7f568c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5691375640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f568c101de0 0x7f568c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 -- 172.21.15.28:0/1790428248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f568c106f10 msgr2=0x7f568c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f568c106f10 0x7f568c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 -- 172.21.15.28:0/1790428248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f568c101de0 msgr2=0x7f568c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f568c101de0 0x7f568c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 -- 172.21.15.28:0/1790428248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c19d3c0 con 0x7f568c102790 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5691b76640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f568c106f10 0x7f568c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f5690b74640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c198950 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f567c005ab0 tx=0x7f567c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56827fc640 1 -- 172.21.15.28:0/1790428248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c03f030 con 0x7f568c102790 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56827fc640 1 -- 172.21.15.28:0/1790428248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f567c003ce0 con 0x7f568c102790 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568c19d650 con 0x7f568c102790 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.981+0000 7f56827fc640 1 -- 172.21.15.28:0/1790428248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c035070 con 0x7f568c102790 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.982+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568c1a4f90 con 0x7f568c102790 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.984+0000 7f56827fc640 1 -- 172.21.15.28:0/1790428248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f567c042030 con 0x7f568c102790 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.984+0000 7f56959b2740 1 Processor -- start 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.984+0000 7f56959b2740 1 -- start start 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56959b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56959b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56959b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56959b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e4b8112f0 con 0x561e4b8ab5d0 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56959b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e4b7b8cf0 con 0x561e4b8ba780 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56959b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561e4b80d2b0 con 0x561e4b8acb40 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59600/0 (socket says 172.21.15.28:59600) 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 -- 172.21.15.28:0/3797862644 learned_addr learned my addr 172.21.15.28:0/3797862644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56827fc640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56780a3430 0x7f56780a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5690b74640 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691b76640 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691b76640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56780a3430 0x7f56780a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 -- 172.21.15.28:0/3797862644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 msgr2=0x561e4b8ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f56827fc640 1 -- 172.21.15.28:0/1790428248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f567c0f0a70 con 0x7f568c102790 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 -- 172.21.15.28:0/3797862644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 msgr2=0x561e4b8bcb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bcb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 -- 172.21.15.28:0/3797862644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e4b79d3a0 con 0x561e4b8ab5d0 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691375640 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8aba10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f568800b820 tx=0x7f568800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23471df3971c1a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.985+0000 7f5691b76640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56780a3430 0x7f56780a58f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5684007310 tx=0x7f56840072a0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56777fe640 1 -- 172.21.15.28:0/3797862644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568800d950 con 0x561e4b8ab5d0 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56777fe640 1 -- 172.21.15.28:0/3797862644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5688019070 con 0x561e4b8ab5d0 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56777fe640 1 -- 172.21.15.28:0/3797862644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56880123e0 con 0x561e4b8ab5d0 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 -- 172.21.15.28:0/3797862644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 msgr2=0x561e4b8aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8aba10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f568800b820 tx=0x7f568800bce0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 -- 172.21.15.28:0/3797862644 shutdown_connections 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8aba10 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 --2- 172.21.15.28:0/3797862644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 -- 172.21.15.28:0/3797862644 >> 172.21.15.28:0/3797862644 conn(0x561e4b874310 msgr2=0x561e4b8a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.986+0000 7f56959b2740 1 -- 172.21.15.28:0/3797862644 shutdown_connections 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 -- 172.21.15.28:0/3797862644 wait complete. 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 -- start start 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 Processor -- start 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8ab5d0 0x561e4b8ba110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8acb40 0x561e4b8b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ba780 0x561e4b8b7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e4b8bd480 con 0x561e4b8ba780 2024-09-15T22:05:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e4b7cc060 con 0x561e4b8acb40 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561e4b8bd600 con 0x561e4b8ab5d0 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f56959b2740 10 client.0 _ll_register_callbacks cb 0x561e4b51f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.987+0000 7f5691b76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ba780 0x561e4b8b7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ba780 0x561e4b8b7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59602/0 (socket says 172.21.15.28:59602) 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 -- 172.21.15.28:0/3136275813 learned_addr learned my addr 172.21.15.28:0/3136275813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691375640 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8ab5d0 0x561e4b8ba110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5690b74640 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8acb40 0x561e4b8b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 -- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8ab5d0 msgr2=0x561e4b8ba110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8ab5d0 0x561e4b8ba110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 -- 172.21.15.28:0/3136275813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8acb40 msgr2=0x561e4b8b73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8acb40 0x561e4b8b73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 -- 172.21.15.28:0/3136275813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e4b5823c0 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5691b76640 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ba780 0x561e4b8b7940 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f568400e6e0 tx=0x7f5684004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f861417d90bd194 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5674ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5674ff9640 1 -- 172.21.15.28:0/3136275813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568401f860 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5674ff9640 1 -- 172.21.15.28:0/3136275813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f568401fa00 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5688019070 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f5674ff9640 1 -- 172.21.15.28:0/3136275813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f5684032050 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f56959b2740 10 client.17025 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.988+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561e4b8c8eb0 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f5674ff9640 1 -- 172.21.15.28:0/3136275813 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5688019070 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f5674ff9640 1 -- 172.21.15.28:0/3136275813 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f568401c040 con 0x561e4b8ba780 2024-09-15T22:05:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f56959b2740 10 client.17025 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f56959b2740 10 client.17025 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f56959b2740 20 client.17025 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f56959b2740 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b8cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f56959b2740 4 client.17025 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e4b81e860 con 0x561e4b8ca370 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.989+0000 7f5691375640 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b8cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.990+0000 7f5691375640 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b8cc7f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7cd82582b1aea92c server_cookie=406def0ee11cce60 in_seq=0 out_seq=0 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.990+0000 7f5674ff9640 10 client.17025 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.990+0000 7f5674ff9640 1 -- 172.21.15.28:0/3136275813 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e4b81e860 con 0x561e4b8ca370 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.990+0000 7f5674ff9640 10 client.17025 handle_command_reply: tid=0 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.990+0000 7f56959b2740 1 client.17025 shutdown 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ba780 msgr2=0x561e4b8b7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ba780 0x561e4b8b7940 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f568400e6e0 tx=0x7f5684004a40 comp rx=0 tx=0).stop 2024-09-15T22:05:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 shutdown_connections 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b8cc7f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f5674ff9640 0 client.17025 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ba780 0x561e4b8b7940 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8acb40 0x561e4b8b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 --2- 172.21.15.28:0/3136275813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8ab5d0 0x561e4b8ba110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 >> 172.21.15.28:0/3136275813 conn(0x561e4b874310 msgr2=0x561e4b8ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 shutdown_connections 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 1 -- 172.21.15.28:0/3136275813 wait complete. 2024-09-15T22:05:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:52.991+0000 7f56959b2740 20 client.17025 trim_cache size 0 max 16384 2024-09-15T22:05:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 Processor -- start 2024-09-15T22:05:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 -- start start 2024-09-15T22:05:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f568c1acd90 con 0x561e4b8ab5d0 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f568c1ae900 con 0x561e4b8ba780 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f56959b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f568c1ad460 con 0x561e4b8acb40 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5691375640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5691375640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8a7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59610/0 (socket says 172.21.15.28:59610) 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5691375640 1 -- 172.21.15.28:0/1948924769 learned_addr learned my addr 172.21.15.28:0/1948924769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5690b74640 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5691375640 1 -- 172.21.15.28:0/1948924769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 msgr2=0x561e4b8c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5691375640 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5691375640 1 -- 172.21.15.28:0/1948924769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 msgr2=0x561e4b8d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.004+0000 7f5691b76640 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.005+0000 7f5691375640 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8d20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.005+0000 7f5691375640 1 -- 172.21.15.28:0/1948924769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e4b8b5910 con 0x561e4b8ab5d0 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.005+0000 7f5691b76640 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.005+0000 7f5691375640 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8a7e10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f47f702fe35aa42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.005+0000 7f5676ffd640 1 -- 172.21.15.28:0/1948924769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568802b070 con 0x561e4b8ab5d0 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.005+0000 7f5676ffd640 1 -- 172.21.15.28:0/1948924769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5688019040 con 0x561e4b8ab5d0 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.005+0000 7f5676ffd640 1 -- 172.21.15.28:0/1948924769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688027770 con 0x561e4b8ab5d0 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 -- 172.21.15.28:0/1948924769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 msgr2=0x561e4b8a7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8a7e10 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688008040 comp rx=0 tx=0).stop 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 -- 172.21.15.28:0/1948924769 shutdown_connections 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 --2- 172.21.15.28:0/1948924769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8a7e10 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 -- 172.21.15.28:0/1948924769 >> 172.21.15.28:0/1948924769 conn(0x561e4b874310 msgr2=0x561e4b8a7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 -- 172.21.15.28:0/1948924769 shutdown_connections 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 -- 172.21.15.28:0/1948924769 wait complete. 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 -- start start 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.006+0000 7f56959b2740 1 Processor -- start 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f568c1ae230 con 0x561e4b8ab5d0 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f568c1adb60 con 0x561e4b8ba780 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f568c1aad90 con 0x561e4b8acb40 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 10 client.0 _ll_register_callbacks cb 0x7f5684010dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8c2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59626/0 (socket says 172.21.15.28:59626) 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 -- 172.21.15.28:0/3131186824 learned_addr learned my addr 172.21.15.28:0/3131186824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691b76640 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5690b74640 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 -- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 msgr2=0x561e4b8c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 -- 172.21.15.28:0/3131186824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 msgr2=0x561e4b8bdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 -- 172.21.15.28:0/3131186824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e4b8be810 con 0x561e4b8ab5d0 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5691375640 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8c2bb0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5688004810 tx=0x7f568800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c416c5ca55d4885 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5674ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5674ff9640 1 -- 172.21.15.28:0/3131186824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688035070 con 0x561e4b8ab5d0 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f5674ff9640 1 -- 172.21.15.28:0/3131186824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f568800db50 con 0x561e4b8ab5d0 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.007+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5688019040 con 0x561e4b8ab5d0 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f5674ff9640 1 -- 172.21.15.28:0/3131186824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f5688041050 con 0x561e4b8ab5d0 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 10 client.17031 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561e4b934be0 con 0x561e4b8ab5d0 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f5674ff9640 1 -- 172.21.15.28:0/3131186824 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5688019040 con 0x561e4b8ab5d0 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f5674ff9640 1 -- 172.21.15.28:0/3131186824 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f5688013070 con 0x561e4b8ab5d0 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 10 client.17031 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 10 client.17031 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 20 client.17031 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b9370c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 4 client.17031 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.008+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e4b818d70 con 0x561e4b8ca370 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f5690b74640 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b9370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f5674ff9640 10 client.17031 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f5690b74640 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b9370c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bfc8ea9f40264285 server_cookie=94341f51831a259f in_seq=0 out_seq=0 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f5674ff9640 1 -- 172.21.15.28:0/3131186824 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e4b818d70 con 0x561e4b8ca370 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f5674ff9640 10 client.17031 handle_command_reply: tid=0 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 client.17031 shutdown 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 msgr2=0x561e4b8c2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8c2bb0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5688004810 tx=0x7f568800bf50 comp rx=0 tx=0).stop 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 shutdown_connections 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561e4b8ca370 0x561e4b9370c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e4b8ba780 0x561e4b8bdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f5674ff9640 0 client.17031 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:05:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561e4b8acb40 0x561e4b8c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 --2- 172.21.15.28:0/3131186824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e4b8ab5d0 0x561e4b8c2bb0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.009+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 >> 172.21.15.28:0/3131186824 conn(0x561e4b874310 msgr2=0x561e4b8b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 shutdown_connections 2024-09-15T22:05:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56959b2740 1 -- 172.21.15.28:0/3131186824 wait complete. 2024-09-15T22:05:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56959b2740 20 client.17031 trim_cache size 0 max 16384 2024-09-15T22:05:53.027 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56780a3430 msgr2=0x7f56780a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56780a3430 0x7f56780a58f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5684007310 tx=0x7f56840072a0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 msgr2=0x7f568c198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c198950 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f567c005ab0 tx=0x7f567c0042f0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 shutdown_connections 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56780a3430 0x7f56780a58f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f568c106f10 0x7f568c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f568c102790 0x7f568c198950 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 --2- 172.21.15.28:0/1790428248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f568c101de0 0x7f568c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:53.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 >> 172.21.15.28:0/1790428248 conn(0x7f568c0fd8c0 msgr2=0x7f568c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:53.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.010+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 shutdown_connections 2024-09-15T22:05:53.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:53.011+0000 7f56935d9640 1 -- 172.21.15.28:0/1790428248 wait complete. 2024-09-15T22:05:53.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:05:53.940 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:54.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:53 smithi028 ceph-mon[30969]: pgmap v261: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:05:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:53 smithi203 ceph-mon[35546]: pgmap v261: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:05:54.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:53 smithi174 ceph-mon[35244]: pgmap v261: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:05:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 -- 172.21.15.28:0/900276116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 msgr2=0x7f4de0107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 --2- 172.21.15.28:0/900276116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0107bc0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009930 tx=0x7f4dd402a420 comp rx=0 tx=0).stop 2024-09-15T22:05:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 -- 172.21.15.28:0/900276116 shutdown_connections 2024-09-15T22:05:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 --2- 172.21.15.28:0/900276116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0108300 0x7f4de010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 --2- 172.21.15.28:0/900276116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0107bc0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 --2- 172.21.15.28:0/900276116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4de0102dd0 0x7f4de01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 -- 172.21.15.28:0/900276116 >> 172.21.15.28:0/900276116 conn(0x7f4de00fc7d0 msgr2=0x7f4de00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 -- 172.21.15.28:0/900276116 shutdown_connections 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 -- 172.21.15.28:0/900276116 wait complete. 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 Processor -- start 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.438+0000 7f4de548c640 1 -- start start 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4de548c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4de0102dd0 0x7f4de01985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4de548c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4de548c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0108300 0x7f4de019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4de548c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4de0112070 con 0x7f4de0103780 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4de548c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4de0111ef0 con 0x7f4de0102dd0 2024-09-15T22:05:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4de548c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4de01121f0 con 0x7f4de0108300 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59630/0 (socket says 172.21.15.28:59630) 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 -- 172.21.15.28:0/822517518 learned_addr learned my addr 172.21.15.28:0/822517518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4de0102dd0 0x7f4de01985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0108300 0x7f4de019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 -- 172.21.15.28:0/822517518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0108300 msgr2=0x7f4de019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0108300 0x7f4de019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 -- 172.21.15.28:0/822517518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4de0102dd0 msgr2=0x7f4de01985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4de0102dd0 0x7f4de01985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 -- 172.21.15.28:0/822517518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de019d5a0 con 0x7f4de0103780 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0108300 0x7f4de019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.439+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0198b30 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4005940 tx=0x7f4dd40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.440+0000 7f4dc7fff640 1 -- 172.21.15.28:0/822517518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd403f030 con 0x7f4de0103780 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.440+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de019d830 con 0x7f4de0103780 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.440+0000 7f4dc7fff640 1 -- 172.21.15.28:0/822517518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd4003ce0 con 0x7f4de0103780 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.440+0000 7f4dc7fff640 1 -- 172.21.15.28:0/822517518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4035070 con 0x7f4de0103780 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.440+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de01a5170 con 0x7f4de0103780 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.442+0000 7f4dc7fff640 1 -- 172.21.15.28:0/822517518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f4dd4042030 con 0x7f4de0103780 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.442+0000 7f4de7865740 1 Processor -- start 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.442+0000 7f4de7865740 1 -- start start 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4de7865740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7288b40 0x55a3c7287090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4de7865740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c7287a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4de7865740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7292370 0x55a3c7294760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4de7865740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3c71ed2f0 con 0x55a3c72875d0 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4de7865740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a3c7194cf0 con 0x55a3c7292370 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4de7865740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a3c71e92b0 con 0x55a3c7288b40 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4dc7fff640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4dc80a33b0 0x7f4dc80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c7287a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c7287a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59642/0 (socket says 172.21.15.28:59642) 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 -- 172.21.15.28:0/1778282044 learned_addr learned my addr 172.21.15.28:0/1778282044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4dc80a33b0 0x7f4dc80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7292370 0x55a3c7294760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7288b40 0x55a3c7287090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4dc7fff640 1 -- 172.21.15.28:0/822517518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f4dd40f05b0 con 0x7f4de0103780 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 -- 172.21.15.28:0/1778282044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7288b40 msgr2=0x55a3c7287090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7288b40 0x55a3c7287090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 -- 172.21.15.28:0/1778282044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7292370 msgr2=0x55a3c7294760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7292370 0x55a3c7294760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 -- 172.21.15.28:0/1778282044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c71793a0 con 0x55a3c72875d0 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7292370 0x55a3c7294760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c7287a10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0003a20 tx=0x7f4dd0003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bba449c54fbfa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.443+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4dc80a33b0 0x7f4dc80a5870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005d50 tx=0x7f4dd8005e90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4dc57fa640 1 -- 172.21.15.28:0/1778282044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0004860 con 0x55a3c72875d0 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4dc57fa640 1 -- 172.21.15.28:0/1778282044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd0015030 con 0x55a3c72875d0 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4dc57fa640 1 -- 172.21.15.28:0/1778282044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0010aa0 con 0x55a3c72875d0 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 -- 172.21.15.28:0/1778282044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 msgr2=0x55a3c7287a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c7287a10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0003a20 tx=0x7f4dd0003ee0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 -- 172.21.15.28:0/1778282044 shutdown_connections 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7292370 0x55a3c7294760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c7287a10 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 --2- 172.21.15.28:0/1778282044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7288b40 0x55a3c7287090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 -- 172.21.15.28:0/1778282044 >> 172.21.15.28:0/1778282044 conn(0x55a3c7250310 msgr2=0x55a3c7283c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 -- 172.21.15.28:0/1778282044 shutdown_connections 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.444+0000 7f4de7865740 1 -- 172.21.15.28:0/1778282044 wait complete. 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 -- start start 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 Processor -- start 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c7291d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c728efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3c7295070 con 0x55a3c7292370 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a3c71a8060 con 0x55a3c7288b40 2024-09-15T22:05:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a3c72951f0 con 0x55a3c72875d0 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4de7865740 10 client.0 _ll_register_callbacks cb 0x55a3c6efb8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4ddf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4ddf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59654/0 (socket says 172.21.15.28:59654) 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c7291d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/3170077611 learned_addr learned my addr 172.21.15.28:0/3170077611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.445+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c728efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/3170077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 msgr2=0x55a3c7291d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c7291d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 msgr2=0x55a3c728efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c728efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/3170077611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c6f5e3c0 con 0x55a3c7292370 2024-09-15T22:05:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728f570 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800bcc0 tx=0x7f4dd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17488805d892bd45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4dc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/3170077611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8023030 con 0x55a3c7292370 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/3170077611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f4dd8011da0 con 0x55a3c7292370 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dd0015030 con 0x55a3c7292370 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/3170077611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4dd802f050 con 0x55a3c7292370 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4de7865740 10 client.17043 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a3c72a0a90 con 0x55a3c7292370 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.446+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/3170077611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd0015030 con 0x55a3c7292370 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/3170077611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4dd801c050 con 0x55a3c7292370 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4de7865740 10 client.17043 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4de7865740 10 client.17043 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4de7865740 20 client.17043 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4de7865740 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c72a1ea0 0x55a3c72a4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4de7865740 4 client.17043 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3c71fa860 con 0x55a3c72a1ea0 2024-09-15T22:05:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c72a1ea0 0x55a3c72a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c72a1ea0 0x55a3c72a4340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6ba1a594d476d2b8 server_cookie=f6fdecc6d5941603 in_seq=0 out_seq=0 2024-09-15T22:05:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.447+0000 7f4dc2ff5640 10 client.17043 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/3170077611 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a3c71fa860 con 0x55a3c72a1ea0 2024-09-15T22:05:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4dc2ff5640 10 client.17043 handle_command_reply: tid=0 2024-09-15T22:05:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 client.17043 shutdown 2024-09-15T22:05:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 msgr2=0x55a3c728f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728f570 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800bcc0 tx=0x7f4dd8004730 comp rx=0 tx=0).stop 2024-09-15T22:05:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 shutdown_connections 2024-09-15T22:05:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c72a1ea0 0x55a3c72a4340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728f570 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c728efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 --2- 172.21.15.28:0/3170077611 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c7291d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4dc2ff5640 0 client.17043 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.448+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 >> 172.21.15.28:0/3170077611 conn(0x55a3c7250310 msgr2=0x55a3c7289960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.449+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 shutdown_connections 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.449+0000 7f4de7865740 1 -- 172.21.15.28:0/3170077611 wait complete. 2024-09-15T22:05:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.449+0000 7f4de7865740 20 client.17043 trim_cache size 0 max 16384 2024-09-15T22:05:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 Processor -- start 2024-09-15T22:05:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 -- start start 2024-09-15T22:05:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c729e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c7290f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7292370 0x55a3c7293760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4de01af110 con 0x55a3c72875d0 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4de01b0c80 con 0x55a3c7288b40 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4de7865740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4de01af7e0 con 0x55a3c7292370 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c729e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c729e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59668/0 (socket says 172.21.15.28:59668) 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 -- 172.21.15.28:0/282835556 learned_addr learned my addr 172.21.15.28:0/282835556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c7290f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7292370 0x55a3c7293760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 -- 172.21.15.28:0/282835556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7292370 msgr2=0x55a3c7293760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7292370 0x55a3c7293760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 -- 172.21.15.28:0/282835556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 msgr2=0x55a3c7290f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c7290f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 -- 172.21.15.28:0/282835556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3c7292090 con 0x55a3c72875d0 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c7290f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7292370 0x55a3c7293760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.462+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c729e310 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00097a0 tx=0x7f4dd0010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886a811beedcfd91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4dc4ff9640 1 -- 172.21.15.28:0/282835556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0004bc0 con 0x55a3c72875d0 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4dc4ff9640 1 -- 172.21.15.28:0/282835556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd0015040 con 0x55a3c72875d0 2024-09-15T22:05:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4dc4ff9640 1 -- 172.21.15.28:0/282835556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0005250 con 0x55a3c72875d0 2024-09-15T22:05:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 -- 172.21.15.28:0/282835556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 msgr2=0x55a3c729e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c729e310 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00097a0 tx=0x7f4dd0010680 comp rx=0 tx=0).stop 2024-09-15T22:05:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 -- 172.21.15.28:0/282835556 shutdown_connections 2024-09-15T22:05:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c7292370 0x55a3c7293760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c7290f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 --2- 172.21.15.28:0/282835556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c72875d0 0x55a3c729e310 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 -- 172.21.15.28:0/282835556 >> 172.21.15.28:0/282835556 conn(0x55a3c7250310 msgr2=0x55a3c7290830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 -- 172.21.15.28:0/282835556 shutdown_connections 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.463+0000 7f4de7865740 1 -- 172.21.15.28:0/282835556 wait complete. 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 -- start start 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 Processor -- start 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c729a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c729a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4de01b05b0 con 0x55a3c7292370 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4de01afee0 con 0x55a3c7288b40 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4de01ad110 con 0x55a3c72875d0 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4de7865740 10 client.0 _ll_register_callbacks cb 0x7f4dd800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4ddf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4ddf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59682/0 (socket says 172.21.15.28:59682) 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/4001917692 learned_addr learned my addr 172.21.15.28:0/4001917692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c729a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.464+0000 7f4dde7fc640 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c729a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/4001917692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 msgr2=0x55a3c729a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c729a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 msgr2=0x55a3c729a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c729a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4ddf7fe640 1 -- 172.21.15.28:0/4001917692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3c728b780 con 0x55a3c7292370 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4ddf7fe640 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728b240 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8011fb0 tx=0x7f4dd8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35e15e53e6a2e24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4dc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/4001917692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8038060 con 0x55a3c7292370 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/4001917692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f4dd8004430 con 0x55a3c7292370 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dd0015040 con 0x55a3c7292370 2024-09-15T22:05:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/4001917692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4dd8023030 con 0x55a3c7292370 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4de7865740 10 client.17049 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a3c728c9f0 con 0x55a3c7292370 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/4001917692 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd0015040 con 0x55a3c7292370 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/4001917692 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4dd801c070 con 0x55a3c7292370 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4de7865740 10 client.17049 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4de7865740 10 client.17049 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.465+0000 7f4de7865740 20 client.17049 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4de7865740 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c730dbf0 0x55a3c730ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4de7865740 4 client.17049 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3c71f4d70 con 0x55a3c730dbf0 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c730dbf0 0x55a3c730ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4ddeffd640 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c730dbf0 0x55a3c730ffe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1bc54e02fdf47401 server_cookie=e9277d7c9aa4bcc8 in_seq=0 out_seq=0 2024-09-15T22:05:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4dc2ff5640 10 client.17049 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4dc2ff5640 1 -- 172.21.15.28:0/4001917692 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3c71f4d70 con 0x55a3c730dbf0 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4dc2ff5640 10 client.17049 handle_command_reply: tid=0 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4de7865740 1 client.17049 shutdown 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.466+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 msgr2=0x55a3c728b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728b240 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8011fb0 tx=0x7f4dd8011a30 comp rx=0 tx=0).stop 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 shutdown_connections 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55a3c730dbf0 0x55a3c730ffe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3c7292370 0x55a3c728b240 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a3c7288b40 0x55a3c729a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 --2- 172.21.15.28:0/4001917692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3c72875d0 0x55a3c729a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 >> 172.21.15.28:0/4001917692 conn(0x55a3c7250310 msgr2=0x55a3c7291050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4dc2ff5640 0 client.17049 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 shutdown_connections 2024-09-15T22:05:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 1 -- 172.21.15.28:0/4001917692 wait complete. 2024-09-15T22:05:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de7865740 20 client.17049 trim_cache size 0 max 16384 2024-09-15T22:05:54.476 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4dc80a33b0 msgr2=0x7f4dc80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4dc80a33b0 0x7f4dc80a5870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005d50 tx=0x7f4dd8005e90 comp rx=0 tx=0).stop 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 msgr2=0x7f4de0198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0198b30 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4005940 tx=0x7f4dd40042f0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 shutdown_connections 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f4dc80a33b0 0x7f4dc80a5870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4de0108300 0x7f4de019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de0103780 0x7f4de0198b30 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 --2- 172.21.15.28:0/822517518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4de0102dd0 0x7f4de01985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.467+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 >> 172.21.15.28:0/822517518 conn(0x7f4de00fc7d0 msgr2=0x7f4de0105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.468+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 shutdown_connections 2024-09-15T22:05:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:54.468+0000 7f4de548c640 1 -- 172.21.15.28:0/822517518 wait complete. 2024-09-15T22:05:55.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:05:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:56.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:55 smithi028 ceph-mon[30969]: pgmap v262: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:56.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:55 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 -- 172.21.15.28:0/2686903044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102ef0 msgr2=0x7f479010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 --2- 172.21.15.28:0/2686903044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102ef0 0x7f479010da40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4788009ca0 tx=0x7f478802a950 comp rx=0 tx=0).stop 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 -- 172.21.15.28:0/2686903044 shutdown_connections 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 --2- 172.21.15.28:0/2686903044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102ef0 0x7f479010da40 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 --2- 172.21.15.28:0/2686903044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4790102570 0x7f47901029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 --2- 172.21.15.28:0/2686903044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4790100e20 0x7f4790101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 -- 172.21.15.28:0/2686903044 >> 172.21.15.28:0/2686903044 conn(0x7f47900fc7d0 msgr2=0x7f47900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 -- 172.21.15.28:0/2686903044 shutdown_connections 2024-09-15T22:05:56.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 -- 172.21.15.28:0/2686903044 wait complete. 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.100+0000 7f4797b9b640 1 Processor -- start 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4797b9b640 1 -- start start 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4797b9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4790100e20 0x7f4790198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4797b9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102570 0x7f47901989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4797b9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4790102ef0 0x7f479019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4797b9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4790110f80 con 0x7f4790102570 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4797b9b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4790110e00 con 0x7f4790100e20 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4797b9b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4790111100 con 0x7f4790102ef0 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102570 0x7f47901989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795937640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4790100e20 0x7f4790198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102570 0x7f47901989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59706/0 (socket says 172.21.15.28:59706) 2024-09-15T22:05:56.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 -- 172.21.15.28:0/3677313040 learned_addr learned my addr 172.21.15.28:0/3677313040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4796138640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4790102ef0 0x7f479019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 -- 172.21.15.28:0/3677313040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4790102ef0 msgr2=0x7f479019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4790102ef0 0x7f479019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 -- 172.21.15.28:0/3677313040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4790100e20 msgr2=0x7f4790198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4790100e20 0x7f4790198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795136640 1 -- 172.21.15.28:0/3677313040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479019d440 con 0x7f4790102570 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.101+0000 7f4795937640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4790100e20 0x7f4790198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.102+0000 7f4795136640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102570 0x7f47901989d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f478000e9f0 tx=0x7f478000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.102+0000 7f4786ffd640 1 -- 172.21.15.28:0/3677313040 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478000cd80 con 0x7f4790102570 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.102+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479019d730 con 0x7f4790102570 2024-09-15T22:05:56.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.102+0000 7f4786ffd640 1 -- 172.21.15.28:0/3677313040 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4780010910 con 0x7f4790102570 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.102+0000 7f4786ffd640 1 -- 172.21.15.28:0/3677313040 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47800188f0 con 0x7f4790102570 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.102+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47901a5010 con 0x7f4790102570 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.104+0000 7f4799f74740 1 Processor -- start 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4799f74740 1 -- start start 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4799f74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f504331b40 0x55f504330090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4799f74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f504330a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4799f74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f50433d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4799f74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f5042962f0 con 0x55f504331b40 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4799f74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f50423dcf0 con 0x55f50433b370 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4799f74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f5042922b0 con 0x55f5043305d0 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4786ffd640 1 -- 172.21.15.28:0/3677313040 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f4780010ac0 con 0x7f4790102570 2024-09-15T22:05:56.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4796138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f504331b40 0x55f504330090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4796138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f504331b40 0x55f504330090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59716/0 (socket says 172.21.15.28:59716) 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4796138640 1 -- 172.21.15.28:0/2795219898 learned_addr learned my addr 172.21.15.28:0/2795219898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4795937640 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f504330a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.105+0000 7f4795136640 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f50433d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4795937640 1 -- 172.21.15.28:0/2795219898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 msgr2=0x55f50433d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4795937640 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f50433d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4795937640 1 -- 172.21.15.28:0/2795219898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f504331b40 msgr2=0x55f504330090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4795937640 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f504331b40 0x55f504330090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4795937640 1 -- 172.21.15.28:0/2795219898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5042223a0 con 0x55f5043305d0 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4796138640 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f504331b40 0x55f504330090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4795937640 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f504330a10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f478c009fd0 tx=0x7f478c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2af658437a60bd61 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f477bfff640 1 -- 172.21.15.28:0/2795219898 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c004310 con 0x55f5043305d0 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f477bfff640 1 -- 172.21.15.28:0/2795219898 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c0044b0 con 0x55f5043305d0 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f477bfff640 1 -- 172.21.15.28:0/2795219898 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c0035f0 con 0x55f5043305d0 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4786ffd640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f477c0a33b0 0x7f477c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4786ffd640 1 -- 172.21.15.28:0/3677313040 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f47800d04c0 con 0x7f4790102570 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.106+0000 7f4796138640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f477c0a33b0 0x7f477c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 -- 172.21.15.28:0/2795219898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 msgr2=0x55f504330a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4796138640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f477c0a33b0 0x7f477c0a5870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4788009ca0 tx=0x7f4788005c50 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f504330a10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f478c009fd0 tx=0x7f478c0118d0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 -- 172.21.15.28:0/2795219898 shutdown_connections 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f50433d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f504330a10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 --2- 172.21.15.28:0/2795219898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f504331b40 0x55f504330090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 -- 172.21.15.28:0/2795219898 >> 172.21.15.28:0/2795219898 conn(0x55f5042f9310 msgr2=0x55f50432cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 -- 172.21.15.28:0/2795219898 shutdown_connections 2024-09-15T22:05:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.107+0000 7f4799f74740 1 -- 172.21.15.28:0/2795219898 wait complete. 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 -- start start 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 Processor -- start 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f5043305d0 0x55f50433ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504338010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f504338600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f50433e070 con 0x55f50433b370 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f504251060 con 0x55f5043305d0 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f50433e1f0 con 0x55f504331b40 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4799f74740 10 client.0 _ll_register_callbacks cb 0x55f503fa48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4795937640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f5043305d0 0x55f50433ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4795136640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504338010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4796138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f504338600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4795937640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f5043305d0 0x55f50433ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:50788/0 (socket says 172.21.15.28:50788) 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4795937640 1 -- 172.21.15.28:0/692776879 learned_addr learned my addr 172.21.15.28:0/692776879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4795136640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504338010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:38470/0 (socket says 172.21.15.28:38470) 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.108+0000 7f4796138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f504338600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59728/0 (socket says 172.21.15.28:59728) 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f4795136640 1 -- 172.21.15.28:0/692776879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f5043305d0 msgr2=0x55f50433ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f4795136640 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f5043305d0 0x55f50433ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f4795136640 1 -- 172.21.15.28:0/692776879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 msgr2=0x55f504338600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f4795136640 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f504338600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f4795136640 1 -- 172.21.15.28:0/692776879 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5040073c0 con 0x55f504331b40 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f4795136640 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504338010 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f47800cd800 tx=0x7f478009c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6baa8164df81cce1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f47797fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f47797fa640 1 -- 172.21.15.28:0/692776879 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47800a0740 con 0x55f504331b40 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f478c0044b0 con 0x55f504331b40 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.109+0000 7f47797fa640 1 -- 172.21.15.28:0/692776879 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f47800cda80 con 0x55f504331b40 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f47797fa640 1 -- 172.21.15.28:0/692776879 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f47800cdc80 con 0x55f504331b40 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f4799f74740 10 client.26464 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f504349b60 con 0x55f504331b40 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f47797fa640 1 -- 172.21.15.28:0/692776879 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47800a5000 con 0x55f504331b40 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f47797fa640 1 -- 172.21.15.28:0/692776879 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4780098390 con 0x55f504331b40 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f4799f74740 10 client.26464 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f4799f74740 10 client.26464 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f4799f74740 20 client.26464 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.110+0000 7f4799f74740 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f50434b3f0 0x55f50434d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.111+0000 7f4799f74740 4 client.26464 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.111+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f478c0044b0 con 0x55f50434b3f0 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.111+0000 7f4795937640 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f50434b3f0 0x55f50434d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.111+0000 7f47797fa640 10 client.26464 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.111+0000 7f4795937640 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f50434b3f0 0x55f50434d810 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70a27acab3ced9d0 server_cookie=cb0158475e937be4 in_seq=0 out_seq=0 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f47797fa640 1 -- 172.21.15.28:0/692776879 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f478c0044b0 con 0x55f50434b3f0 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f47797fa640 10 client.26464 handle_command_reply: tid=0 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 client.26464 shutdown 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 msgr2=0x55f504338010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504338010 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f47800cd800 tx=0x7f478009c300 comp rx=0 tx=0).stop 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 shutdown_connections 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f50434b3f0 0x55f50434d810 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f504338600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504338010 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 --2- 172.21.15.28:0/692776879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f5043305d0 0x55f50433ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f47797fa640 0 client.26464 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 >> 172.21.15.28:0/692776879 conn(0x55f5042f9310 msgr2=0x55f504332960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 shutdown_connections 2024-09-15T22:05:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 1 -- 172.21.15.28:0/692776879 wait complete. 2024-09-15T22:05:56.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.112+0000 7f4799f74740 20 client.26464 trim_cache size 0 max 16384 2024-09-15T22:05:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.125+0000 7f4799f74740 1 Processor -- start 2024-09-15T22:05:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4799f74740 1 -- start start 2024-09-15T22:05:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4799f74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5043305d0 0x55f50432bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4799f74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504333120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4799f74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f504333660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4799f74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47901b1310 con 0x55f5043305d0 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4799f74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f47901b2e80 con 0x55f50433b370 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4799f74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f47901b19e0 con 0x55f504331b40 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5043305d0 0x55f50432bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795136640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504333120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5043305d0 0x55f50432bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59736/0 (socket says 172.21.15.28:59736) 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 -- 172.21.15.28:0/140910694 learned_addr learned my addr 172.21.15.28:0/140910694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4796138640 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f504333660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 -- 172.21.15.28:0/140910694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 msgr2=0x55f504333120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504333120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 -- 172.21.15.28:0/140910694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 msgr2=0x55f504333660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f504333660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795937640 1 -- 172.21.15.28:0/140910694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f504336560 con 0x55f5043305d0 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.126+0000 7f4795136640 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504333120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.127+0000 7f4795937640 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5043305d0 0x55f50432bbf0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f478c014040 tx=0x7f478c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1527411d54ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.127+0000 7f477b7fe640 1 -- 172.21.15.28:0/140910694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c004740 con 0x55f5043305d0 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.127+0000 7f477b7fe640 1 -- 172.21.15.28:0/140910694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c0048e0 con 0x55f5043305d0 2024-09-15T22:05:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.127+0000 7f477b7fe640 1 -- 172.21.15.28:0/140910694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c02e6f0 con 0x55f5043305d0 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 -- 172.21.15.28:0/140910694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5043305d0 msgr2=0x55f50432bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5043305d0 0x55f50432bbf0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f478c014040 tx=0x7f478c005d50 comp rx=0 tx=0).stop 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 -- 172.21.15.28:0/140910694 shutdown_connections 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f50433b370 0x55f504333660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f504331b40 0x55f504333120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 --2- 172.21.15.28:0/140910694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5043305d0 0x55f50432bbf0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 -- 172.21.15.28:0/140910694 >> 172.21.15.28:0/140910694 conn(0x55f5042f9310 msgr2=0x55f50432b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 -- 172.21.15.28:0/140910694 shutdown_connections 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 -- 172.21.15.28:0/140910694 wait complete. 2024-09-15T22:05:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 -- start start 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.128+0000 7f4799f74740 1 Processor -- start 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4799f74740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f5043510e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4799f74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f504331b40 0x55f504351620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4799f74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f5043b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4799f74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47901b27b0 con 0x55f50433b370 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4799f74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f47901b20e0 con 0x55f504331b40 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4799f74740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f47901af310 con 0x55f5043305d0 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4799f74740 10 client.0 _ll_register_callbacks cb 0x7f478000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4796138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f5043b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4796138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f5043b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59748/0 (socket says 172.21.15.28:59748) 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4796138640 1 -- 172.21.15.28:0/3784126517 learned_addr learned my addr 172.21.15.28:0/3784126517 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795937640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f5043510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795136640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f504331b40 0x55f504351620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795937640 1 -- 172.21.15.28:0/3784126517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f504331b40 msgr2=0x55f504351620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795937640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f504331b40 0x55f504351620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795937640 1 -- 172.21.15.28:0/3784126517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 msgr2=0x55f5043b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795937640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f5043b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795937640 1 -- 172.21.15.28:0/3784126517 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f504351f00 con 0x55f5043305d0 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4796138640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f5043b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f4795937640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f5043510e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f478c0058c0 tx=0x7f478c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4dea65f3781bcd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.129+0000 7f47797fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f47797fa640 1 -- 172.21.15.28:0/3784126517 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c035070 con 0x55f5043305d0 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f47797fa640 1 -- 172.21.15.28:0/3784126517 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f478c003a00 con 0x55f5043305d0 2024-09-15T22:05:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f478c0048e0 con 0x55f5043305d0 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f47797fa640 1 -- 172.21.15.28:0/3784126517 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f478c003c00 con 0x55f5043305d0 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 10 client.26470 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f5043b6630 con 0x55f5043305d0 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f47797fa640 1 -- 172.21.15.28:0/3784126517 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c0048e0 con 0x55f5043305d0 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f47797fa640 1 -- 172.21.15.28:0/3784126517 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f478c00f600 con 0x55f5043305d0 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 10 client.26470 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 10 client.26470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 20 client.26470 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f5043b6af0 0x55f5043b8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 4 client.26470 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f5042a3860 con 0x55f5043b6af0 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.130+0000 7f4795136640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f5043b6af0 0x55f5043b8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4795136640 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f5043b6af0 0x55f5043b8ee0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ec37af66c701d26 server_cookie=a38b0f1c6ef06709 in_seq=0 out_seq=0 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f47797fa640 10 client.26470 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f47797fa640 1 -- 172.21.15.28:0/3784126517 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f5042a3860 con 0x55f5043b6af0 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f47797fa640 10 client.26470 handle_command_reply: tid=0 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 client.26470 shutdown 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 msgr2=0x55f5043510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f5043510e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f478c0058c0 tx=0x7f478c0047c0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 shutdown_connections 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55f5043b6af0 0x55f5043b8ee0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50433b370 0x55f5043b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f504331b40 0x55f504351620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f47797fa640 0 client.26470 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 --2- 172.21.15.28:0/3784126517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f5043305d0 0x55f5043510e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 >> 172.21.15.28:0/3784126517 conn(0x55f5042f9310 msgr2=0x55f50432bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 shutdown_connections 2024-09-15T22:05:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.131+0000 7f4799f74740 1 -- 172.21.15.28:0/3784126517 wait complete. 2024-09-15T22:05:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4799f74740 20 client.26470 trim_cache size 0 max 16384 2024-09-15T22:05:56.140 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f477c0a33b0 msgr2=0x7f477c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f477c0a33b0 0x7f477c0a5870 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4788009ca0 tx=0x7f4788005c50 comp rx=0 tx=0).stop 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102570 msgr2=0x7f47901989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102570 0x7f47901989d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f478000e9f0 tx=0x7f478000eeb0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 shutdown_connections 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f477c0a33b0 0x7f477c0a5870 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4790102ef0 0x7f479019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4790102570 0x7f47901989d0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 --2- 172.21.15.28:0/3677313040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4790100e20 0x7f4790198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.132+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 >> 172.21.15.28:0/3677313040 conn(0x7f47900fc7d0 msgr2=0x7f479006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.133+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 shutdown_connections 2024-09-15T22:05:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:56.133+0000 7f4797b9b640 1 -- 172.21.15.28:0/3677313040 wait complete. 2024-09-15T22:05:56.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:55 smithi203 ceph-mon[35546]: pgmap v262: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:56.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:55 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:05:56.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:55 smithi174 ceph-mon[35244]: pgmap v262: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:56.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:55 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:05:56.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:05:57.134 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:05:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:05:57.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:05:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:05:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:05:57.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:05:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:05:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:05:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 -- 172.21.15.28:0/592387659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d41002f0 msgr2=0x7ff1d41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 --2- 172.21.15.28:0/592387659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d41002f0 0x7ff1d41006d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009930 tx=0x7ff1c402a420 comp rx=0 tx=0).stop 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 -- 172.21.15.28:0/592387659 shutdown_connections 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 --2- 172.21.15.28:0/592387659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d4106f50 0x7ff1d410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 --2- 172.21.15.28:0/592387659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d4100ca0 0x7ff1d4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 --2- 172.21.15.28:0/592387659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d41002f0 0x7ff1d41006d0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 -- 172.21.15.28:0/592387659 >> 172.21.15.28:0/592387659 conn(0x7ff1d4077210 msgr2=0x7ff1d40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 -- 172.21.15.28:0/592387659 shutdown_connections 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.636+0000 7ff1d9415640 1 -- 172.21.15.28:0/592387659 wait complete. 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 Processor -- start 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 -- start start 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d41002f0 0x7ff1d41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d4100ca0 0x7ff1d4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d4106f50 0x7ff1d419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4111fa0 con 0x7ff1d4106f50 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4111e20 con 0x7ff1d41002f0 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d9415640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4112120 con 0x7ff1d4100ca0 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d4106f50 0x7ff1d419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d4106f50 0x7ff1d419cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59768/0 (socket says 172.21.15.28:59768) 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d37fe640 1 -- 172.21.15.28:0/3026716166 learned_addr learned my addr 172.21.15.28:0/3026716166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d41002f0 0x7ff1d41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.637+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d4100ca0 0x7ff1d4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d37fe640 1 -- 172.21.15.28:0/3026716166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d4100ca0 msgr2=0x7ff1d4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d37fe640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d4100ca0 0x7ff1d4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d37fe640 1 -- 172.21.15.28:0/3026716166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d41002f0 msgr2=0x7ff1d41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d37fe640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d41002f0 0x7ff1d41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d37fe640 1 -- 172.21.15.28:0/3026716166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d419d480 con 0x7ff1d4106f50 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d37fe640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d4106f50 0x7ff1d419cda0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00eaa0 tx=0x7ff1cc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1bbfff640 1 -- 172.21.15.28:0/3026716166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc00cdb0 con 0x7ff1d4106f50 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d419d770 con 0x7ff1d4106f50 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1bbfff640 1 -- 172.21.15.28:0/3026716166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1cc005020 con 0x7ff1d4106f50 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1bbfff640 1 -- 172.21.15.28:0/3026716166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0116b0 con 0x7ff1d4106f50 2024-09-15T22:05:57.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.638+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d41a5050 con 0x7ff1d4106f50 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.641+0000 7ff1bbfff640 1 -- 172.21.15.28:0/3026716166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7ff1cc005560 con 0x7ff1d4106f50 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.641+0000 7ff1db7ee740 1 Processor -- start 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.641+0000 7ff1db7ee740 1 -- start start 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.641+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f539090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.641+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f539a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f546760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a69f49f2f0 con 0x55a69f5395d0 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a69f446cf0 con 0x55a69f544370 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a69f49b2b0 con 0x55a69f53ab40 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f539a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f539a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59782/0 (socket says 172.21.15.28:59782) 2024-09-15T22:05:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2921278134 learned_addr learned my addr 172.21.15.28:0/2921278134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f539090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f546760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2921278134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 msgr2=0x55a69f539090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f539090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2921278134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 msgr2=0x55a69f546760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f546760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2921278134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69f42b3a0 con 0x55a69f5395d0 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f546760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f539a10 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff1c402a940 tx=0x7ff1c4005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f895b2c475d788c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1bbfff640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1bc0a33b0 0x7ff1bc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1b97fa640 1 -- 172.21.15.28:0/2921278134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4040070 con 0x55a69f5395d0 2024-09-15T22:05:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1b97fa640 1 -- 172.21.15.28:0/2921278134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c40319f0 con 0x55a69f5395d0 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1bc0a33b0 0x7ff1bc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.642+0000 7ff1b97fa640 1 -- 172.21.15.28:0/2921278134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4035040 con 0x55a69f5395d0 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1bbfff640 1 -- 172.21.15.28:0/3026716166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7ff1cc0d0210 con 0x7ff1d4106f50 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1bc0a33b0 0x7ff1bc0a5870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55a69f53a070 tx=0x7ff1c800b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2921278134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 msgr2=0x55a69f539a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f539a10 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff1c402a940 tx=0x7ff1c4005c00 comp rx=0 tx=0).stop 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2921278134 shutdown_connections 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f546760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f539a10 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2921278134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f539090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2921278134 >> 172.21.15.28:0/2921278134 conn(0x55a69f502310 msgr2=0x55a69f535c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2921278134 shutdown_connections 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.643+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2921278134 wait complete. 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 -- start start 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 Processor -- start 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f5395d0 0x55a69f543c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f53ab40 0x55a69f540ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f541420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a69f547010 con 0x55a69f53ab40 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a69f45a060 con 0x55a69f544370 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a69f547190 con 0x55a69f5395d0 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1db7ee740 10 client.0 _ll_register_callbacks cb 0x55a69f1ad8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f53ab40 0x55a69f540ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f53ab40 0x55a69f540ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59792/0 (socket says 172.21.15.28:59792) 2024-09-15T22:05:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 -- 172.21.15.28:0/4224429471 learned_addr learned my addr 172.21.15.28:0/4224429471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f5395d0 0x55a69f543c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d37fe640 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f541420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 -- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f5395d0 msgr2=0x55a69f543c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f5395d0 0x55a69f543c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 -- 172.21.15.28:0/4224429471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 msgr2=0x55a69f541420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f541420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.644+0000 7ff1d27fc640 1 -- 172.21.15.28:0/4224429471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a69f2103c0 con 0x55a69f53ab40 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f53ab40 0x55a69f540ee0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8005880 tx=0x7ff1c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e55d68b319bd4c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/4224429471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8027470 con 0x55a69f53ab40 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/4224429471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7ff1c8027610 con 0x55a69f53ab40 2024-09-15T22:05:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1c40319f0 con 0x55a69f53ab40 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/4224429471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff1c8027810 con 0x55a69f53ab40 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 10 client.17079 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a69f552970 con 0x55a69f53ab40 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/4224429471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c40319f0 con 0x55a69f53ab40 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/4224429471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7ff1c801e670 con 0x55a69f53ab40 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 10 client.17079 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 10 client.17079 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 20 client.17079 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f553e30 0x55a69f556280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 4 client.17079 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.645+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a69f4ac860 con 0x55a69f553e30 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.646+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f553e30 0x55a69f556280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.646+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f553e30 0x55a69f556280 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=257623a7c865aea4 server_cookie=a3e088b0c4dc89ce in_seq=0 out_seq=0 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.646+0000 7ff1b6ff5640 10 client.17079 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/4224429471 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a69f4ac860 con 0x55a69f553e30 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1b6ff5640 10 client.17079 handle_command_reply: tid=0 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 client.17079 shutdown 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f53ab40 msgr2=0x55a69f540ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f53ab40 0x55a69f540ee0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8005880 tx=0x7ff1c80046c0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 shutdown_connections 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f553e30 0x55a69f556280 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f541420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f53ab40 0x55a69f540ee0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1b6ff5640 0 client.17079 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/4224429471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f5395d0 0x55a69f543c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 >> 172.21.15.28:0/4224429471 conn(0x55a69f502310 msgr2=0x55a69f53b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 shutdown_connections 2024-09-15T22:05:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 1 -- 172.21.15.28:0/4224429471 wait complete. 2024-09-15T22:05:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.647+0000 7ff1db7ee740 20 client.17079 trim_cache size 0 max 16384 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.661+0000 7ff1db7ee740 1 Processor -- start 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.661+0000 7ff1db7ee740 1 -- start start 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f542e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f53ab40 0x55a69f53c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f544370 0x55a69f53c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41af130 con 0x55a69f5395d0 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41b0ca0 con 0x55a69f53ab40 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41af800 con 0x55a69f544370 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f542e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f542e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59794/0 (socket says 172.21.15.28:59794) 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/554765483 learned_addr learned my addr 172.21.15.28:0/554765483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f53ab40 0x55a69f53c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d37fe640 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f544370 0x55a69f53c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/554765483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f544370 msgr2=0x55a69f53c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f544370 0x55a69f53c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/554765483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f53ab40 msgr2=0x55a69f53c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f53ab40 0x55a69f53c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/554765483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69f5354f0 con 0x55a69f5395d0 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d37fe640 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f544370 0x55a69f53c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f53ab40 0x55a69f53c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f542e50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x55a69f536ac0 tx=0x7ff1c4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad031fce9c5a689 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:57.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.662+0000 7ff1b8ff9640 1 -- 172.21.15.28:0/554765483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4040050 con 0x55a69f5395d0 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1b8ff9640 1 -- 172.21.15.28:0/554765483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c4005520 con 0x55a69f5395d0 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1b8ff9640 1 -- 172.21.15.28:0/554765483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4035040 con 0x55a69f5395d0 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 -- 172.21.15.28:0/554765483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 msgr2=0x55a69f542e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f542e50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x55a69f536ac0 tx=0x7ff1c4000bc0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 -- 172.21.15.28:0/554765483 shutdown_connections 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f544370 0x55a69f53c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f53ab40 0x55a69f53c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/554765483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f542e50 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 -- 172.21.15.28:0/554765483 >> 172.21.15.28:0/554765483 conn(0x55a69f502310 msgr2=0x55a69f542740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 -- 172.21.15.28:0/554765483 shutdown_connections 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.663+0000 7ff1db7ee740 1 -- 172.21.15.28:0/554765483 wait complete. 2024-09-15T22:05:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 -- start start 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 Processor -- start 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f54ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f53dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f53e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41b05d0 con 0x55a69f5395d0 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41aff00 con 0x55a69f544370 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41ad130 con 0x55a69f53ab40 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1db7ee740 10 client.0 _ll_register_callbacks cb 0x7ff1c8003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f54ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f54ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59796/0 (socket says 172.21.15.28:59796) 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2453907656 learned_addr learned my addr 172.21.15.28:0/2453907656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f53dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 msgr2=0x55a69f53dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f53dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d37fe640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f53e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2453907656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 msgr2=0x55a69f53e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f53e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d2ffd640 1 -- 172.21.15.28:0/2453907656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a69f5bdb20 con 0x55a69f5395d0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.664+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f53dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1d2ffd640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f54ccc0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009430 tx=0x7ff1c402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa89b6f1c476a24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/2453907656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4035070 con 0x55a69f5395d0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/2453907656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7ff1c4004040 con 0x55a69f5395d0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1c4005520 con 0x55a69f5395d0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/2453907656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff1c4040020 con 0x55a69f5395d0 2024-09-15T22:05:57.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1db7ee740 10 client.17085 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a69f5bebe0 con 0x55a69f5395d0 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/2453907656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c4005520 con 0x55a69f5395d0 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/2453907656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7ff1c40037b0 con 0x55a69f5395d0 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1db7ee740 10 client.17085 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1db7ee740 10 client.17085 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.665+0000 7ff1db7ee740 20 client.17085 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f5beeb0 0x55a69f5c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1db7ee740 4 client.17085 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a69f4a6d70 con 0x55a69f5beeb0 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f5beeb0 0x55a69f5c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1b6ff5640 10 client.17085 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1d27fc640 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f5beeb0 0x55a69f5c12a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ac788a6d89d5f95 server_cookie=2a0835d659ed01eb in_seq=0 out_seq=0 2024-09-15T22:05:57.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1b6ff5640 1 -- 172.21.15.28:0/2453907656 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a69f4a6d70 con 0x55a69f5beeb0 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.666+0000 7ff1b6ff5640 10 client.17085 handle_command_reply: tid=0 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 client.17085 shutdown 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 msgr2=0x55a69f54ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f54ccc0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009430 tx=0x7ff1c402a420 comp rx=0 tx=0).stop 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 shutdown_connections 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55a69f5beeb0 0x55a69f5c12a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a69f544370 0x55a69f53e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a69f53ab40 0x55a69f53dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 --2- 172.21.15.28:0/2453907656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a69f5395d0 0x55a69f54ccc0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 >> 172.21.15.28:0/2453907656 conn(0x55a69f502310 msgr2=0x55a69f542f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1b6ff5640 0 client.17085 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 shutdown_connections 2024-09-15T22:05:57.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 1 -- 172.21.15.28:0/2453907656 wait complete. 2024-09-15T22:05:57.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1db7ee740 20 client.17085 trim_cache size 0 max 16384 2024-09-15T22:05:57.676 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:57.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1bc0a33b0 msgr2=0x7ff1bc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1d9415640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1bc0a33b0 0x7ff1bc0a5870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55a69f53a070 tx=0x7ff1c800b390 comp rx=0 tx=0).stop 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d4106f50 msgr2=0x7ff1d419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1d9415640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d4106f50 0x7ff1d419cda0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00eaa0 tx=0x7ff1cc00ef60 comp rx=0 tx=0).stop 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 shutdown_connections 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1d9415640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff1bc0a33b0 0x7ff1bc0a5870 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.667+0000 7ff1d9415640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1d4106f50 0x7ff1d419cda0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.668+0000 7ff1d9415640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1d4100ca0 0x7ff1d4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.668+0000 7ff1d9415640 1 --2- 172.21.15.28:0/3026716166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1d41002f0 0x7ff1d41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.668+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 >> 172.21.15.28:0/3026716166 conn(0x7ff1d4077210 msgr2=0x7ff1d41052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.668+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 shutdown_connections 2024-09-15T22:05:57.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:57.668+0000 7ff1d9415640 1 -- 172.21.15.28:0/3026716166 wait complete. 2024-09-15T22:05:58.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:57 smithi203 ceph-mon[35546]: pgmap v263: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:57 smithi174 ceph-mon[35244]: pgmap v263: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:58.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:57 smithi028 ceph-mon[30969]: pgmap v263: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:05:58.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:05:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:05:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 -- 172.21.15.28:0/3778340192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24103660 msgr2=0x7efe2410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 --2- 172.21.15.28:0/3778340192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24103660 0x7efe2410bb50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7efe14009900 tx=0x7efe1402a480 comp rx=0 tx=0).stop 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 -- 172.21.15.28:0/3778340192 shutdown_connections 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 --2- 172.21.15.28:0/3778340192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe2410c170 0x7efe2410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 --2- 172.21.15.28:0/3778340192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24103660 0x7efe2410bb50 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 --2- 172.21.15.28:0/3778340192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe24102d40 0x7efe24103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 -- 172.21.15.28:0/3778340192 >> 172.21.15.28:0/3778340192 conn(0x7efe240fc7a0 msgr2=0x7efe240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 -- 172.21.15.28:0/3778340192 shutdown_connections 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.138+0000 7efe2b61f640 1 -- 172.21.15.28:0/3778340192 wait complete. 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 Processor -- start 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 -- start start 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24102d40 0x7efe24198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe24103660 0x7efe241989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe2410c170 0x7efe2419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe24111f70 con 0x7efe24102d40 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efe24111df0 con 0x7efe2410c170 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe2b61f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efe241120f0 con 0x7efe24103660 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24102d40 0x7efe24198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24102d40 0x7efe24198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59814/0 (socket says 172.21.15.28:59814) 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 -- 172.21.15.28:0/1872149917 learned_addr learned my addr 172.21.15.28:0/1872149917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe28bba640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe24103660 0x7efe241989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe29bbc640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe2410c170 0x7efe2419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 -- 172.21.15.28:0/1872149917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe24103660 msgr2=0x7efe241989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe24103660 0x7efe241989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 -- 172.21.15.28:0/1872149917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe2410c170 msgr2=0x7efe2419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe2410c170 0x7efe2419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.139+0000 7efe293bb640 1 -- 172.21.15.28:0/1872149917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe2419d420 con 0x7efe24102d40 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.140+0000 7efe293bb640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24102d40 0x7efe24198470 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7efe2000d950 tx=0x7efe2000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.140+0000 7efe1a7fc640 1 -- 172.21.15.28:0/1872149917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20017070 con 0x7efe24102d40 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.140+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe2419d710 con 0x7efe24102d40 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.140+0000 7efe1a7fc640 1 -- 172.21.15.28:0/1872149917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe20009c30 con 0x7efe24102d40 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.140+0000 7efe1a7fc640 1 -- 172.21.15.28:0/1872149917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2000f6b0 con 0x7efe24102d40 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.140+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe241a4ff0 con 0x7efe24102d40 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.142+0000 7efe1a7fc640 1 -- 172.21.15.28:0/1872149917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7efe200109d0 con 0x7efe24102d40 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 Processor -- start 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 -- start start 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddcb40 0x55609fddb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddb5d0 0x55609fddba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fde8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55609fd412f0 con 0x55609fddb5d0 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55609fce8cf0 con 0x55609fde6370 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55609fd3d2b0 con 0x55609fddcb40 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe1a7fc640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efe100a33b0 0x7efe100a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe293bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddb5d0 0x55609fddba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe293bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddb5d0 0x55609fddba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59822/0 (socket says 172.21.15.28:59822) 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe293bb640 1 -- 172.21.15.28:0/3724837130 learned_addr learned my addr 172.21.15.28:0/3724837130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe29bbc640 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddcb40 0x55609fddb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe29bbc640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efe100a33b0 0x7efe100a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe28bba640 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fde8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe1a7fc640 1 -- 172.21.15.28:0/1872149917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7efe200d0060 con 0x7efe24102d40 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe293bb640 1 -- 172.21.15.28:0/3724837130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddcb40 msgr2=0x55609fddb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.143+0000 7efe293bb640 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddcb40 0x55609fddb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe293bb640 1 -- 172.21.15.28:0/3724837130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 msgr2=0x55609fde8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe293bb640 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fde8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe293bb640 1 -- 172.21.15.28:0/3724837130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609fccd3a0 con 0x55609fddb5d0 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe29bbc640 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddcb40 0x55609fddb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe28bba640 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fde8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe293bb640 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddb5d0 0x55609fddba10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7efe20009610 tx=0x7efe20010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62b7b4188a8463 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe29bbc640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efe100a33b0 0x7efe100a5870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe1c005c40 tx=0x7efe1c005bd0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe0f7fe640 1 -- 172.21.15.28:0/3724837130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20020070 con 0x55609fddb5d0 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe0f7fe640 1 -- 172.21.15.28:0/3724837130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe20009de0 con 0x55609fddb5d0 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.144+0000 7efe0f7fe640 1 -- 172.21.15.28:0/3724837130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe200cd680 con 0x55609fddb5d0 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3724837130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddb5d0 msgr2=0x55609fddba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddb5d0 0x55609fddba10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7efe20009610 tx=0x7efe20010d70 comp rx=0 tx=0).stop 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3724837130 shutdown_connections 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fde8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddb5d0 0x55609fddba10 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3724837130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddcb40 0x55609fddb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3724837130 >> 172.21.15.28:0/3724837130 conn(0x55609fda4310 msgr2=0x55609fdd7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3724837130 shutdown_connections 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3724837130 wait complete. 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 -- start start 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.145+0000 7efe2d9f8740 1 Processor -- start 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddb5d0 0x55609fde5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fde6370 0x55609fde3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55609fde9010 con 0x55609fddcb40 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55609fcfc060 con 0x55609fddb5d0 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55609fde9190 con 0x55609fde6370 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe2d9f8740 10 client.0 _ll_register_callbacks cb 0x55609fa4f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59832/0 (socket says 172.21.15.28:59832) 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 -- 172.21.15.28:0/914858791 learned_addr learned my addr 172.21.15.28:0/914858791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe293bb640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddb5d0 0x55609fde5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe29bbc640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fde6370 0x55609fde3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 -- 172.21.15.28:0/914858791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fde6370 msgr2=0x55609fde3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fde6370 0x55609fde3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 -- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddb5d0 msgr2=0x55609fde5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddb5d0 0x55609fde5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 -- 172.21.15.28:0/914858791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609fab23c0 con 0x55609fddcb40 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe293bb640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddb5d0 0x55609fde5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe29bbc640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fde6370 0x55609fde3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe28bba640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde3080 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7efe1402a9a0 tx=0x7efe14031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42fe2c516e6288e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.146+0000 7efe0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe0cff9640 1 -- 172.21.15.28:0/914858791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14035040 con 0x55609fddcb40 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe0cff9640 1 -- 172.21.15.28:0/914858791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7efe14004240 con 0x55609fddcb40 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe20009de0 con 0x55609fddcb40 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe0cff9640 1 -- 172.21.15.28:0/914858791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7efe14004440 con 0x55609fddcb40 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 10 client.17097 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55609fdf4b80 con 0x55609fddcb40 2024-09-15T22:05:59.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe0cff9640 1 -- 172.21.15.28:0/914858791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe20009de0 con 0x55609fddcb40 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe0cff9640 1 -- 172.21.15.28:0/914858791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7efe140035f0 con 0x55609fddcb40 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 10 client.17097 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 20 client.17097 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fdf6040 0x55609fdf8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 4 client.17097 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.147+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55609fd4e860 con 0x55609fdf6040 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.148+0000 7efe293bb640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fdf6040 0x55609fdf8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.148+0000 7efe293bb640 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fdf6040 0x55609fdf8460 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2469a984984bbec server_cookie=e05c9c433ebf2ca4 in_seq=0 out_seq=0 2024-09-15T22:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.148+0000 7efe0cff9640 10 client.17097 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe0cff9640 1 -- 172.21.15.28:0/914858791 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55609fd4e860 con 0x55609fdf6040 2024-09-15T22:05:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe0cff9640 10 client.17097 handle_command_reply: tid=0 2024-09-15T22:05:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 client.17097 shutdown 2024-09-15T22:05:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 msgr2=0x55609fde3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde3080 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7efe1402a9a0 tx=0x7efe14031860 comp rx=0 tx=0).stop 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 shutdown_connections 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fdf6040 0x55609fdf8460 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fde6370 0x55609fde3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde3080 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/914858791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddb5d0 0x55609fde5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 >> 172.21.15.28:0/914858791 conn(0x55609fda4310 msgr2=0x55609fddd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe0cff9640 0 client.17097 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 shutdown_connections 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 1 -- 172.21.15.28:0/914858791 wait complete. 2024-09-15T22:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.149+0000 7efe2d9f8740 20 client.17097 trim_cache size 0 max 16384 2024-09-15T22:05:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 Processor -- start 2024-09-15T22:05:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 -- start start 2024-09-15T22:05:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fdee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe241af070 con 0x55609fddcb40 2024-09-15T22:05:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efe241b0be0 con 0x55609fde6370 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efe241af740 con 0x55609fddb5d0 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.162+0000 7efe28bba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59844/0 (socket says 172.21.15.28:59844) 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 -- 172.21.15.28:0/3824471343 learned_addr learned my addr 172.21.15.28:0/3824471343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe29bbc640 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fdee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe293bb640 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 -- 172.21.15.28:0/3824471343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 msgr2=0x55609fde3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 -- 172.21.15.28:0/3824471343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 msgr2=0x55609fdee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fdee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 -- 172.21.15.28:0/3824471343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609fdf22b0 con 0x55609fddcb40 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe29bbc640 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fdee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:05:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe28bba640 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde5dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7efe14009900 tx=0x7efe1402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c02c66be495c11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:59.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe0effd640 1 -- 172.21.15.28:0/3824471343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14056070 con 0x55609fddcb40 2024-09-15T22:05:59.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe0effd640 1 -- 172.21.15.28:0/3824471343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe14005510 con 0x55609fddcb40 2024-09-15T22:05:59.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.163+0000 7efe0effd640 1 -- 172.21.15.28:0/3824471343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14035070 con 0x55609fddcb40 2024-09-15T22:05:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3824471343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 msgr2=0x55609fde5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde5dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7efe14009900 tx=0x7efe1402a480 comp rx=0 tx=0).stop 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3824471343 shutdown_connections 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fde6370 0x55609fdee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fddcb40 0x55609fde5dd0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/3824471343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3824471343 >> 172.21.15.28:0/3824471343 conn(0x55609fda4310 msgr2=0x55609fde2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3824471343 shutdown_connections 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 -- 172.21.15.28:0/3824471343 wait complete. 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 -- start start 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.164+0000 7efe2d9f8740 1 Processor -- start 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddcb40 0x55609fde0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe2d9f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fde6370 0x55609fdfd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe241b0510 con 0x55609fde6370 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efe241afe40 con 0x55609fddcb40 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe2d9f8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efe241ad070 con 0x55609fddb5d0 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe2d9f8740 10 client.0 _ll_register_callbacks cb 0x7efe14009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe29bbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fde6370 0x55609fdfd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe293bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe29bbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fde6370 0x55609fdfd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59850/0 (socket says 172.21.15.28:59850) 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe293bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:38584/0 (socket says 172.21.15.28:38584) 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe29bbc640 1 -- 172.21.15.28:0/961735240 learned_addr learned my addr 172.21.15.28:0/961735240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe28bba640 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddcb40 0x55609fde0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe293bb640 1 -- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddcb40 msgr2=0x55609fde0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe293bb640 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddcb40 0x55609fde0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe293bb640 1 -- 172.21.15.28:0/961735240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fde6370 msgr2=0x55609fdfd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe293bb640 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fde6370 0x55609fdfd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe293bb640 1 -- 172.21.15.28:0/961735240 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609fdfd6a0 con 0x55609fddb5d0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.165+0000 7efe29bbc640 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fde6370 0x55609fdfd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe293bb640 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde04e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55609fddd140 tx=0x7efe2009b970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c472c225028867b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe0cff9640 1 -- 172.21.15.28:0/961735240 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20020040 con 0x55609fddb5d0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe0cff9640 1 -- 172.21.15.28:0/961735240 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7efe20010460 con 0x55609fddb5d0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe14005510 con 0x55609fddb5d0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe0cff9640 1 -- 172.21.15.28:0/961735240 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7efe2009bd60 con 0x55609fddb5d0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe2d9f8740 10 client.26500 fetch_fsmap learned FSMap version 34 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55609fe645d0 con 0x55609fddb5d0 2024-09-15T22:05:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.166+0000 7efe0cff9640 1 -- 172.21.15.28:0/961735240 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe14005510 con 0x55609fddb5d0 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe0cff9640 1 -- 172.21.15.28:0/961735240 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7efe20084030 con 0x55609fddb5d0 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe2d9f8740 10 client.26500 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe2d9f8740 10 client.26500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe2d9f8740 20 client.26500 populate_metadata read hostname 'smithi028' 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fe64860 0x55609fe66c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe2d9f8740 4 client.26500 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55609fd48d70 con 0x55609fe64860 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe28bba640 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fe64860 0x55609fe66c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe28bba640 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fe64860 0x55609fe66c50 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264321f169c70edb server_cookie=f7cca9775be4f35c in_seq=0 out_seq=0 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.167+0000 7efe0cff9640 10 client.26500 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe0cff9640 1 -- 172.21.15.28:0/961735240 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55609fd48d70 con 0x55609fe64860 2024-09-15T22:05:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe0cff9640 10 client.26500 handle_command_reply: tid=0 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 client.26500 shutdown 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 msgr2=0x55609fde04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde04e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55609fddd140 tx=0x7efe2009b970 comp rx=0 tx=0).stop 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 shutdown_connections 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55609fe64860 0x55609fe66c50 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55609fde6370 0x55609fdfd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe0cff9640 0 client.26500 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55609fddcb40 0x55609fde0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 --2- 172.21.15.28:0/961735240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55609fddb5d0 0x55609fde04e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 >> 172.21.15.28:0/961735240 conn(0x55609fda4310 msgr2=0x55609fdf1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 shutdown_connections 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 1 -- 172.21.15.28:0/961735240 wait complete. 2024-09-15T22:05:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.168+0000 7efe2d9f8740 20 client.26500 trim_cache size 0 max 16384 2024-09-15T22:05:59.176 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:05:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efe100a33b0 msgr2=0x7efe100a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efe100a33b0 0x7efe100a5870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe1c005c40 tx=0x7efe1c005bd0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24102d40 msgr2=0x7efe24198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24102d40 0x7efe24198470 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7efe2000d950 tx=0x7efe2000de10 comp rx=0 tx=0).stop 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 shutdown_connections 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7efe100a33b0 0x7efe100a5870 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe2410c170 0x7efe2419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efe24103660 0x7efe241989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 --2- 172.21.15.28:0/1872149917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe24102d40 0x7efe24198470 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 >> 172.21.15.28:0/1872149917 conn(0x7efe240fc7a0 msgr2=0x7efe240fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 shutdown_connections 2024-09-15T22:05:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:05:59.169+0000 7efe2b61f640 1 -- 172.21.15.28:0/1872149917 wait complete. 2024-09-15T22:06:00.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:05:59 smithi203 ceph-mon[35546]: pgmap v264: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:06:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:05:59 smithi174 ceph-mon[35244]: pgmap v264: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:06:00.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:05:59 smithi028 ceph-mon[30969]: pgmap v264: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:06:01.048 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T22:06:01.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T22:06:01.324 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 -- 172.21.15.28:0/2516248510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14102570 msgr2=0x7f1e141029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 --2- 172.21.15.28:0/2516248510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14102570 0x7f1e141029b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009900 tx=0x7f1dfc02a480 comp rx=0 tx=0).stop 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 -- 172.21.15.28:0/2516248510 shutdown_connections 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 --2- 172.21.15.28:0/2516248510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e14102ef0 0x7f1e1410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 --2- 172.21.15.28:0/2516248510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14102570 0x7f1e141029b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 --2- 172.21.15.28:0/2516248510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e14100e20 0x7f1e14101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 -- 172.21.15.28:0/2516248510 >> 172.21.15.28:0/2516248510 conn(0x7f1e140fc7d0 msgr2=0x7f1e140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 -- 172.21.15.28:0/2516248510 shutdown_connections 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.815+0000 7f1e18deb640 1 -- 172.21.15.28:0/2516248510 wait complete. 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 Processor -- start 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 -- start start 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14100e20 0x7f1e1410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e14102570 0x7f1e1410ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e14102ef0 0x7f1e1410ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141120a0 con 0x7f1e14100e20 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14111f20 con 0x7f1e14102ef0 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e18deb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14112220 con 0x7f1e14102570 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14100e20 0x7f1e1410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14100e20 0x7f1e1410e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42148/0 (socket says 172.21.15.28:42148) 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 -- 172.21.15.28:0/3313240386 learned_addr learned my addr 172.21.15.28:0/3313240386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e14102570 0x7f1e1410ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e14102ef0 0x7f1e1410ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 -- 172.21.15.28:0/3313240386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e14102570 msgr2=0x7f1e1410ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e14102570 0x7f1e1410ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 -- 172.21.15.28:0/3313240386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e14102ef0 msgr2=0x7f1e1410ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e14102ef0 0x7f1e1410ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.816+0000 7f1e1259c640 1 -- 172.21.15.28:0/3313240386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e141a6a00 con 0x7f1e14100e20 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.817+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e14102ef0 0x7f1e1410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.817+0000 7f1e1259c640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14100e20 0x7f1e1410e4d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400eab0 tx=0x7f1e0400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.817+0000 7f1e0b7fe640 1 -- 172.21.15.28:0/3313240386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0400ce40 con 0x7f1e14100e20 2024-09-15T22:06:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.817+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e141a6cf0 con 0x7f1e14100e20 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.817+0000 7f1e0b7fe640 1 -- 172.21.15.28:0/3313240386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e04011950 con 0x7f1e14100e20 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.817+0000 7f1e0b7fe640 1 -- 172.21.15.28:0/3313240386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e04010ae0 con 0x7f1e14100e20 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.817+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e141a7220 con 0x7f1e14100e20 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.819+0000 7f1e0b7fe640 1 -- 172.21.15.28:0/3313240386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f1e04011b00 con 0x7f1e14100e20 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 Processor -- start 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 -- start start 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a049b60 0x55a68a049f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a054800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a057130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a689faf280 con 0x55a68a054d40 2024-09-15T22:06:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a689f56b10 con 0x55a68a049b60 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a689fab240 con 0x55a68a04a480 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e11d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a057130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e11d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a057130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42154/0 (socket says 172.21.15.28:42154) 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e11d9b640 1 -- 172.21.15.28:0/773541295 learned_addr learned my addr 172.21.15.28:0/773541295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e1259c640 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a054800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a049b60 0x55a68a049f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e0b7fe640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1e000a33b0 0x7f1e000a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e11d9b640 1 -- 172.21.15.28:0/773541295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 msgr2=0x55a68a054800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a054800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e11d9b640 1 -- 172.21.15.28:0/773541295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a049b60 msgr2=0x55a68a049f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.820+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a049b60 0x55a68a049f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.821+0000 7f1e11d9b640 1 -- 172.21.15.28:0/773541295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a689f631f0 con 0x55a68a054d40 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.821+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1e000a33b0 0x7f1e000a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.821+0000 7f1e0b7fe640 1 -- 172.21.15.28:0/3313240386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f1e040d08b0 con 0x7f1e14100e20 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.821+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a057130 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0314e0 tx=0x7f1dfc004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2596c5198471afba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.821+0000 7f1e08ff9640 1 -- 172.21.15.28:0/773541295 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc035030 con 0x55a68a054d40 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.821+0000 7f1e08ff9640 1 -- 172.21.15.28:0/773541295 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc005520 con 0x55a68a054d40 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.821+0000 7f1e08ff9640 1 -- 172.21.15.28:0/773541295 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc003850 con 0x55a68a054d40 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1e000a33b0 0x7f1e000a5870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c005fd0 tx=0x7f1e0c0046c0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/773541295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 msgr2=0x55a68a057130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a057130 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0314e0 tx=0x7f1dfc004170 comp rx=0 tx=0).stop 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/773541295 shutdown_connections 2024-09-15T22:06:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a057130 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a054800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/773541295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a049b60 0x55a68a049f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/773541295 >> 172.21.15.28:0/773541295 conn(0x55a68a0460f0 msgr2=0x55a68a0464d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/773541295 shutdown_connections 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/773541295 wait complete. 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.822+0000 7f1e1b1c4740 1 -- start start 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 1 Processor -- start 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a051ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a04a480 0x55a68a0520e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a052620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a68a057910 con 0x55a68a054d40 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a689f69f10 con 0x55a68a04a480 2024-09-15T22:06:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a68a057a90 con 0x55a68a049b60 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1b1c4740 10 client.0 _ll_register_callbacks cb 0x55a689f32a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a051ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a051ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58368/0 (socket says 172.21.15.28:58368) 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 -- 172.21.15.28:0/1945633753 learned_addr learned my addr 172.21.15.28:0/1945633753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a052620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a04a480 0x55a68a0520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 -- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a04a480 msgr2=0x55a68a0520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a04a480 0x55a68a0520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 -- 172.21.15.28:0/1945633753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 msgr2=0x55a68a052620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a052620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.823+0000 7f1e1259c640 1 -- 172.21.15.28:0/1945633753 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a689d206a0 con 0x55a68a049b60 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1259c640 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a051ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e040991b0 tx=0x7f1e04010540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd77419f4b106624 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1df9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1df9ffb640 1 -- 172.21.15.28:0/1945633753 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e040204b0 con 0x55a68a049b60 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1df9ffb640 1 -- 172.21.15.28:0/1945633753 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f1e0409ca40 con 0x55a68a049b60 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dfc005520 con 0x55a68a049b60 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1df9ffb640 1 -- 172.21.15.28:0/1945633753 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1e0409cee0 con 0x55a68a049b60 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1b1c4740 10 client.26509 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:01.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a68a0634a0 con 0x55a68a049b60 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1df9ffb640 1 -- 172.21.15.28:0/1945633753 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc005520 con 0x55a68a049b60 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1df9ffb640 1 -- 172.21.15.28:0/1945633753 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f1e0400f040 con 0x55a68a049b60 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1b1c4740 10 client.26509 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1b1c4740 10 client.26509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1b1c4740 20 client.26509 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.824+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0648b0 0x55a68a066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.825+0000 7f1e1b1c4740 4 client.26509 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.825+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a689fbc7f0 con 0x55a68a0648b0 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.825+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0648b0 0x55a68a066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.825+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0648b0 0x55a68a066ce0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69be8de193b11c65 server_cookie=11f3bb4ce94cdf53 in_seq=0 out_seq=0 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.825+0000 7f1df9ffb640 10 client.26509 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:01.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1df9ffb640 1 -- 172.21.15.28:0/1945633753 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a689fbc7f0 con 0x55a68a0648b0 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1df9ffb640 10 client.26509 handle_command_reply: tid=0 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 client.26509 shutdown 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 msgr2=0x55a68a051ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a051ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e040991b0 tx=0x7f1e04010540 comp rx=0 tx=0).stop 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 shutdown_connections 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0648b0 0x55a68a066ce0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a054d40 0x55a68a052620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a04a480 0x55a68a0520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1df9ffb640 0 client.26509 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/1945633753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a051ba0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 >> 172.21.15.28:0/1945633753 conn(0x55a68a0460f0 msgr2=0x55a68a04bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:01.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 shutdown_connections 2024-09-15T22:06:01.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/1945633753 wait complete. 2024-09-15T22:06:01.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.826+0000 7f1e1b1c4740 20 client.26509 trim_cache size 0 max 16384 2024-09-15T22:06:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 Processor -- start 2024-09-15T22:06:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 -- start start 2024-09-15T22:06:01.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a050b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a04a480 0x55a68a06c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a06cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141b1220 con 0x55a68a04a480 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141b2d90 con 0x55a68a054d40 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141b18f0 con 0x55a68a049b60 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a04a480 0x55a68a06c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a04a480 0x55a68a06c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42168/0 (socket says 172.21.15.28:42168) 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 -- 172.21.15.28:0/306960220 learned_addr learned my addr 172.21.15.28:0/306960220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1259c640 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a050b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a06cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 -- 172.21.15.28:0/306960220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 msgr2=0x55a68a050b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a050b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 -- 172.21.15.28:0/306960220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 msgr2=0x55a68a06cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a06cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 -- 172.21.15.28:0/306960220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a68a061150 con 0x55a68a04a480 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e1259c640 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a050b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a06cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.844+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a04a480 0x55a68a06c600 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc005c40 tx=0x7f1dfc004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc325ad0ab0a01a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1dfbfff640 1 -- 172.21.15.28:0/306960220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc035040 con 0x55a68a04a480 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1dfbfff640 1 -- 172.21.15.28:0/306960220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc005250 con 0x55a68a04a480 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1dfbfff640 1 -- 172.21.15.28:0/306960220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc003aa0 con 0x55a68a04a480 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/306960220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a04a480 msgr2=0x55a68a06c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a04a480 0x55a68a06c600 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc005c40 tx=0x7f1dfc004d00 comp rx=0 tx=0).stop 2024-09-15T22:06:01.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/306960220 shutdown_connections 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a06cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a04a480 0x55a68a06c600 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/306960220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a049b60 0x55a68a050b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/306960220 >> 172.21.15.28:0/306960220 conn(0x55a68a0460f0 msgr2=0x55a68a04bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/306960220 shutdown_connections 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.845+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/306960220 wait complete. 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 -- start start 2024-09-15T22:06:01.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 Processor -- start 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a049b60 0x55a68a05cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a05d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141b26c0 con 0x55a68a049b60 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141b1ff0 con 0x55a68a054d40 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e141af220 con 0x55a68a04a480 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1b1c4740 10 client.0 _ll_register_callbacks cb 0x7f1e04098410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a049b60 0x55a68a05cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a049b60 0x55a68a05cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42170/0 (socket says 172.21.15.28:42170) 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e1259c640 1 -- 172.21.15.28:0/2911308738 learned_addr learned my addr 172.21.15.28:0/2911308738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.846+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a05d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e12d9d640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1259c640 1 -- 172.21.15.28:0/2911308738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 msgr2=0x55a68a05d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1259c640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a05d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1259c640 1 -- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 msgr2=0x55a68a059cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1259c640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a059cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1259c640 1 -- 172.21.15.28:0/2911308738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a68a05a520 con 0x55a68a049b60 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a05d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1259c640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a049b60 0x55a68a05cec0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400c700 tx=0x7f1e040d0c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a861be8130bfce32 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1df9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1df9ffb640 1 -- 172.21.15.28:0/2911308738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0409c140 con 0x55a68a049b60 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1df9ffb640 1 -- 172.21.15.28:0/2911308738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f1e04011470 con 0x55a68a049b60 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dfc005250 con 0x55a68a049b60 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1df9ffb640 1 -- 172.21.15.28:0/2911308738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1e040204b0 con 0x55a68a049b60 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1b1c4740 10 client.17121 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a68a0d00f0 con 0x55a68a049b60 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.847+0000 7f1df9ffb640 1 -- 172.21.15.28:0/2911308738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc005250 con 0x55a68a049b60 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1df9ffb640 1 -- 172.21.15.28:0/2911308738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f1e04085ae0 con 0x55a68a049b60 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e1b1c4740 10 client.17121 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e1b1c4740 10 client.17121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e1b1c4740 20 client.17121 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0d0810 0x55a68a0d2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e1b1c4740 4 client.17121 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a689fb6d00 con 0x55a68a0d0810 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0d0810 0x55a68a0d2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1e11d9b640 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0d0810 0x55a68a0d2c20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f82cecc6ba2a173a server_cookie=3b68e5ddd9f3cc69 in_seq=0 out_seq=0 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.848+0000 7f1df9ffb640 10 client.17121 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.849+0000 7f1df9ffb640 1 -- 172.21.15.28:0/2911308738 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a689fb6d00 con 0x55a68a0d0810 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.849+0000 7f1df9ffb640 10 client.17121 handle_command_reply: tid=0 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.849+0000 7f1e1b1c4740 1 client.17121 shutdown 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a049b60 msgr2=0x55a68a05cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a049b60 0x55a68a05cec0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400c700 tx=0x7f1e040d0c40 comp rx=0 tx=0).stop 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 shutdown_connections 2024-09-15T22:06:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a68a0d0810 0x55a68a0d2c20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a68a054d40 0x55a68a059cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a68a04a480 0x55a68a05d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 --2- 172.21.15.28:0/2911308738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a68a049b60 0x55a68a05cec0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 >> 172.21.15.28:0/2911308738 conn(0x55a68a0460f0 msgr2=0x55a68a044a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1df9ffb640 0 client.17121 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 shutdown_connections 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 1 -- 172.21.15.28:0/2911308738 wait complete. 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e1b1c4740 20 client.17121 trim_cache size 0 max 16384 2024-09-15T22:06:01.857 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:01.858 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T22:06:01.858 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "affbcf5d-08c1-4a80-9273-0631de1d07a3", 2024-09-15T22:06:01.858 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T22:06:01.858 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1e000a33b0 msgr2=0x7f1e000a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e18deb640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1e000a33b0 0x7f1e000a5870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c005fd0 tx=0x7f1e0c0046c0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14100e20 msgr2=0x7f1e1410e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.850+0000 7f1e18deb640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14100e20 0x7f1e1410e4d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400eab0 tx=0x7f1e0400ef70 comp rx=0 tx=0).stop 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 shutdown_connections 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1e000a33b0 0x7f1e000a5870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e14102ef0 0x7f1e1410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e14102570 0x7f1e1410ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 --2- 172.21.15.28:0/3313240386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e14100e20 0x7f1e1410e4d0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 >> 172.21.15.28:0/3313240386 conn(0x7f1e140fc7d0 msgr2=0x7f1e140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 shutdown_connections 2024-09-15T22:06:01.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:01.851+0000 7f1e18deb640 1 -- 172.21.15.28:0/3313240386 wait complete. 2024-09-15T22:06:02.169 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:01 smithi028 ceph-mon[30969]: pgmap v265: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:06:02.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:01 smithi203 ceph-mon[35546]: pgmap v265: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:06:02.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:01 smithi174 ceph-mon[35244]: pgmap v265: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:06:02.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:06:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:03.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:02 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T22:06:03.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:02 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:03.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:02 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:06:03.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:02 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:03.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:02 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T22:06:03.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:02 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:03.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:02 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:06:03.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:02 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:03.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:02 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T22:06:03.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:02 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:03.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:02 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:06:03.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:02 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 -- 172.21.15.28:0/2187158321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c106a80 msgr2=0x7f251c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 --2- 172.21.15.28:0/2187158321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c106a80 0x7f251c106e60 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f250c009930 tx=0x7f250c02a420 comp rx=0 tx=0).stop 2024-09-15T22:06:03.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 -- 172.21.15.28:0/2187158321 shutdown_connections 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 --2- 172.21.15.28:0/2187158321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f251c101420 0x7f251c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 --2- 172.21.15.28:0/2187158321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f251c100a80 0x7f251c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 --2- 172.21.15.28:0/2187158321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c106a80 0x7f251c106e60 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 -- 172.21.15.28:0/2187158321 >> 172.21.15.28:0/2187158321 conn(0x7f251c0fc7a0 msgr2=0x7f251c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.443+0000 7f2521d3c640 1 -- 172.21.15.28:0/2187158321 shutdown_connections 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.444+0000 7f2521d3c640 1 -- 172.21.15.28:0/2187158321 wait complete. 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.444+0000 7f2521d3c640 1 Processor -- start 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.444+0000 7f2521d3c640 1 -- start start 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.444+0000 7f2521d3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c100a80 0x7f251c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.444+0000 7f2521d3c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f251c101420 0x7f251c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.444+0000 7f2521d3c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f251c106a80 0x7f251c19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.444+0000 7f2521d3c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f251c110ed0 con 0x7f251c100a80 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f2521d3c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f251c110d50 con 0x7f251c106a80 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f2521d3c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f251c111050 con 0x7f251c101420 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c100a80 0x7f251c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c100a80 0x7f251c1984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42202/0 (socket says 172.21.15.28:42202) 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 -- 172.21.15.28:0/1573757103 learned_addr learned my addr 172.21.15.28:0/1573757103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251affd640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f251c101420 0x7f251c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 -- 172.21.15.28:0/1573757103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f251c101420 msgr2=0x7f251c198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f251c101420 0x7f251c198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251bfff640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f251c106a80 0x7f251c19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 -- 172.21.15.28:0/1573757103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f251c106a80 msgr2=0x7f251c19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f251c106a80 0x7f251c19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 -- 172.21.15.28:0/1573757103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c19d470 con 0x7f251c100a80 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251bfff640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f251c106a80 0x7f251c19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f251b7fe640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c100a80 0x7f251c1984c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f250c02a940 tx=0x7f250c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f2518ff9640 1 -- 172.21.15.28:0/1573757103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c03f070 con 0x7f251c100a80 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f2518ff9640 1 -- 172.21.15.28:0/1573757103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f250c031c90 con 0x7f251c100a80 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251c19d700 con 0x7f251c100a80 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f2518ff9640 1 -- 172.21.15.28:0/1573757103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c035030 con 0x7f251c100a80 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.445+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f251c1a5040 con 0x7f251c100a80 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 Processor -- start 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2518ff9640 1 -- 172.21.15.28:0/1573757103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f250c042070 con 0x7f251c100a80 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 -- start start 2024-09-15T22:06:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc2f3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc302550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc304e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556dbc2592e0 con 0x556dbc302a90 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556dbc200ce0 con 0x556dbc2f33a0 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f2524115740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556dbc2552a0 con 0x556dbc2f3cc0 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.448+0000 7f251affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc304e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc304e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42206/0 (socket says 172.21.15.28:42206) 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 -- 172.21.15.28:0/3859586936 learned_addr learned my addr 172.21.15.28:0/3859586936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251b7fe640 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc302550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251bfff640 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc2f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 -- 172.21.15.28:0/3859586936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 msgr2=0x556dbc302550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc302550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 -- 172.21.15.28:0/3859586936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 msgr2=0x556dbc2f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc2f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 -- 172.21.15.28:0/3859586936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbc1e5440 con 0x556dbc302a90 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f2518ff9640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25040a33b0 0x7f25040a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251affd640 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc304e80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f251000bc80 tx=0x7f251000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1708d2482d5759 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f251bfff640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25040a33b0 0x7f25040a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f2518ff9640 1 -- 172.21.15.28:0/1573757103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f250c0f0940 con 0x7f251c100a80 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f2501ffb640 1 -- 172.21.15.28:0/3859586936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510015070 con 0x556dbc302a90 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f2501ffb640 1 -- 172.21.15.28:0/3859586936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f251001a940 con 0x556dbc302a90 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.449+0000 7f2501ffb640 1 -- 172.21.15.28:0/3859586936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25100119b0 con 0x556dbc302a90 2024-09-15T22:06:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 -- 172.21.15.28:0/3859586936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 msgr2=0x556dbc304e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc304e80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f251000bc80 tx=0x7f251000f730 comp rx=0 tx=0).stop 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f251bfff640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25040a33b0 0x7f25040a5870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2514005fd0 tx=0x7f2514009530 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 -- 172.21.15.28:0/3859586936 shutdown_connections 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc304e80 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc302550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 --2- 172.21.15.28:0/3859586936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc2f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 -- 172.21.15.28:0/3859586936 >> 172.21.15.28:0/3859586936 conn(0x556dbc2efa30 msgr2=0x556dbc2efe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 -- 172.21.15.28:0/3859586936 shutdown_connections 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 -- 172.21.15.28:0/3859586936 wait complete. 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.450+0000 7f2524115740 1 -- start start 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 1 Processor -- start 2024-09-15T22:06:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc3024b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc2ff760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc2ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556dbc3056c0 con 0x556dbc302a90 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556dbc214050 con 0x556dbc2f33a0 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556dbc305840 con 0x556dbc2f3cc0 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f2524115740 10 client.0 _ll_register_callbacks cb 0x556dbc254160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc2ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc2ffd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42208/0 (socket says 172.21.15.28:42208) 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 -- 172.21.15.28:0/534202079 learned_addr learned my addr 172.21.15.28:0/534202079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251affd640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc2ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251b7fe640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc3024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 -- 172.21.15.28:0/534202079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 msgr2=0x556dbc2ff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc2ff760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 -- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 msgr2=0x556dbc3024b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc3024b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251bfff640 1 -- 172.21.15.28:0/534202079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbbfca3c0 con 0x556dbc302a90 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251b7fe640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc3024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.451+0000 7f251affd640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc2ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f251bfff640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc2ffd70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2514012930 tx=0x7f2514004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3cdc23843f30f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f24faffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f24faffd640 1 -- 172.21.15.28:0/534202079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514022070 con 0x556dbc302a90 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f251001a940 con 0x556dbc302a90 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f24faffd640 1 -- 172.21.15.28:0/534202079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f251400fd60 con 0x556dbc302a90 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f24faffd640 1 -- 172.21.15.28:0/534202079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f251402b5c0 con 0x556dbc302a90 2024-09-15T22:06:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 10 client.17133 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556dbc3111f0 con 0x556dbc302a90 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f24faffd640 1 -- 172.21.15.28:0/534202079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f251001a940 con 0x556dbc302a90 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f24faffd640 1 -- 172.21.15.28:0/534202079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f2514027460 con 0x556dbc302a90 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 10 client.17133 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 20 client.17133 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc314b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 4 client.17133 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.452+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dbc266850 con 0x556dbc3126b0 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.453+0000 7f251b7fe640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc314b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.453+0000 7f251b7fe640 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc314b20 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1d7b199483da67c server_cookie=75a99a9d4d270af7 in_seq=0 out_seq=0 2024-09-15T22:06:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.453+0000 7f24faffd640 10 client.17133 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f24faffd640 1 -- 172.21.15.28:0/534202079 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556dbc266850 con 0x556dbc3126b0 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f24faffd640 10 client.17133 handle_command_reply: tid=0 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 client.17133 shutdown 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 msgr2=0x556dbc2ffd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc2ffd70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2514012930 tx=0x7f2514004a40 comp rx=0 tx=0).stop 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 shutdown_connections 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc314b20 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc2ffd70 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f3cc0 0x556dbc2ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f24faffd640 0 client.17133 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 --2- 172.21.15.28:0/534202079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f33a0 0x556dbc3024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 >> 172.21.15.28:0/534202079 conn(0x556dbc2efa30 msgr2=0x556dbc2f5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 shutdown_connections 2024-09-15T22:06:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 1 -- 172.21.15.28:0/534202079 wait complete. 2024-09-15T22:06:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.454+0000 7f2524115740 20 client.17133 trim_cache size 0 max 16384 2024-09-15T22:06:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.471+0000 7f2524115740 1 Processor -- start 2024-09-15T22:06:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.471+0000 7f2524115740 1 -- start start 2024-09-15T22:06:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f2524115740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f2524115740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc2fac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f2524115740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc31a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f2524115740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1aceb0 con 0x556dbc302a90 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f2524115740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1aea20 con 0x556dbc2f3cc0 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f2524115740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1ad580 con 0x556dbc2f33a0 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc31a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc31a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42218/0 (socket says 172.21.15.28:42218) 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 -- 172.21.15.28:0/3095763221 learned_addr learned my addr 172.21.15.28:0/3095763221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251affd640 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc2fac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251b7fe640 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 -- 172.21.15.28:0/3095763221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 msgr2=0x556dbc30e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 -- 172.21.15.28:0/3095763221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 msgr2=0x556dbc2fac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc2fac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 -- 172.21.15.28:0/3095763221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dbc302820 con 0x556dbc302a90 2024-09-15T22:06:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251affd640 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc2fac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251b7fe640 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.472+0000 7f251bfff640 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc31a3f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2514011c40 tx=0x7f25140097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1df1f3b977d3d57 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f25017fa640 1 -- 172.21.15.28:0/3095763221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514022070 con 0x556dbc302a90 2024-09-15T22:06:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f25017fa640 1 -- 172.21.15.28:0/3095763221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f251400f7a0 con 0x556dbc302a90 2024-09-15T22:06:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f25017fa640 1 -- 172.21.15.28:0/3095763221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514004ab0 con 0x556dbc302a90 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 -- 172.21.15.28:0/3095763221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 msgr2=0x556dbc31a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc31a3f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2514011c40 tx=0x7f25140097a0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 -- 172.21.15.28:0/3095763221 shutdown_connections 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc31a3f0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc2fac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 --2- 172.21.15.28:0/3095763221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 -- 172.21.15.28:0/3095763221 >> 172.21.15.28:0/3095763221 conn(0x556dbc2efa30 msgr2=0x556dbc2fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 -- 172.21.15.28:0/3095763221 shutdown_connections 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.473+0000 7f2524115740 1 -- 172.21.15.28:0/3095763221 wait complete. 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 -- start start 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 Processor -- start 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc306450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc306990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1ae350 con 0x556dbc302a90 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1adc80 con 0x556dbc2f3cc0 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.474+0000 7f2524115740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1aaeb0 con 0x556dbc2f33a0 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc306990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f2524115740 10 client.0 _ll_register_callbacks cb 0x7f2514009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc306990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42232/0 (socket says 172.21.15.28:42232) 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 -- 172.21.15.28:0/1046223182 learned_addr learned my addr 172.21.15.28:0/1046223182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251b7fe640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251affd640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc306450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 -- 172.21.15.28:0/1046223182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 msgr2=0x556dbc30b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 -- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 msgr2=0x556dbc306450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc306450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 -- 172.21.15.28:0/1046223182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dbc3071c0 con 0x556dbc302a90 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251affd640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc306450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f251bfff640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc306990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2514027850 tx=0x7f251402b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bd1ed44f742cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f24faffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f24faffd640 1 -- 172.21.15.28:0/1046223182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514022040 con 0x556dbc302a90 2024-09-15T22:06:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f24faffd640 1 -- 172.21.15.28:0/1046223182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f2514004a40 con 0x556dbc302a90 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.475+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f251400f7a0 con 0x556dbc302a90 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f24faffd640 1 -- 172.21.15.28:0/1046223182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f251404d050 con 0x556dbc302a90 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 10 client.17139 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556dbc37d240 con 0x556dbc302a90 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f24faffd640 1 -- 172.21.15.28:0/1046223182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f251400f7a0 con 0x556dbc302a90 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f24faffd640 1 -- 172.21.15.28:0/1046223182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f25140052a0 con 0x556dbc302a90 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 10 client.17139 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 10 client.17139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 20 client.17139 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc37f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 4 client.17139 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.476+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556dbc260d60 con 0x556dbc3126b0 2024-09-15T22:06:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.477+0000 7f251b7fe640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc37f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f24faffd640 10 client.17139 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f251b7fe640 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc37f890 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cdf68dd0b3fc2ed server_cookie=11ef9cf64c95d6ce in_seq=0 out_seq=0 2024-09-15T22:06:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f24faffd640 1 -- 172.21.15.28:0/1046223182 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556dbc260d60 con 0x556dbc3126b0 2024-09-15T22:06:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f24faffd640 10 client.17139 handle_command_reply: tid=0 2024-09-15T22:06:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 client.17139 shutdown 2024-09-15T22:06:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 msgr2=0x556dbc306990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc306990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2514027850 tx=0x7f251402b390 comp rx=0 tx=0).stop 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 shutdown_connections 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x556dbc3126b0 0x556dbc37f890 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dbc302a90 0x556dbc306990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f24faffd640 0 client.17139 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556dbc2f3cc0 0x556dbc306450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 --2- 172.21.15.28:0/1046223182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556dbc2f33a0 0x556dbc30b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 >> 172.21.15.28:0/1046223182 conn(0x556dbc2efa30 msgr2=0x556dbc3017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 shutdown_connections 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.479+0000 7f2524115740 1 -- 172.21.15.28:0/1046223182 wait complete. 2024-09-15T22:06:03.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2524115740 20 client.17139 trim_cache size 0 max 16384 2024-09-15T22:06:03.486 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:03.486 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:06:03.486 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:06:03.486 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25040a33b0 msgr2=0x7f25040a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25040a33b0 0x7f25040a5870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2514005fd0 tx=0x7f2514009530 comp rx=0 tx=0).stop 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c100a80 msgr2=0x7f251c1984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c100a80 0x7f251c1984c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f250c02a940 tx=0x7f250c003b90 comp rx=0 tx=0).stop 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 shutdown_connections 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f25040a33b0 0x7f25040a5870 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f251c106a80 0x7f251c19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f251c101420 0x7f251c198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 --2- 172.21.15.28:0/1573757103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c100a80 0x7f251c1984c0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.480+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 >> 172.21.15.28:0/1573757103 conn(0x7f251c0fc7a0 msgr2=0x7f251c06a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.481+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 shutdown_connections 2024-09-15T22:06:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:03.481+0000 7f2521d3c640 1 -- 172.21.15.28:0/1573757103 wait complete. 2024-09-15T22:06:04.172 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:06:04.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:06:04.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:03 smithi028 ceph-mon[30969]: pgmap v266: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:06:04.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:03 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:06:04.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:03 smithi203 ceph-mon[35546]: pgmap v266: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:06:04.219 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:03 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:06:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:03 smithi174 ceph-mon[35244]: pgmap v266: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:06:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:03 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:06:04.433 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 -- 172.21.15.28:0/1356031170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 msgr2=0x7fbb3010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/1356031170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3010e610 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009ca0 tx=0x7fbb2802a950 comp rx=0 tx=0).stop 2024-09-15T22:06:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 -- 172.21.15.28:0/1356031170 shutdown_connections 2024-09-15T22:06:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/1356031170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3010e610 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/1356031170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb30103660 0x7fbb3010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/1356031170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb30102d40 0x7fbb30103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 -- 172.21.15.28:0/1356031170 >> 172.21.15.28:0/1356031170 conn(0x7fbb300fc7a0 msgr2=0x7fbb300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 -- 172.21.15.28:0/1356031170 shutdown_connections 2024-09-15T22:06:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 -- 172.21.15.28:0/1356031170 wait complete. 2024-09-15T22:06:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.916+0000 7fbb35cf5640 1 Processor -- start 2024-09-15T22:06:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb35cf5640 1 -- start start 2024-09-15T22:06:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb35cf5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb30102d40 0x7fbb3019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb35cf5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb30103660 0x7fbb3019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb35cf5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb35cf5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30111f30 con 0x7fbb3010c170 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb35cf5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30111db0 con 0x7fbb30102d40 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb35cf5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301120b0 con 0x7fbb30103660 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42250/0 (socket says 172.21.15.28:42250) 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 -- 172.21.15.28:0/3442732466 learned_addr learned my addr 172.21.15.28:0/3442732466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2f7fe640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb30102d40 0x7fbb3019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2effd640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb30103660 0x7fbb3019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 -- 172.21.15.28:0/3442732466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb30103660 msgr2=0x7fbb3019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb30103660 0x7fbb3019ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 -- 172.21.15.28:0/3442732466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb30102d40 msgr2=0x7fbb3019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb30102d40 0x7fbb3019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 -- 172.21.15.28:0/3442732466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3019f580 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.917+0000 7fbb2ffff640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3019eea0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbb2802ae70 tx=0x7fbb2802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.918+0000 7fbb2cff9640 1 -- 172.21.15.28:0/3442732466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28034040 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.918+0000 7fbb2cff9640 1 -- 172.21.15.28:0/3442732466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb28030920 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.918+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3019f810 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.918+0000 7fbb2cff9640 1 -- 172.21.15.28:0/3442732466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28045480 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.918+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb301a7150 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.919+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3006b070 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.920+0000 7fbb2cff9640 1 -- 172.21.15.28:0/3442732466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fbb28037020 con 0x7fbb3010c170 2024-09-15T22:06:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.921+0000 7fbb2cff9640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb180a33b0 0x7fbb180a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.921+0000 7fbb2f7fe640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb180a33b0 0x7fbb180a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.921+0000 7fbb2cff9640 1 -- 172.21.15.28:0/3442732466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fbb280f0fb0 con 0x7fbb3010c170 2024-09-15T22:06:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.921+0000 7fbb2f7fe640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb180a33b0 0x7fbb180a5870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb20005fd0 tx=0x7fbb2000b590 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:04.925+0000 7fbb2cff9640 1 -- 172.21.15.28:0/3442732466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbb280bd6f0 con 0x7fbb3010c170 2024-09-15T22:06:05.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.140+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb301a7320 con 0x7fbb3010c170 2024-09-15T22:06:05.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.141+0000 7fbb2cff9640 1 -- 172.21.15.28:0/3442732466 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11988 (secure 0 0 0) 0x7fbb280bcff0 con 0x7fbb3010c170 2024-09-15T22:06:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:06:05.143 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"btime":"2024-09-15T22:04:37:022811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:04:36.713632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:06:05.144 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-09-15T22:06:05.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb180a33b0 msgr2=0x7fbb180a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:05.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb180a33b0 0x7fbb180a5870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb20005fd0 tx=0x7fbb2000b590 comp rx=0 tx=0).stop 2024-09-15T22:06:05.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 msgr2=0x7fbb3019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:05.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3019eea0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbb2802ae70 tx=0x7fbb2802f5b0 comp rx=0 tx=0).stop 2024-09-15T22:06:05.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 shutdown_connections 2024-09-15T22:06:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb180a33b0 0x7fbb180a5870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3010c170 0x7fbb3019eea0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb30103660 0x7fbb3019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 --2- 172.21.15.28:0/3442732466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb30102d40 0x7fbb3019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 >> 172.21.15.28:0/3442732466 conn(0x7fbb300fc7a0 msgr2=0x7fbb300fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 shutdown_connections 2024-09-15T22:06:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:05.144+0000 7fbb35cf5640 1 -- 172.21.15.28:0/3442732466 wait complete. 2024-09-15T22:06:05.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:06:06.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:05 smithi028 ceph-mon[30969]: pgmap v267: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-09-15T22:06:06.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:05 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3442732466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:06.213 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:06.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:05 smithi203 ceph-mon[35546]: pgmap v267: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-09-15T22:06:06.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:05 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3442732466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:06.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:05 smithi174 ceph-mon[35244]: pgmap v267: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-09-15T22:06:06.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:05 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3442732466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 -- 172.21.15.28:0/63279644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 msgr2=0x7f2dd0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 --2- 172.21.15.28:0/63279644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd0100ee0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009930 tx=0x7f2dc002a420 comp rx=0 tx=0).stop 2024-09-15T22:06:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 -- 172.21.15.28:0/63279644 shutdown_connections 2024-09-15T22:06:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 --2- 172.21.15.28:0/63279644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd0101420 0x7f2dd010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 --2- 172.21.15.28:0/63279644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd0100ee0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 --2- 172.21.15.28:0/63279644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dd0106a80 0x7f2dd0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 -- 172.21.15.28:0/63279644 >> 172.21.15.28:0/63279644 conn(0x7f2dd00fc7a0 msgr2=0x7f2dd00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 -- 172.21.15.28:0/63279644 shutdown_connections 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.720+0000 7f2dd7576640 1 -- 172.21.15.28:0/63279644 wait complete. 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 Processor -- start 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 -- start start 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd0101420 0x7f2dd019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dd0106a80 0x7f2dd019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd0111fa0 con 0x7f2dd0100a80 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd0111e20 con 0x7f2dd0106a80 2024-09-15T22:06:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd7576640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd0112120 con 0x7f2dd0101420 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd019a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42258/0 (socket says 172.21.15.28:42258) 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 -- 172.21.15.28:0/2131504454 learned_addr learned my addr 172.21.15.28:0/2131504454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dd0106a80 0x7f2dd019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 -- 172.21.15.28:0/2131504454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd0101420 msgr2=0x7f2dd019abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd0101420 0x7f2dd019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd0101420 0x7f2dd019abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 -- 172.21.15.28:0/2131504454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dd0106a80 msgr2=0x7f2dd019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dd0106a80 0x7f2dd019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5312640 1 -- 172.21.15.28:0/2131504454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd019f620 con 0x7f2dd0100a80 2024-09-15T22:06:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dd0106a80 0x7f2dd019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.721+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd0101420 0x7f2dd019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.722+0000 7f2dd5312640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd019a670 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00d950 tx=0x7f2dcc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.722+0000 7f2dc67fc640 1 -- 172.21.15.28:0/2131504454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc017070 con 0x7f2dd0100a80 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.722+0000 7f2dc67fc640 1 -- 172.21.15.28:0/2131504454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dcc009c30 con 0x7f2dd0100a80 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.722+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd019f910 con 0x7f2dd0100a80 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.722+0000 7f2dc67fc640 1 -- 172.21.15.28:0/2131504454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc00f6b0 con 0x7f2dd0100a80 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.722+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd01a71f0 con 0x7f2dd0100a80 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.724+0000 7f2dc67fc640 1 -- 172.21.15.28:0/2131504454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f2dcc0109d0 con 0x7f2dd0100a80 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 Processor -- start 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 -- start start 2024-09-15T22:06:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d6e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6e5d0 0x556242d6ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d7b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556242cd42f0 con 0x556242d6fb40 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556242c7bcf0 con 0x556242d79370 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd994f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556242cd02b0 con 0x556242d6e5d0 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd5b13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d6e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd5b13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d6e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42262/0 (socket says 172.21.15.28:42262) 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd5b13640 1 -- 172.21.15.28:0/3347213357 learned_addr learned my addr 172.21.15.28:0/3347213357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dc67fc640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2dbc0a33b0 0x7f2dbc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d7b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.725+0000 7f2dd5312640 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6e5d0 0x556242d6ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 -- 172.21.15.28:0/3347213357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6e5d0 msgr2=0x556242d6ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6e5d0 0x556242d6ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 -- 172.21.15.28:0/3347213357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 msgr2=0x556242d7b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d7b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 -- 172.21.15.28:0/3347213357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556242c603a0 con 0x556242d6fb40 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5312640 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6e5d0 0x556242d6ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d7b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dc67fc640 1 -- 172.21.15.28:0/2131504454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f2dcc0cfd70 con 0x7f2dd0100a80 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2dbc0a33b0 0x7f2dbc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d6e090 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800f4b0 tx=0x7f2dc800f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61f7e5002d6ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/3347213357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8018070 con 0x556242d6fb40 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/3347213357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dc8021070 con 0x556242d6fb40 2024-09-15T22:06:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/3347213357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8010b50 con 0x556242d6fb40 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.726+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2dbc0a33b0 0x7f2dbc0a5870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8020000 tx=0x7f2dc801e000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 -- 172.21.15.28:0/3347213357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 msgr2=0x556242d6e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d6e090 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800f4b0 tx=0x7f2dc800f970 comp rx=0 tx=0).stop 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 -- 172.21.15.28:0/3347213357 shutdown_connections 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d7b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6e5d0 0x556242d6ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3347213357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d6e090 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 -- 172.21.15.28:0/3347213357 >> 172.21.15.28:0/3347213357 conn(0x556242d37310 msgr2=0x556242d6ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 -- 172.21.15.28:0/3347213357 shutdown_connections 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 -- 172.21.15.28:0/3347213357 wait complete. 2024-09-15T22:06:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 -- start start 2024-09-15T22:06:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.727+0000 7f2dd994f740 1 Processor -- start 2024-09-15T22:06:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d78cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6fb40 0x556242d76020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d76580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd994f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556242d7c070 con 0x556242d6e5d0 2024-09-15T22:06:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd994f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556242c8f060 con 0x556242d79370 2024-09-15T22:06:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd994f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556242d7c1f0 con 0x556242d6fb40 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd994f740 10 client.0 _ll_register_callbacks cb 0x5562429e28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d78cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d78cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42270/0 (socket says 172.21.15.28:42270) 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 -- 172.21.15.28:0/9105140 learned_addr learned my addr 172.21.15.28:0/9105140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d76580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6fb40 0x556242d76020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 -- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6fb40 msgr2=0x556242d76020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6fb40 0x556242d76020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 -- 172.21.15.28:0/9105140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 msgr2=0x556242d76580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d76580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 -- 172.21.15.28:0/9105140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556242a453c0 con 0x556242d6e5d0 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d76580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6fb40 0x556242d76020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2dd5312640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d78cd0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc09bbd0 tx=0x7f2dcc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8d021066d3cb88 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.728+0000 7f2db8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2db8ff9640 1 -- 172.21.15.28:0/9105140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc020030 con 0x556242d6e5d0 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2db8ff9640 1 -- 172.21.15.28:0/9105140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f2dcc0860e0 con 0x556242d6e5d0 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dc8021070 con 0x556242d6e5d0 2024-09-15T22:06:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2db8ff9640 1 -- 172.21.15.28:0/9105140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2dcc0cd530 con 0x556242d6e5d0 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 10 client.17157 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556242d87a40 con 0x556242d6e5d0 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2db8ff9640 1 -- 172.21.15.28:0/9105140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dc8021070 con 0x556242d6e5d0 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2db8ff9640 1 -- 172.21.15.28:0/9105140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f2dcc09b120 con 0x556242d6e5d0 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 10 client.17157 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 10 client.17157 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 20 client.17157 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242d88d90 0x556242d8b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 4 client.17157 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.729+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556242ce1860 con 0x556242d88d90 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.730+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242d88d90 0x556242d8b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.730+0000 7f2db8ff9640 10 client.17157 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.730+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242d88d90 0x556242d8b1c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4cc217ed025a86d7 server_cookie=dafa6d2cf245071d in_seq=0 out_seq=0 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2db8ff9640 1 -- 172.21.15.28:0/9105140 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556242ce1860 con 0x556242d88d90 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2db8ff9640 10 client.17157 handle_command_reply: tid=0 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 client.17157 shutdown 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 msgr2=0x556242d78cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d78cd0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc09bbd0 tx=0x7f2dcc000bc0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 shutdown_connections 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242d88d90 0x556242d8b1c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d79370 0x556242d76580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d6fb40 0x556242d76020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2db8ff9640 0 client.17157 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 --2- 172.21.15.28:0/9105140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d78cd0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 >> 172.21.15.28:0/9105140 conn(0x556242d37310 msgr2=0x556242d70960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 shutdown_connections 2024-09-15T22:06:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 1 -- 172.21.15.28:0/9105140 wait complete. 2024-09-15T22:06:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.731+0000 7f2dd994f740 20 client.17157 trim_cache size 0 max 16384 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.744+0000 7f2dd994f740 1 Processor -- start 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd994f740 1 -- start start 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6e5d0 0x556242d85310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d90a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242d90fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd994f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd01b12f0 con 0x556242d6fb40 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd994f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd01b2e60 con 0x556242d6e5d0 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd994f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd01b19c0 con 0x556242d79370 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d90a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d90a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42274/0 (socket says 172.21.15.28:42274) 2024-09-15T22:06:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 -- 172.21.15.28:0/3194866321 learned_addr learned my addr 172.21.15.28:0/3194866321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd5312640 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6e5d0 0x556242d85310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 -- 172.21.15.28:0/3194866321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 msgr2=0x556242d90fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242d90fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242d90fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 -- 172.21.15.28:0/3194866321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6e5d0 msgr2=0x556242d85310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6e5d0 0x556242d85310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 -- 172.21.15.28:0/3194866321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556242d74570 con 0x556242d6fb40 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242d90fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd5312640 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6e5d0 0x556242d85310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.745+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d90a90 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0005ab0 tx=0x7f2dc0004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2251e772d51d2519 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dbaffd640 1 -- 172.21.15.28:0/3194866321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0035040 con 0x556242d6fb40 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dbaffd640 1 -- 172.21.15.28:0/3194866321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dc0005520 con 0x556242d6fb40 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dbaffd640 1 -- 172.21.15.28:0/3194866321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00035f0 con 0x556242d6fb40 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 -- 172.21.15.28:0/3194866321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 msgr2=0x556242d90a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d90a90 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0005ab0 tx=0x7f2dc0004260 comp rx=0 tx=0).stop 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 -- 172.21.15.28:0/3194866321 shutdown_connections 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242d90fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6fb40 0x556242d90a90 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3194866321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6e5d0 0x556242d85310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 -- 172.21.15.28:0/3194866321 >> 172.21.15.28:0/3194866321 conn(0x556242d37310 msgr2=0x556242d712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 -- 172.21.15.28:0/3194866321 shutdown_connections 2024-09-15T22:06:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.746+0000 7f2dd994f740 1 -- 172.21.15.28:0/3194866321 wait complete. 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 -- start start 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 Processor -- start 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d7da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6fb40 0x556242d7dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242decaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd01b2790 con 0x556242d6e5d0 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd01b20c0 con 0x556242d6fb40 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd01af2f0 con 0x556242d79370 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.747+0000 7f2dd994f740 10 client.0 _ll_register_callbacks cb 0x7f2dcc00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d7da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d7da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42286/0 (socket says 172.21.15.28:42286) 2024-09-15T22:06:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 -- 172.21.15.28:0/3923117976 learned_addr learned my addr 172.21.15.28:0/3923117976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5b13640 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242decaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6fb40 0x556242d7dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 -- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 msgr2=0x556242decaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242decaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 -- 172.21.15.28:0/3923117976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6fb40 msgr2=0x556242d7dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6fb40 0x556242d7dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 -- 172.21.15.28:0/3923117976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556242ded0e0 con 0x556242d6e5d0 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd5312640 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d7da90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0a4380 tx=0x7f2dcc0cd560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eada541c1c2a6721 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2db8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2db8ff9640 1 -- 172.21.15.28:0/3923117976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc09b5a0 con 0x556242d6e5d0 2024-09-15T22:06:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2db8ff9640 1 -- 172.21.15.28:0/3923117976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7f2dcc0860b0 con 0x556242d6e5d0 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dc0005520 con 0x556242d6e5d0 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2db8ff9640 1 -- 172.21.15.28:0/3923117976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2dcc09b2a0 con 0x556242d6e5d0 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd994f740 10 client.17163 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.748+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556242df4400 con 0x556242d6e5d0 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2db8ff9640 1 -- 172.21.15.28:0/3923117976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dc0005520 con 0x556242d6e5d0 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2db8ff9640 1 -- 172.21.15.28:0/3923117976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f2dcc09ab00 con 0x556242d6e5d0 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2dd994f740 10 client.17163 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2dd994f740 10 client.17163 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2dd994f740 20 client.17163 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242df4a20 0x556242df6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2dd994f740 4 client.17163 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556242cdbd70 con 0x556242df4a20 2024-09-15T22:06:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.749+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242df4a20 0x556242df6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd4b11640 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242df4a20 0x556242df6e40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7c6df7ebbda93619 server_cookie=d3d832fdde9a2136 in_seq=0 out_seq=0 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2db8ff9640 10 client.17163 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2db8ff9640 1 -- 172.21.15.28:0/3923117976 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556242cdbd70 con 0x556242df4a20 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2db8ff9640 10 client.17163 handle_command_reply: tid=0 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 client.17163 shutdown 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 msgr2=0x556242d7da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d7da90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0a4380 tx=0x7f2dcc0cd560 comp rx=0 tx=0).stop 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 shutdown_connections 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x556242df4a20 0x556242df6e40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556242d79370 0x556242decaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556242d6fb40 0x556242d7dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 --2- 172.21.15.28:0/3923117976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556242d6e5d0 0x556242d7da90 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 >> 172.21.15.28:0/3923117976 conn(0x556242d37310 msgr2=0x556242d70110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2db8ff9640 0 client.17163 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 shutdown_connections 2024-09-15T22:06:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 1 -- 172.21.15.28:0/3923117976 wait complete. 2024-09-15T22:06:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.750+0000 7f2dd994f740 20 client.17163 trim_cache size 0 max 16384 2024-09-15T22:06:06.763 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2dbc0a33b0 msgr2=0x7f2dbc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2dbc0a33b0 0x7f2dbc0a5870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8020000 tx=0x7f2dc801e000 comp rx=0 tx=0).stop 2024-09-15T22:06:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 msgr2=0x7f2dd019a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd019a670 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00d950 tx=0x7f2dcc00de10 comp rx=0 tx=0).stop 2024-09-15T22:06:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 shutdown_connections 2024-09-15T22:06:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f2dbc0a33b0 0x7f2dbc0a5870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dd0106a80 0x7f2dd019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dd0101420 0x7f2dd019abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 --2- 172.21.15.28:0/2131504454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dd0100a80 0x7f2dd019a670 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:06.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 >> 172.21.15.28:0/2131504454 conn(0x7f2dd00fc7a0 msgr2=0x7f2dd00fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:06.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 shutdown_connections 2024-09-15T22:06:06.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:06.751+0000 7f2dd7576640 1 -- 172.21.15.28:0/2131504454 wait complete. 2024-09-15T22:06:07.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:06:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:08.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:07 smithi028 ceph-mon[30969]: pgmap v268: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:06:08.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:07 smithi203 ceph-mon[35546]: pgmap v268: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:06:08.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 -- 172.21.15.28:0/3620979903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 msgr2=0x7fa8f41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3620979903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f41067e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc009900 tx=0x7fa8dc02a410 comp rx=0 tx=0).stop 2024-09-15T22:06:08.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 -- 172.21.15.28:0/3620979903 shutdown_connections 2024-09-15T22:06:08.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3620979903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f4106f10 0x7fa8f410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3620979903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f41067e0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3620979903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8f4101de0 0x7fa8f41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 -- 172.21.15.28:0/3620979903 >> 172.21.15.28:0/3620979903 conn(0x7fa8f40fd8c0 msgr2=0x7fa8f40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 -- 172.21.15.28:0/3620979903 shutdown_connections 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.213+0000 7fa8f9136640 1 -- 172.21.15.28:0/3620979903 wait complete. 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 Processor -- start 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 -- start start 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f4101de0 0x7fa8f419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f419ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8f4106f10 0x7fa8f419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f4111fb0 con 0x7fa8f4102790 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f4111e30 con 0x7fa8f4101de0 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.214+0000 7fa8f9136640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f4112130 con 0x7fa8f4106f10 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f419ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f419ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42306/0 (socket says 172.21.15.28:42306) 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 -- 172.21.15.28:0/3628346823 learned_addr learned my addr 172.21.15.28:0/3628346823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f4101de0 0x7fa8f419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f359e640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8f4106f10 0x7fa8f419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 -- 172.21.15.28:0/3628346823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8f4106f10 msgr2=0x7fa8f419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8f4106f10 0x7fa8f419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 -- 172.21.15.28:0/3628346823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f4101de0 msgr2=0x7fa8f419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f4101de0 0x7fa8f419a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 -- 172.21.15.28:0/3628346823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f419f6b0 con 0x7fa8f4102790 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f259c640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f419ac40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc005ab0 tx=0x7fa8dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8ebfff640 1 -- 172.21.15.28:0/3628346823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc03f030 con 0x7fa8f4102790 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8ebfff640 1 -- 172.21.15.28:0/3628346823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8dc003ce0 con 0x7fa8f4102790 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f419f940 con 0x7fa8f4102790 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8ebfff640 1 -- 172.21.15.28:0/3628346823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc035070 con 0x7fa8f4102790 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.215+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f41a7280 con 0x7fa8f4102790 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.218+0000 7fa8ebfff640 1 -- 172.21.15.28:0/3628346823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fa8dc042030 con 0x7fa8f4102790 2024-09-15T22:06:08.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.218+0000 7fa8fb50f740 1 Processor -- start 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.218+0000 7fa8fb50f740 1 -- start start 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.218+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b1b40 0x55b76b2b0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.218+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b05d0 0x55b76b2b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.218+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2bd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b76b2162f0 con 0x55b76b2b1b40 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b76b1bdcf0 con 0x55b76b2bb370 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b76b2122b0 con 0x55b76b2b05d0 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b1b40 0x55b76b2b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8ebfff640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa8e00a33b0 0x7fa8e00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b1b40 0x55b76b2b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42318/0 (socket says 172.21.15.28:42318) 2024-09-15T22:06:08.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 -- 172.21.15.28:0/1433871278 learned_addr learned my addr 172.21.15.28:0/1433871278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b05d0 0x55b76b2b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa8e00a33b0 0x7fa8e00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8ebfff640 1 -- 172.21.15.28:0/3628346823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fa8dc0f0620 con 0x7fa8f4102790 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f259c640 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2bd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 -- 172.21.15.28:0/1433871278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b05d0 msgr2=0x55b76b2b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b05d0 0x55b76b2b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 -- 172.21.15.28:0/1433871278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 msgr2=0x55b76b2bd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2bd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 -- 172.21.15.28:0/1433871278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b76b1a23a0 con 0x55b76b2b1b40 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b05d0 0x55b76b2b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:08.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.219+0000 7fa8f359e640 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b1b40 0x55b76b2b0090 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec012890 tx=0x7fa8ec012d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50747d1eddef5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8e97fa640 1 -- 172.21.15.28:0/1433871278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec003d40 con 0x55b76b2b1b40 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8e97fa640 1 -- 172.21.15.28:0/1433871278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec004b60 con 0x55b76b2b1b40 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8e97fa640 1 -- 172.21.15.28:0/1433871278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec005050 con 0x55b76b2b1b40 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8fb50f740 1 -- 172.21.15.28:0/1433871278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b1b40 msgr2=0x55b76b2b0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b1b40 0x55b76b2b0090 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec012890 tx=0x7fa8ec012d50 comp rx=0 tx=0).stop 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8f359e640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa8e00a33b0 0x7fa8e00a5870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec01f000 tx=0x7fa8ec01e000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8fb50f740 1 -- 172.21.15.28:0/1433871278 shutdown_connections 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2bd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b05d0 0x55b76b2b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/1433871278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b1b40 0x55b76b2b0090 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.220+0000 7fa8fb50f740 1 -- 172.21.15.28:0/1433871278 >> 172.21.15.28:0/1433871278 conn(0x55b76b279310 msgr2=0x55b76b2acc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 -- 172.21.15.28:0/1433871278 shutdown_connections 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 -- 172.21.15.28:0/1433871278 wait complete. 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 -- start start 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 Processor -- start 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b05d0 0x55b76b2bad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2b8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2b8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b76b2be070 con 0x55b76b2b05d0 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b76b1d1060 con 0x55b76b2bb370 2024-09-15T22:06:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b76b2be1f0 con 0x55b76b2b1b40 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.221+0000 7fa8fb50f740 10 client.0 _ll_register_callbacks cb 0x55b76af248c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b05d0 0x55b76b2bad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b05d0 0x55b76b2bad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42322/0 (socket says 172.21.15.28:42322) 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 -- 172.21.15.28:0/2622866635 learned_addr learned my addr 172.21.15.28:0/2622866635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f359e640 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f259c640 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 -- 172.21.15.28:0/2622866635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 msgr2=0x55b76b2b8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2b8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 -- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 msgr2=0x55b76b2b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 -- 172.21.15.28:0/2622866635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b76af873c0 con 0x55b76b2b05d0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b05d0 0x55b76b2bad70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009ed0 tx=0x7fa8e4011630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1aff03524c9f5e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2622866635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4019070 con 0x55b76b2b05d0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2622866635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fa8e4015870 con 0x55b76b2b05d0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.222+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ec004b60 con 0x55b76b2b05d0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2622866635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa8e4025050 con 0x55b76b2b05d0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 10 client.17175 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b76b2c9b70 con 0x55b76b2b05d0 2024-09-15T22:06:08.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2622866635 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec004b60 con 0x55b76b2b05d0 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2622866635 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa8e4014ae0 con 0x55b76b2b05d0 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 10 client.17175 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 10 client.17175 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 20 client.17175 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b2caf80 0x55b76b2cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 4 client.17175 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.223+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b76b223860 con 0x55b76b2caf80 2024-09-15T22:06:08.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.224+0000 7fa8f259c640 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b2caf80 0x55b76b2cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.224+0000 7fa8f259c640 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b2caf80 0x55b76b2cd3d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d78c4aa918c57d70 server_cookie=92e6ea9e20d10de6 in_seq=0 out_seq=0 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.224+0000 7fa8da7fc640 10 client.17175 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2622866635 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b76b223860 con 0x55b76b2caf80 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8da7fc640 10 client.17175 handle_command_reply: tid=0 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 client.17175 shutdown 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b05d0 msgr2=0x55b76b2bad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b05d0 0x55b76b2bad70 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009ed0 tx=0x7fa8e4011630 comp rx=0 tx=0).stop 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 shutdown_connections 2024-09-15T22:06:08.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2bb370 0x55b76b2b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2b8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b2caf80 0x55b76b2cd3d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2622866635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2b05d0 0x55b76b2bad70 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 >> 172.21.15.28:0/2622866635 conn(0x55b76b279310 msgr2=0x55b76b2b2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8da7fc640 0 client.17175 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 shutdown_connections 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2622866635 wait complete. 2024-09-15T22:06:08.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.225+0000 7fa8fb50f740 20 client.17175 trim_cache size 0 max 16384 2024-09-15T22:06:08.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.238+0000 7fa8fb50f740 1 Processor -- start 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.238+0000 7fa8fb50f740 1 -- start start 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2d2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41b1230 con 0x55b76b2bb370 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41b2da0 con 0x55b76b2b05d0 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41b1900 con 0x55b76b2b1b40 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2d2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2d2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42338/0 (socket says 172.21.15.28:42338) 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 -- 172.21.15.28:0/505309572 learned_addr learned my addr 172.21.15.28:0/505309572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 -- 172.21.15.28:0/505309572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 msgr2=0x55b76b2b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 -- 172.21.15.28:0/505309572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 msgr2=0x55b76b2c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 -- 172.21.15.28:0/505309572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b76b2ad960 con 0x55b76b2bb370 2024-09-15T22:06:08.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8f359e640 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2d2ca0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec012890 tx=0x7fa8ec00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d4d3bd59df823f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8e8ff9640 1 -- 172.21.15.28:0/505309572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec038070 con 0x55b76b2bb370 2024-09-15T22:06:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8e8ff9640 1 -- 172.21.15.28:0/505309572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec005050 con 0x55b76b2bb370 2024-09-15T22:06:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.239+0000 7fa8e8ff9640 1 -- 172.21.15.28:0/505309572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0058e0 con 0x55b76b2bb370 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 -- 172.21.15.28:0/505309572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 msgr2=0x55b76b2d2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2d2ca0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec012890 tx=0x7fa8ec00e000 comp rx=0 tx=0).stop 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 -- 172.21.15.28:0/505309572 shutdown_connections 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2d2ca0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/505309572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 -- 172.21.15.28:0/505309572 >> 172.21.15.28:0/505309572 conn(0x55b76b279310 msgr2=0x55b76b2b68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 -- 172.21.15.28:0/505309572 shutdown_connections 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.240+0000 7fa8fb50f740 1 -- 172.21.15.28:0/505309572 wait complete. 2024-09-15T22:06:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 -- start start 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 Processor -- start 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2becd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2bf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41b26d0 con 0x55b76b2bb370 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41b2000 con 0x55b76b2b05d0 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f41af230 con 0x55b76b2b1b40 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8fb50f740 10 client.0 _ll_register_callbacks cb 0x7fa8e4009410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2bf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2bf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42354/0 (socket says 172.21.15.28:42354) 2024-09-15T22:06:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 -- 172.21.15.28:0/2536753144 learned_addr learned my addr 172.21.15.28:0/2536753144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 -- 172.21.15.28:0/2536753144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 msgr2=0x55b76b2becd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2becd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 -- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 msgr2=0x55b76b2c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.241+0000 7fa8f359e640 1 -- 172.21.15.28:0/2536753144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b76b2bf7d0 con 0x55b76b2bb370 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8f359e640 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2bf210 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec036040 tx=0x7fa8ec005200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345a75016c064fef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2536753144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec041060 con 0x55b76b2bb370 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ec005050 con 0x55b76b2bb370 2024-09-15T22:06:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2536753144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9467+0+0 (secure 0 0 0) 0x7fa8ec004a00 con 0x55b76b2bb370 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2536753144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa8ec038040 con 0x55b76b2bb370 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 10 client.17181 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b76b335a40 con 0x55b76b2bb370 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2536753144 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec005050 con 0x55b76b2bb370 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2536753144 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa8ec025a70 con 0x55b76b2bb370 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 10 client.17181 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 10 client.17181 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 20 client.17181 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b335f40 0x55b76b338360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 4 client.17181 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.242+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b76b21dd70 con 0x55b76b335f40 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.243+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b335f40 0x55b76b338360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.243+0000 7fa8f2d9d640 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b335f40 0x55b76b338360 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4b70720558db1a12 server_cookie=e3d3e15b47947122 in_seq=0 out_seq=0 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.243+0000 7fa8da7fc640 10 client.17181 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.243+0000 7fa8da7fc640 1 -- 172.21.15.28:0/2536753144 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b76b21dd70 con 0x55b76b335f40 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.243+0000 7fa8da7fc640 10 client.17181 handle_command_reply: tid=0 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.243+0000 7fa8fb50f740 1 client.17181 shutdown 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 msgr2=0x55b76b2bf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2bf210 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec036040 tx=0x7fa8ec005200 comp rx=0 tx=0).stop 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 shutdown_connections 2024-09-15T22:06:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b76b335f40 0x55b76b338360 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76b2bb370 0x55b76b2bf210 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b76b2b1b40 0x55b76b2becd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 --2- 172.21.15.28:0/2536753144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b76b2b05d0 0x55b76b2c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 >> 172.21.15.28:0/2536753144 conn(0x55b76b279310 msgr2=0x55b76b2b68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8da7fc640 0 client.17181 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 shutdown_connections 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 1 -- 172.21.15.28:0/2536753144 wait complete. 2024-09-15T22:06:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8fb50f740 20 client.17181 trim_cache size 0 max 16384 2024-09-15T22:06:08.253 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa8e00a33b0 msgr2=0x7fa8e00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa8e00a33b0 0x7fa8e00a5870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec01f000 tx=0x7fa8ec01e000 comp rx=0 tx=0).stop 2024-09-15T22:06:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 msgr2=0x7fa8f419ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f419ac40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc005ab0 tx=0x7fa8dc0042f0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 shutdown_connections 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa8e00a33b0 0x7fa8e00a5870 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8f4106f10 0x7fa8f419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8f4102790 0x7fa8f419ac40 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 --2- 172.21.15.28:0/3628346823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f4101de0 0x7fa8f419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.244+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 >> 172.21.15.28:0/3628346823 conn(0x7fa8f40fd8c0 msgr2=0x7fa8f40feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.245+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 shutdown_connections 2024-09-15T22:06:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:08.245+0000 7fa8f9136640 1 -- 172.21.15.28:0/3628346823 wait complete. 2024-09-15T22:06:08.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:07 smithi174 ceph-mon[35244]: pgmap v268: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:06:08.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:06:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:10.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:09 smithi028 ceph-mon[30969]: pgmap v269: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 30 op/s 2024-09-15T22:06:10.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:09 smithi028 ceph-mon[30969]: osdmap e96: 12 total, 12 up, 12 in 2024-09-15T22:06:10.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:09 smithi203 ceph-mon[35546]: pgmap v269: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 30 op/s 2024-09-15T22:06:10.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:09 smithi203 ceph-mon[35546]: osdmap e96: 12 total, 12 up, 12 in 2024-09-15T22:06:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:09 smithi174 ceph-mon[35244]: pgmap v269: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 45 KiB/s wr, 30 op/s 2024-09-15T22:06:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:09 smithi174 ceph-mon[35244]: osdmap e96: 12 total, 12 up, 12 in 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.259+0000 7f988c345640 1 -- 172.21.15.28:0/2604511770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884102570 msgr2=0x7f98841029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.259+0000 7f988c345640 1 --2- 172.21.15.28:0/2604511770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884102570 0x7f98841029b0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f987c009ca0 tx=0x7f987c02a980 comp rx=0 tx=0).stop 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.260+0000 7f988c345640 1 -- 172.21.15.28:0/2604511770 shutdown_connections 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.260+0000 7f988c345640 1 --2- 172.21.15.28:0/2604511770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9884102ef0 0x7f988410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.260+0000 7f988c345640 1 --2- 172.21.15.28:0/2604511770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884102570 0x7f98841029b0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.260+0000 7f988c345640 1 --2- 172.21.15.28:0/2604511770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884100e20 0x7f9884101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.260+0000 7f988c345640 1 -- 172.21.15.28:0/2604511770 >> 172.21.15.28:0/2604511770 conn(0x7f98840fc7d0 msgr2=0x7f98840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.260+0000 7f988c345640 1 -- 172.21.15.28:0/2604511770 shutdown_connections 2024-09-15T22:06:10.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.260+0000 7f988c345640 1 -- 172.21.15.28:0/2604511770 wait complete. 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.261+0000 7f988c345640 1 Processor -- start 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f988c345640 1 -- start start 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f988c345640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884100e20 0x7f988419e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f988c345640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884102570 0x7f988419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f988c345640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9884102ef0 0x7f98841965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f988c345640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98841110b0 con 0x7f9884100e20 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f988c345640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9884110f30 con 0x7f9884102570 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f988c345640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9884111230 con 0x7f9884102ef0 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f98898e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884102570 0x7f988419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f98898e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884102570 0x7f988419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:57026/0 (socket says 172.21.15.28:57026) 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.262+0000 7f98898e0640 1 -- 172.21.15.28:0/1284140137 learned_addr learned my addr 172.21.15.28:0/1284140137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f988a0e1640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884100e20 0x7f988419e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9884102ef0 0x7f98841965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f98898e0640 1 -- 172.21.15.28:0/1284140137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9884102ef0 msgr2=0x7f98841965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f98898e0640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9884102ef0 0x7f98841965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f98898e0640 1 -- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884100e20 msgr2=0x7f988419e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f98898e0640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884100e20 0x7f988419e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f98898e0640 1 -- 172.21.15.28:0/1284140137 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9884196eb0 con 0x7f9884102570 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f988a0e1640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884100e20 0x7f988419e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9884102ef0 0x7f98841965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.263+0000 7f98898e0640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884102570 0x7f988419e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f987c009dc0 tx=0x7f987c030920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.264+0000 7f987b7fe640 1 -- 172.21.15.28:0/1284140137 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c047070 con 0x7f9884102570 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.264+0000 7f987b7fe640 1 -- 172.21.15.28:0/1284140137 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f987c004040 con 0x7f9884102570 2024-09-15T22:06:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.264+0000 7f987b7fe640 1 -- 172.21.15.28:0/1284140137 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c034070 con 0x7f9884102570 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.264+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9884197140 con 0x7f9884102570 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.264+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98841b3e40 con 0x7f9884102570 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.268+0000 7f988e71e740 1 Processor -- start 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.272+0000 7f988e71e740 1 -- start start 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.272+0000 7f988e71e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40b7f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.272+0000 7f988e71e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b7f5d0 0x562e40b7fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.272+0000 7f988e71e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.272+0000 7f988e71e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e40ae52f0 con 0x562e40b80b40 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.272+0000 7f988e71e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562e40a8ccf0 con 0x562e40b8a370 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.272+0000 7f988e71e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562e40ae12b0 con 0x562e40b7f5d0 2024-09-15T22:06:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.273+0000 7f987b7fe640 1 -- 172.21.15.28:0/1284140137 <== mon.2 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f987c04a070 con 0x7f9884102570 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.274+0000 7f987b7fe640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98700a35c0 0x7f98700a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.274+0000 7f987b7fe640 1 -- 172.21.15.28:0/1284140137 <== mon.2 v2:172.21.15.174:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f987c0f0e50 con 0x7f9884102570 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b8c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b8c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:57028/0 (socket says 172.21.15.28:57028) 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 -- 172.21.15.28:0/2860934027 learned_addr learned my addr 172.21.15.28:0/2860934027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f988a0e1640 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b7f5d0 0x562e40b7fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 -- 172.21.15.28:0/2860934027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b7f5d0 msgr2=0x562e40b7fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b7f5d0 0x562e40b7fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 -- 172.21.15.28:0/2860934027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 msgr2=0x562e40b7f090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40b7f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f98898e0640 1 -- 172.21.15.28:0/2860934027 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e40a713a0 con 0x562e40b8a370 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.275+0000 7f988a0e1640 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b7f5d0 0x562e40b7fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:10.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.276+0000 7f98898e0640 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b8c760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b9690 tx=0x7f987c0bccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59029ea011ef1c26 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.276+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98700a35c0 0x7f98700a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.276+0000 7f9878dea640 1 -- 172.21.15.28:0/2860934027 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c0bad60 con 0x562e40b8a370 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.277+0000 7f9878dea640 1 -- 172.21.15.28:0/2860934027 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f987c0bafc0 con 0x562e40b8a370 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 -- 172.21.15.28:0/2860934027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 msgr2=0x562e40b8c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b8c760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b9690 tx=0x7f987c0bccc0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98700a35c0 0x7f98700a5a80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f988000a7b0 tx=0x7f9880008040 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 -- 172.21.15.28:0/2860934027 shutdown_connections 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b8c760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b7f5d0 0x562e40b7fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 --2- 172.21.15.28:0/2860934027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40b7f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 -- 172.21.15.28:0/2860934027 >> 172.21.15.28:0/2860934027 conn(0x562e40b48310 msgr2=0x562e40b7bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 -- 172.21.15.28:0/2860934027 shutdown_connections 2024-09-15T22:06:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.278+0000 7f988e71e740 1 -- 172.21.15.28:0/2860934027 wait complete. 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.279+0000 7f988e71e740 1 -- start start 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.281+0000 7f988e71e740 1 Processor -- start 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.282+0000 7f988e71e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b7f5d0 0x562e40b89da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.282+0000 7f988e71e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b80b40 0x562e40b87050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.282+0000 7f988e71e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b877e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.282+0000 7f988e71e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e40a8ccf0 con 0x562e40b7f5d0 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.282+0000 7f988e71e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562e40aa0060 con 0x562e40b8a370 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.282+0000 7f988e71e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562e40b9de40 con 0x562e40b80b40 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.282+0000 7f988e71e740 10 client.0 _ll_register_callbacks cb 0x562e407f38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f98898e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b80b40 0x562e40b87050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f98898e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b80b40 0x562e40b87050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58586/0 (socket says 172.21.15.28:58586) 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f98898e0640 1 -- 172.21.15.28:0/1967071527 learned_addr learned my addr 172.21.15.28:0/1967071527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:10.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b877e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a0e1640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b7f5d0 0x562e40b89da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a8e2640 1 -- 172.21.15.28:0/1967071527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b80b40 msgr2=0x562e40b87050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b80b40 0x562e40b87050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a8e2640 1 -- 172.21.15.28:0/1967071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b7f5d0 msgr2=0x562e40b89da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b7f5d0 0x562e40b89da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a8e2640 1 -- 172.21.15.28:0/1967071527 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e408563c0 con 0x562e40b8a370 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.283+0000 7f988a0e1640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b7f5d0 0x562e40b89da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.284+0000 7f988a8e2640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b877e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f988000ccf0 tx=0x7f9880004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f81021d5f77cf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.284+0000 7f986dffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.284+0000 7f986dffb640 1 -- 172.21.15.28:0/1967071527 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880021020 con 0x562e40b8a370 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.284+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562e40af2860 con 0x562e40b8a370 2024-09-15T22:06:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.284+0000 7f986dffb640 1 -- 172.21.15.28:0/1967071527 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f988001d3f0 con 0x562e40b8a370 2024-09-15T22:06:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.285+0000 7f986dffb640 1 -- 172.21.15.28:0/1967071527 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9880019040 con 0x562e40b8a370 2024-09-15T22:06:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.288+0000 7f988e71e740 10 client.26594 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.288+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e40ba98b0 con 0x562e40b8a370 2024-09-15T22:06:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.289+0000 7f986dffb640 1 -- 172.21.15.28:0/1967071527 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562e40af2860 con 0x562e40b8a370 2024-09-15T22:06:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.289+0000 7f986dffb640 1 -- 172.21.15.28:0/1967071527 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f988002e020 con 0x562e40b8a370 2024-09-15T22:06:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.291+0000 7f988e71e740 10 client.26594 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.291+0000 7f988e71e740 10 client.26594 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.291+0000 7f988e71e740 20 client.26594 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.291+0000 7f988e71e740 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40baad70 0x562e40bad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.291+0000 7f988e71e740 4 client.26594 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.291+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e40aecd70 con 0x562e40baad70 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.298+0000 7f988a0e1640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40baad70 0x562e40bad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.305+0000 7f986dffb640 10 client.26594 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.305+0000 7f988a0e1640 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40baad70 0x562e40bad160 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cb4d308ac42a488f server_cookie=da7c4b19c25f741e in_seq=0 out_seq=0 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.309+0000 7f986dffb640 1 -- 172.21.15.28:0/1967071527 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562e40aecd70 con 0x562e40baad70 2024-09-15T22:06:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.309+0000 7f986dffb640 10 client.26594 handle_command_reply: tid=0 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.310+0000 7f988e71e740 1 client.26594 shutdown 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 msgr2=0x562e40b877e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b877e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f988000ccf0 tx=0x7f9880004750 comp rx=0 tx=0).stop 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 shutdown_connections 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40baad70 0x562e40bad160 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b8a370 0x562e40b877e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b80b40 0x562e40b87050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 --2- 172.21.15.28:0/1967071527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b7f5d0 0x562e40b89da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 >> 172.21.15.28:0/1967071527 conn(0x562e40b48310 msgr2=0x562e40b819c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:10.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.312+0000 7f986dffb640 0 client.26594 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:10.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.318+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 shutdown_connections 2024-09-15T22:06:10.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.318+0000 7f988e71e740 1 -- 172.21.15.28:0/1967071527 wait complete. 2024-09-15T22:06:10.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.319+0000 7f988e71e740 20 client.26594 trim_cache size 0 max 16384 2024-09-15T22:06:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.345+0000 7f988e71e740 1 Processor -- start 2024-09-15T22:06:10.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.346+0000 7f988e71e740 1 -- start start 2024-09-15T22:06:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.346+0000 7f988e71e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40bb2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.346+0000 7f988e71e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40bb2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.346+0000 7f988e71e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b842b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.346+0000 7f988e71e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98841b5920 con 0x562e40b80b40 2024-09-15T22:06:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.346+0000 7f988e71e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98841b7490 con 0x562e40b7f5d0 2024-09-15T22:06:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.346+0000 7f988e71e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98841b5ff0 con 0x562e40b8a370 2024-09-15T22:06:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.348+0000 7f98898e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40bb2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.348+0000 7f98898e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40bb2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42384/0 (socket says 172.21.15.28:42384) 2024-09-15T22:06:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.348+0000 7f98898e0640 1 -- 172.21.15.28:0/871618669 learned_addr learned my addr 172.21.15.28:0/871618669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.348+0000 7f988a0e1640 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40bb2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.348+0000 7f98898e0640 1 -- 172.21.15.28:0/871618669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 msgr2=0x562e40b842b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:06:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.348+0000 7f98898e0640 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b842b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.348+0000 7f98898e0640 1 -- 172.21.15.28:0/871618669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 msgr2=0x562e40bb2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.352+0000 7f98898e0640 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40bb2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.352+0000 7f98898e0640 1 -- 172.21.15.28:0/871618669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e40b7fbd0 con 0x562e40b80b40 2024-09-15T22:06:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.352+0000 7f988a0e1640 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40bb2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.355+0000 7f98898e0640 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40bb2f40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b9690 tx=0x7f987c0b9090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcd1777304c6645 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.359+0000 7f986ffff640 1 -- 172.21.15.28:0/871618669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c0d9020 con 0x562e40b80b40 2024-09-15T22:06:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.359+0000 7f986ffff640 1 -- 172.21.15.28:0/871618669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f987c0bad60 con 0x562e40b80b40 2024-09-15T22:06:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.360+0000 7f988e71e740 1 -- 172.21.15.28:0/871618669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 msgr2=0x562e40bb2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.360+0000 7f988e71e740 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40bb2f40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b9690 tx=0x7f987c0b9090 comp rx=0 tx=0).stop 2024-09-15T22:06:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.363+0000 7f988e71e740 1 -- 172.21.15.28:0/871618669 shutdown_connections 2024-09-15T22:06:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.363+0000 7f988e71e740 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b842b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.363+0000 7f988e71e740 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40bb2f40 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.363+0000 7f988e71e740 1 --2- 172.21.15.28:0/871618669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40bb2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.363+0000 7f988e71e740 1 -- 172.21.15.28:0/871618669 >> 172.21.15.28:0/871618669 conn(0x562e40b48310 msgr2=0x562e40b86140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:10.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.367+0000 7f988e71e740 1 -- 172.21.15.28:0/871618669 shutdown_connections 2024-09-15T22:06:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.369+0000 7f988e71e740 1 -- 172.21.15.28:0/871618669 wait complete. 2024-09-15T22:06:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.369+0000 7f988e71e740 1 -- start start 2024-09-15T22:06:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.369+0000 7f988e71e740 1 Processor -- start 2024-09-15T22:06:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988e71e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40b9f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988e71e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40b9f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988e71e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b9fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988e71e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98841b6dc0 con 0x562e40b80b40 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988e71e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98841b66f0 con 0x562e40b7f5d0 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988e71e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9884197790 con 0x562e40b8a370 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988e71e740 10 client.0 _ll_register_callbacks cb 0x7f9880006e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988a8e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b9fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988a8e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b9fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:58606/0 (socket says 172.21.15.28:58606) 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.370+0000 7f988a8e2640 1 -- 172.21.15.28:0/350895652 learned_addr learned my addr 172.21.15.28:0/350895652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.371+0000 7f988a8e2640 1 -- 172.21.15.28:0/350895652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 msgr2=0x562e40b9f120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.371+0000 7f988a8e2640 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40b9f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.371+0000 7f988a8e2640 1 -- 172.21.15.28:0/350895652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 msgr2=0x562e40b9f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.371+0000 7f988a8e2640 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40b9f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.371+0000 7f988a8e2640 1 -- 172.21.15.28:0/350895652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e40c14c20 con 0x562e40b8a370 2024-09-15T22:06:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.372+0000 7f988a8e2640 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b9fba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9880000c00 tx=0x7f9880026c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c2d425f2e38b97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.372+0000 7f986dffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:06:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.375+0000 7f986dffb640 1 -- 172.21.15.28:0/350895652 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880035020 con 0x562e40b8a370 2024-09-15T22:06:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.375+0000 7f986dffb640 1 -- 172.21.15.28:0/350895652 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f9880004250 con 0x562e40b8a370 2024-09-15T22:06:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.376+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f987c0bad60 con 0x562e40b8a370 2024-09-15T22:06:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.376+0000 7f986dffb640 1 -- 172.21.15.28:0/350895652 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9880021070 con 0x562e40b8a370 2024-09-15T22:06:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.376+0000 7f988e71e740 10 client.26557 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.376+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e40c15f60 con 0x562e40b8a370 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f986dffb640 1 -- 172.21.15.28:0/350895652 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f987c0bad60 con 0x562e40b8a370 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f986dffb640 1 -- 172.21.15.28:0/350895652 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f988002e030 con 0x562e40b8a370 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f988e71e740 10 client.26557 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f988e71e740 10 client.26557 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f988e71e740 20 client.26557 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f988e71e740 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40c16710 0x562e40c18b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f988e71e740 4 client.26557 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e40a69170 con 0x562e40c16710 2024-09-15T22:06:10.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.382+0000 7f988a0e1640 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40c16710 0x562e40c18b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.383+0000 7f986dffb640 10 client.26557 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.384+0000 7f988a0e1640 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40c16710 0x562e40c18b70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1140b2f39846a83c server_cookie=ee143e552dc349ea in_seq=0 out_seq=0 2024-09-15T22:06:10.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.393+0000 7f986dffb640 1 -- 172.21.15.28:0/350895652 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e40a69170 con 0x562e40c16710 2024-09-15T22:06:10.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.393+0000 7f986dffb640 10 client.26557 handle_command_reply: tid=0 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 client.26557 shutdown 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 msgr2=0x562e40b9fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b9fba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9880000c00 tx=0x7f9880026c40 comp rx=0 tx=0).stop 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 shutdown_connections 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x562e40c16710 0x562e40c18b70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e40b8a370 0x562e40b9fba0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e40b80b40 0x562e40b9f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 --2- 172.21.15.28:0/350895652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e40b7f5d0 0x562e40b9f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 >> 172.21.15.28:0/350895652 conn(0x562e40b48310 msgr2=0x562e40b888b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f986dffb640 0 client.26557 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 shutdown_connections 2024-09-15T22:06:10.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 1 -- 172.21.15.28:0/350895652 wait complete. 2024-09-15T22:06:10.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.394+0000 7f988e71e740 20 client.26557 trim_cache size 0 max 16384 2024-09-15T22:06:10.398 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:10.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.395+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98700a35c0 msgr2=0x7f98700a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.395+0000 7f988c345640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98700a35c0 0x7f98700a5a80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f988000a7b0 tx=0x7f9880008040 comp rx=0 tx=0).stop 2024-09-15T22:06:10.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.395+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884102570 msgr2=0x7f988419e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:10.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.395+0000 7f988c345640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884102570 0x7f988419e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f987c009dc0 tx=0x7f987c030920 comp rx=0 tx=0).stop 2024-09-15T22:06:10.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.396+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 shutdown_connections 2024-09-15T22:06:10.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.396+0000 7f988c345640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f98700a35c0 0x7f98700a5a80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.396+0000 7f988c345640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9884102ef0 0x7f98841965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.396+0000 7f988c345640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9884102570 0x7f988419e770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.396+0000 7f988c345640 1 --2- 172.21.15.28:0/1284140137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9884100e20 0x7f988419e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:10.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.396+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 >> 172.21.15.28:0/1284140137 conn(0x7f98840fc7d0 msgr2=0x7f9884103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:10.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.397+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 shutdown_connections 2024-09-15T22:06:10.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:10.397+0000 7f988c345640 1 -- 172.21.15.28:0/1284140137 wait complete. 2024-09-15T22:06:11.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:06:11.593 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:12.044 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:11 smithi028 ceph-mon[30969]: pgmap v271: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:06:12.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:11 smithi203 ceph-mon[35546]: pgmap v271: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:06:12.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:11 smithi174 ceph-mon[35244]: pgmap v271: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:06:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 -- 172.21.15.28:0/3422876787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e4102df0 msgr2=0x7f37e41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 --2- 172.21.15.28:0/3422876787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e4102df0 0x7f37e41031d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009900 tx=0x7f37d402a410 comp rx=0 tx=0).stop 2024-09-15T22:06:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 -- 172.21.15.28:0/3422876787 shutdown_connections 2024-09-15T22:06:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 --2- 172.21.15.28:0/3422876787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37e410c2f0 0x7f37e410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 --2- 172.21.15.28:0/3422876787 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e41037a0 0x7f37e410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 --2- 172.21.15.28:0/3422876787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e4102df0 0x7f37e41031d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 -- 172.21.15.28:0/3422876787 >> 172.21.15.28:0/3422876787 conn(0x7f37e40fc7d0 msgr2=0x7f37e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 -- 172.21.15.28:0/3422876787 shutdown_connections 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.146+0000 7f37e9423640 1 -- 172.21.15.28:0/3422876787 wait complete. 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 Processor -- start 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 -- start start 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e4102df0 0x7f37e41a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e41037a0 0x7f37e41a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37e410c2f0 0x7f37e419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4112110 con 0x7f37e41037a0 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4111f90 con 0x7f37e4102df0 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e9423640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4112290 con 0x7f37e410c2f0 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e41037a0 0x7f37e41a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e41037a0 0x7f37e41a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38472/0 (socket says 172.21.15.28:38472) 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e27fc640 1 -- 172.21.15.28:0/2975016406 learned_addr learned my addr 172.21.15.28:0/2975016406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e37fe640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37e410c2f0 0x7f37e419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e4102df0 0x7f37e41a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e27fc640 1 -- 172.21.15.28:0/2975016406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37e410c2f0 msgr2=0x7f37e419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.147+0000 7f37e27fc640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37e410c2f0 0x7f37e419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37e27fc640 1 -- 172.21.15.28:0/2975016406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e4102df0 msgr2=0x7f37e41a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37e27fc640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e4102df0 0x7f37e41a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37e27fc640 1 -- 172.21.15.28:0/2975016406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e419af50 con 0x7f37e41037a0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37e27fc640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e41037a0 0x7f37e41a0b50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f37d800d740 tx=0x7f37d800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37cbfff640 1 -- 172.21.15.28:0/2975016406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8017070 con 0x7f37e41037a0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37cbfff640 1 -- 172.21.15.28:0/2975016406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d800de50 con 0x7f37e41037a0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e419b240 con 0x7f37e41037a0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37cbfff640 1 -- 172.21.15.28:0/2975016406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d800f6b0 con 0x7f37e41037a0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.148+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e41a7590 con 0x7f37e41037a0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.150+0000 7f37cbfff640 1 -- 172.21.15.28:0/2975016406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f37d801a070 con 0x7f37e41037a0 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 Processor -- start 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 -- start start 2024-09-15T22:06:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a707090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a707a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a714760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe0a66d2f0 con 0x55fe0a712370 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe0a614cf0 con 0x55fe0a7075d0 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fe0a6692b0 con 0x55fe0a708b40 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a707090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a714760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a707090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:50612/0 (socket says 172.21.15.28:50612) 2024-09-15T22:06:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 -- 172.21.15.28:0/1683042783 learned_addr learned my addr 172.21.15.28:0/1683042783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a707a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 -- 172.21.15.28:0/1683042783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 msgr2=0x55fe0a707a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a707a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 -- 172.21.15.28:0/1683042783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 msgr2=0x55fe0a714760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a714760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37e37fe640 1 -- 172.21.15.28:0/1683042783 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe0a5f93a0 con 0x55fe0a708b40 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.151+0000 7f37cbfff640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f37cc0a33b0 0x7f37cc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f37cc0a33b0 0x7f37cc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37e37fe640 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a707090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009f50 tx=0x7f37dc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3466f578d5609907 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37cbfff640 1 -- 172.21.15.28:0/2975016406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f37d80cfbb0 con 0x7f37e41037a0 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37c97fa640 1 -- 172.21.15.28:0/1683042783 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc004d20 con 0x55fe0a708b40 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37c97fa640 1 -- 172.21.15.28:0/1683042783 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc004050 con 0x55fe0a708b40 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37c97fa640 1 -- 172.21.15.28:0/1683042783 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc005240 con 0x55fe0a708b40 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1683042783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 msgr2=0x55fe0a707090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f37cc0a33b0 0x7f37cc0a5870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37d4005940 tx=0x7f37d40058b0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.152+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a707090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009f50 tx=0x7f37dc0118d0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1683042783 shutdown_connections 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a714760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a707a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1683042783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a707090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1683042783 >> 172.21.15.28:0/1683042783 conn(0x55fe0a6d0310 msgr2=0x55fe0a703c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1683042783 shutdown_connections 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1683042783 wait complete. 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- start start 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 Processor -- start 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a711cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a70ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a70f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe0a7150c0 con 0x55fe0a712370 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe0a628060 con 0x55fe0a7075d0 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.153+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fe0a715240 con 0x55fe0a708b40 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37eb7fc740 10 client.0 _ll_register_callbacks cb 0x55fe0a37b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a70f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e27fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a70ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a70f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38500/0 (socket says 172.21.15.28:38500) 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e37fe640 1 -- 172.21.15.28:0/3781046908 learned_addr learned my addr 172.21.15.28:0/3781046908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a711cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e27fc640 1 -- 172.21.15.28:0/3781046908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 msgr2=0x55fe0a711cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e27fc640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a711cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e27fc640 1 -- 172.21.15.28:0/3781046908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 msgr2=0x55fe0a70f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e27fc640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a70f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e27fc640 1 -- 172.21.15.28:0/3781046908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe0a3de3c0 con 0x55fe0a708b40 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a711cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e37fe640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a70f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37e27fc640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a70ef60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f37d80cce90 tx=0x7f37d8097d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae5f386da1691834 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37c6ff5640 1 -- 172.21.15.28:0/3781046908 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8021070 con 0x55fe0a708b40 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37c6ff5640 1 -- 172.21.15.28:0/3781046908 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f37d80cd110 con 0x55fe0a708b40 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37dc004050 con 0x55fe0a708b40 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37c6ff5640 1 -- 172.21.15.28:0/3781046908 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f37d8009c30 con 0x55fe0a708b40 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.154+0000 7f37eb7fc740 10 client.26569 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fe0a720ab0 con 0x55fe0a708b40 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37c6ff5640 1 -- 172.21.15.28:0/3781046908 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc004050 con 0x55fe0a708b40 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37c6ff5640 1 -- 172.21.15.28:0/3781046908 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f37d809ae70 con 0x55fe0a708b40 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37eb7fc740 10 client.26569 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37eb7fc740 10 client.26569 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37eb7fc740 20 client.26569 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a721f70 0x55fe0a724360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37eb7fc740 4 client.26569 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe0a67a860 con 0x55fe0a721f70 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.155+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a721f70 0x55fe0a724360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a721f70 0x55fe0a724360 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=882834386e5a590a server_cookie=d1f1668637419c92 in_seq=0 out_seq=0 2024-09-15T22:06:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37c6ff5640 10 client.26569 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37c6ff5640 1 -- 172.21.15.28:0/3781046908 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe0a67a860 con 0x55fe0a721f70 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37c6ff5640 10 client.26569 handle_command_reply: tid=0 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 client.26569 shutdown 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 msgr2=0x55fe0a70ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a70ef60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f37d80cce90 tx=0x7f37d8097d80 comp rx=0 tx=0).stop 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 shutdown_connections 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a721f70 0x55fe0a724360 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a712370 0x55fe0a70f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37c6ff5640 0 client.26569 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a70ef60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.156+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/3781046908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a711cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.157+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 >> 172.21.15.28:0/3781046908 conn(0x55fe0a6d0310 msgr2=0x55fe0a709960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.157+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 shutdown_connections 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.157+0000 7f37eb7fc740 1 -- 172.21.15.28:0/3781046908 wait complete. 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.157+0000 7f37eb7fc740 20 client.26569 trim_cache size 0 max 16384 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.173+0000 7f37eb7fc740 1 Processor -- start 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.174+0000 7f37eb7fc740 1 -- start start 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.175+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a7075d0 0x55fe0a71e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.175+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a729c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.175+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a712370 0x55fe0a72a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.175+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41b13e0 con 0x55fe0a7075d0 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.175+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a712370 0x55fe0a72a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.175+0000 7f37e37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a712370 0x55fe0a72a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:33912/0 (socket says 172.21.15.28:33912) 2024-09-15T22:06:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.175+0000 7f37e37fe640 1 -- 172.21.15.28:0/282292889 learned_addr learned my addr 172.21.15.28:0/282292889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37eb7fc740 1 -- 172.21.15.28:0/282292889 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41b2f50 con 0x55fe0a712370 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e27fc640 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a729c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37eb7fc740 1 -- 172.21.15.28:0/282292889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41b1ab0 con 0x55fe0a708b40 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a7075d0 0x55fe0a71e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e37fe640 1 -- 172.21.15.28:0/282292889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 msgr2=0x55fe0a729c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e37fe640 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a729c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e37fe640 1 -- 172.21.15.28:0/282292889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a7075d0 msgr2=0x55fe0a71e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e37fe640 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a7075d0 0x55fe0a71e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e37fe640 1 -- 172.21.15.28:0/282292889 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe0a70a120 con 0x55fe0a712370 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.176+0000 7f37e37fe640 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a712370 0x55fe0a72a170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f37dc012fd0 tx=0x7f37dc0097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f89dbd04956f66d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.177+0000 7f37c8ff9640 1 -- 172.21.15.28:0/282292889 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc004590 con 0x55fe0a712370 2024-09-15T22:06:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.177+0000 7f37c8ff9640 1 -- 172.21.15.28:0/282292889 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc004820 con 0x55fe0a712370 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.177+0000 7f37c8ff9640 1 -- 172.21.15.28:0/282292889 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc005530 con 0x55fe0a712370 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 -- 172.21.15.28:0/282292889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a712370 msgr2=0x55fe0a72a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a712370 0x55fe0a72a170 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f37dc012fd0 tx=0x7f37dc0097b0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 -- 172.21.15.28:0/282292889 shutdown_connections 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a712370 0x55fe0a72a170 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a708b40 0x55fe0a729c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/282292889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a7075d0 0x55fe0a71e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 -- 172.21.15.28:0/282292889 >> 172.21.15.28:0/282292889 conn(0x55fe0a6d0310 msgr2=0x55fe0a70d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 -- 172.21.15.28:0/282292889 shutdown_connections 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 -- 172.21.15.28:0/282292889 wait complete. 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 -- start start 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.178+0000 7f37eb7fc740 1 Processor -- start 2024-09-15T22:06:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a71aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a708b40 0x55fe0a715e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37eb7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a712370 0x55fe0a716350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41b2880 con 0x55fe0a708b40 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41b21b0 con 0x55fe0a7075d0 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37eb7fc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41af3e0 con 0x55fe0a712370 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37eb7fc740 10 client.0 _ll_register_callbacks cb 0x7f37d80d3350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a708b40 0x55fe0a715e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a71aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a708b40 0x55fe0a715e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38516/0 (socket says 172.21.15.28:38516) 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e27fc640 1 -- 172.21.15.28:0/1220168310 learned_addr learned my addr 172.21.15.28:0/1220168310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e37fe640 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a712370 0x55fe0a716350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e2ffd640 1 -- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a712370 msgr2=0x55fe0a716350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a712370 0x55fe0a716350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e2ffd640 1 -- 172.21.15.28:0/1220168310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a708b40 msgr2=0x55fe0a715e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a708b40 0x55fe0a715e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e2ffd640 1 -- 172.21.15.28:0/1220168310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe0a716a40 con 0x55fe0a7075d0 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e27fc640 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a708b40 0x55fe0a715e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37e2ffd640 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a71aae0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009230 tx=0x7f37d403e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e44210d2c9b8509e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.179+0000 7f37c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37c6ff5640 1 -- 172.21.15.28:0/1220168310 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4035040 con 0x55fe0a7075d0 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37dc004820 con 0x55fe0a7075d0 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37c6ff5640 1 -- 172.21.15.28:0/1220168310 <== mon.2 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f37d4005040 con 0x55fe0a7075d0 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37c6ff5640 1 -- 172.21.15.28:0/1220168310 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f37d4005240 con 0x55fe0a7075d0 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 10 client.26615 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fe0a78cb20 con 0x55fe0a7075d0 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37c6ff5640 1 -- 172.21.15.28:0/1220168310 <== mon.2 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc004820 con 0x55fe0a7075d0 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37c6ff5640 1 -- 172.21.15.28:0/1220168310 <== mon.2 v2:172.21.15.174:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f37d4003680 con 0x55fe0a7075d0 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 10 client.26615 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 10 client.26615 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 20 client.26615 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a78d1a0 0x55fe0a78f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 4 client.26615 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe0a674d70 con 0x55fe0a78d1a0 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.180+0000 7f37e27fc640 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a78d1a0 0x55fe0a78f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37c6ff5640 10 client.26615 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37e27fc640 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a78d1a0 0x55fe0a78f590 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f34bea7f2cf49168 server_cookie=72849f59f7e2dbba in_seq=0 out_seq=0 2024-09-15T22:06:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37c6ff5640 1 -- 172.21.15.28:0/1220168310 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe0a674d70 con 0x55fe0a78d1a0 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37c6ff5640 10 client.26615 handle_command_reply: tid=0 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 client.26615 shutdown 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 msgr2=0x55fe0a71aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a71aae0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009230 tx=0x7f37d403e070 comp rx=0 tx=0).stop 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 shutdown_connections 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55fe0a78d1a0 0x55fe0a78f590 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe0a712370 0x55fe0a716350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37c6ff5640 0 client.26615 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe0a708b40 0x55fe0a715e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 --2- 172.21.15.28:0/1220168310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe0a7075d0 0x55fe0a71aae0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 >> 172.21.15.28:0/1220168310 conn(0x55fe0a6d0310 msgr2=0x55fe0a709110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 shutdown_connections 2024-09-15T22:06:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 1 -- 172.21.15.28:0/1220168310 wait complete. 2024-09-15T22:06:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.181+0000 7f37eb7fc740 20 client.26615 trim_cache size 0 max 16384 2024-09-15T22:06:13.083 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f37cc0a33b0 msgr2=0x7f37cc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f37cc0a33b0 0x7f37cc0a5870 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37d4005940 tx=0x7f37d40058b0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e41037a0 msgr2=0x7f37e41a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e41037a0 0x7f37e41a0b50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f37d800d740 tx=0x7f37d800dc00 comp rx=0 tx=0).stop 2024-09-15T22:06:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 shutdown_connections 2024-09-15T22:06:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f37cc0a33b0 0x7f37cc0a5870 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37e410c2f0 0x7f37e419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37e41037a0 0x7f37e41a0b50 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 --2- 172.21.15.28:0/2975016406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37e4102df0 0x7f37e41a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 >> 172.21.15.28:0/2975016406 conn(0x7f37e40fc7d0 msgr2=0x7f37e410d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 shutdown_connections 2024-09-15T22:06:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:12.182+0000 7f37e9423640 1 -- 172.21.15.28:0/2975016406 wait complete. 2024-09-15T22:06:13.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:06:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:14.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:13 smithi028 ceph-mon[30969]: pgmap v272: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:14.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:13 smithi174 ceph-mon[35244]: pgmap v272: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:14.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:13 smithi203 ceph-mon[35546]: pgmap v272: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 -- 172.21.15.28:0/1700011391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 msgr2=0x7fadb8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 --2- 172.21.15.28:0/1700011391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb8100ee0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fada0009930 tx=0x7fada002a420 comp rx=0 tx=0).stop 2024-09-15T22:06:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 -- 172.21.15.28:0/1700011391 shutdown_connections 2024-09-15T22:06:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 --2- 172.21.15.28:0/1700011391 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fadb8101420 0x7fadb810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 --2- 172.21.15.28:0/1700011391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb8100ee0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 --2- 172.21.15.28:0/1700011391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb8106a80 0x7fadb8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 -- 172.21.15.28:0/1700011391 >> 172.21.15.28:0/1700011391 conn(0x7fadb80fc7a0 msgr2=0x7fadb80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 -- 172.21.15.28:0/1700011391 shutdown_connections 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.554+0000 7fadbe346640 1 -- 172.21.15.28:0/1700011391 wait complete. 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 Processor -- start 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 -- start start 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb810e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb8101420 0x7fadb810e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fadb8106a80 0x7fadb810ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadb8112020 con 0x7fadb8100a80 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fadb8111ea0 con 0x7fadb8106a80 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbe346640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fadb81121a0 con 0x7fadb8101420 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb810e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb810e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38534/0 (socket says 172.21.15.28:38534) 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadb7fff640 1 -- 172.21.15.28:0/3264439436 learned_addr learned my addr 172.21.15.28:0/3264439436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fadb8106a80 0x7fadb810ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadaffff640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb8101420 0x7fadb810e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.555+0000 7fadb7fff640 1 -- 172.21.15.28:0/3264439436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb8101420 msgr2=0x7fadb810e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb7fff640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb8101420 0x7fadb810e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb7fff640 1 -- 172.21.15.28:0/3264439436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fadb8106a80 msgr2=0x7fadb810ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb7fff640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fadb8106a80 0x7fadb810ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb7fff640 1 -- 172.21.15.28:0/3264439436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb81a6a80 con 0x7fadb8100a80 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb7fff640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb810e3e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fada800b4a0 tx=0x7fada800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb5ffb640 1 -- 172.21.15.28:0/3264439436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8014070 con 0x7fadb8100a80 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb81a6d70 con 0x7fadb8100a80 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb5ffb640 1 -- 172.21.15.28:0/3264439436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fada8010af0 con 0x7fadb8100a80 2024-09-15T22:06:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadb5ffb640 1 -- 172.21.15.28:0/3264439436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8017d10 con 0x7fadb8100a80 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.556+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb81a71d0 con 0x7fadb8100a80 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.558+0000 7fadb5ffb640 1 -- 172.21.15.28:0/3264439436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fada8017070 con 0x7fadb8100a80 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.558+0000 7fadc071f740 1 Processor -- start 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.558+0000 7fadc071f740 1 -- start start 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadc071f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadc071f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadc071f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadc071f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba0a42b2f0 con 0x55ba0a4c55d0 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadc071f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ba0a3d2cf0 con 0x55ba0a4d0370 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadc071f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ba0a4272b0 con 0x55ba0a4c6b40 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadbc8e3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38550/0 (socket says 172.21.15.28:38550) 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 -- 172.21.15.28:0/1298310147 learned_addr learned my addr 172.21.15.28:0/1298310147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb5ffb640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fada40a33b0 0x7fada40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadaffff640 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fada40a33b0 0x7fada40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb5ffb640 1 -- 172.21.15.28:0/3264439436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7fada80cf4a0 con 0x7fadb8100a80 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 -- 172.21.15.28:0/1298310147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 msgr2=0x55ba0a4c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 -- 172.21.15.28:0/1298310147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4d0370 msgr2=0x55ba0a4d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadb7fff640 1 -- 172.21.15.28:0/1298310147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba0a3b73a0 con 0x55ba0a4c55d0 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.559+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.560+0000 7fadb7fff640 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4c5a10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fada80978e0 tx=0x7fada80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9b045f0e0ca0e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.560+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fada40a33b0 0x7fada40a5870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009fd0 tx=0x7fadb0004a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.560+0000 7fadae7fc640 1 -- 172.21.15.28:0/1298310147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada801e070 con 0x55ba0a4c55d0 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.560+0000 7fadae7fc640 1 -- 172.21.15.28:0/1298310147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fada809bb80 con 0x55ba0a4c55d0 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.560+0000 7fadae7fc640 1 -- 172.21.15.28:0/1298310147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada80cca60 con 0x55ba0a4c55d0 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.560+0000 7fadc071f740 1 -- 172.21.15.28:0/1298310147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c55d0 msgr2=0x55ba0a4c5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.560+0000 7fadc071f740 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4c5a10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fada80978e0 tx=0x7fada80047d0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- 172.21.15.28:0/1298310147 shutdown_connections 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4c5a10 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 --2- 172.21.15.28:0/1298310147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- 172.21.15.28:0/1298310147 >> 172.21.15.28:0/1298310147 conn(0x55ba0a48e310 msgr2=0x55ba0a4c1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- 172.21.15.28:0/1298310147 shutdown_connections 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- 172.21.15.28:0/1298310147 wait complete. 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- start start 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 Processor -- start 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4d0370 0x55ba0a4cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba0a4d3070 con 0x55ba0a4c6b40 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ba0a3e6060 con 0x55ba0a4c55d0 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ba0a4d31f0 con 0x55ba0a4d0370 2024-09-15T22:06:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadc071f740 10 client.0 _ll_register_callbacks cb 0x55ba0a1398c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.561+0000 7fadaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38562/0 (socket says 172.21.15.28:38562) 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 -- 172.21.15.28:0/380986427 learned_addr learned my addr 172.21.15.28:0/380986427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadb7fff640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 -- 172.21.15.28:0/380986427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4d0370 msgr2=0x55ba0a4cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4d0370 0x55ba0a4cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4d0370 0x55ba0a4cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 -- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 msgr2=0x55ba0a4cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 -- 172.21.15.28:0/380986427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba0a19c3c0 con 0x55ba0a4c6b40 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4d0370 0x55ba0a4cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadb7fff640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadaffff640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cd090 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fada0031550 tx=0x7fada0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d191826209dc2aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fad977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fad977fe640 1 -- 172.21.15.28:0/380986427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada0035040 con 0x55ba0a4c6b40 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fad977fe640 1 -- 172.21.15.28:0/380986427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7fada0005020 con 0x55ba0a4c6b40 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fada809bb80 con 0x55ba0a4c6b40 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fad977fe640 1 -- 172.21.15.28:0/380986427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fada004e050 con 0x55ba0a4c6b40 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadc071f740 10 client.17217 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.562+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ba0a4deba0 con 0x55ba0a4c6b40 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fad977fe640 1 -- 172.21.15.28:0/380986427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fada809bb80 con 0x55ba0a4c6b40 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fad977fe640 1 -- 172.21.15.28:0/380986427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fada00035f0 con 0x55ba0a4c6b40 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fadc071f740 10 client.17217 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fadc071f740 10 client.17217 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fadc071f740 20 client.17217 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fadc071f740 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a4e0060 0x55ba0a4e2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fadc071f740 4 client.17217 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba0a438860 con 0x55ba0a4e0060 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.563+0000 7fadb7fff640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a4e0060 0x55ba0a4e2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.564+0000 7fad977fe640 10 client.17217 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.564+0000 7fadb7fff640 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a4e0060 0x55ba0a4e2490 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb0100be91af862b server_cookie=53b073e07c4ebcd6 in_seq=0 out_seq=0 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fad977fe640 1 -- 172.21.15.28:0/380986427 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ba0a438860 con 0x55ba0a4e0060 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fad977fe640 10 client.17217 handle_command_reply: tid=0 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 client.17217 shutdown 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c6b40 msgr2=0x55ba0a4cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cd090 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fada0031550 tx=0x7fada0004270 comp rx=0 tx=0).stop 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 shutdown_connections 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a4e0060 0x55ba0a4e2490 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4d0370 0x55ba0a4cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cd090 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 --2- 172.21.15.28:0/380986427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fad977fe640 0 client.17217 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 >> 172.21.15.28:0/380986427 conn(0x55ba0a48e310 msgr2=0x55ba0a4c7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 shutdown_connections 2024-09-15T22:06:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 1 -- 172.21.15.28:0/380986427 wait complete. 2024-09-15T22:06:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.565+0000 7fadc071f740 20 client.17217 trim_cache size 0 max 16384 2024-09-15T22:06:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.578+0000 7fadc071f740 1 Processor -- start 2024-09-15T22:06:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.578+0000 7fadc071f740 1 -- start start 2024-09-15T22:06:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.578+0000 7fadc071f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4dc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadc071f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadc071f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadc071f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadb81b1470 con 0x55ba0a4d0370 2024-09-15T22:06:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadc071f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fadb81b2fe0 con 0x55ba0a4c55d0 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadc071f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fadb81b1b40 con 0x55ba0a4c6b40 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4e7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38574/0 (socket says 172.21.15.28:38574) 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 -- 172.21.15.28:0/2511834710 learned_addr learned my addr 172.21.15.28:0/2511834710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadaffff640 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadb7fff640 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4dc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 -- 172.21.15.28:0/2511834710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 msgr2=0x55ba0a4cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 -- 172.21.15.28:0/2511834710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 msgr2=0x55ba0a4dc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4dc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 -- 172.21.15.28:0/2511834710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba0a4dc850 con 0x55ba0a4d0370 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4e7d60 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fadb000cb60 tx=0x7fadb00098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797bdc397f80d9fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadadffb640 1 -- 172.21.15.28:0/2511834710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0015070 con 0x55ba0a4d0370 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadadffb640 1 -- 172.21.15.28:0/2511834710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb0016040 con 0x55ba0a4d0370 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.579+0000 7fadadffb640 1 -- 172.21.15.28:0/2511834710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb001ddf0 con 0x55ba0a4d0370 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 -- 172.21.15.28:0/2511834710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 msgr2=0x55ba0a4e7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4e7d60 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fadb000cb60 tx=0x7fadb00098d0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 -- 172.21.15.28:0/2511834710 shutdown_connections 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4e7d60 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 --2- 172.21.15.28:0/2511834710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4dc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 -- 172.21.15.28:0/2511834710 >> 172.21.15.28:0/2511834710 conn(0x55ba0a48e310 msgr2=0x55ba0a4cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 -- 172.21.15.28:0/2511834710 shutdown_connections 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 -- 172.21.15.28:0/2511834710 wait complete. 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.580+0000 7fadc071f740 1 -- start start 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 1 Processor -- start 2024-09-15T22:06:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4d8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4d4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadb81b2910 con 0x55ba0a4d0370 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fadb81b2240 con 0x55ba0a4c55d0 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fadb81af470 con 0x55ba0a4c6b40 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadbc8e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadc071f740 10 client.0 _ll_register_callbacks cb 0x7fada0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadbc8e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38580/0 (socket says 172.21.15.28:38580) 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadbc8e3640 1 -- 172.21.15.28:0/3949237912 learned_addr learned my addr 172.21.15.28:0/3949237912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadaffff640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4d4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.581+0000 7fadb7fff640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4d8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadbc8e3640 1 -- 172.21.15.28:0/3949237912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 msgr2=0x55ba0a4d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4d4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadbc8e3640 1 -- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 msgr2=0x55ba0a4d8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4d8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadbc8e3640 1 -- 172.21.15.28:0/3949237912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba0a4d4e40 con 0x55ba0a4d0370 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadb7fff640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4d8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadbc8e3640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d4610 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fadb000a7d0 tx=0x7fadb0009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7650c07d05d95cbd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fad977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fad977fe640 1 -- 172.21.15.28:0/3949237912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0015040 con 0x55ba0a4d0370 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadb0016040 con 0x55ba0a4d0370 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fad977fe640 1 -- 172.21.15.28:0/3949237912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7fadb000a900 con 0x55ba0a4d0370 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fad977fe640 1 -- 172.21.15.28:0/3949237912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fadb000ab00 con 0x55ba0a4d0370 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadc071f740 10 client.17223 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ba0a54ae40 con 0x55ba0a4d0370 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.582+0000 7fad977fe640 1 -- 172.21.15.28:0/3949237912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb0016040 con 0x55ba0a4d0370 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fad977fe640 1 -- 172.21.15.28:0/3949237912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fadb0028c60 con 0x55ba0a4d0370 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadc071f740 10 client.17223 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadc071f740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadc071f740 20 client.17223 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadc071f740 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a54b500 0x55ba0a54d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadc071f740 4 client.17223 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba0a432d70 con 0x55ba0a54b500 2024-09-15T22:06:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadb7fff640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a54b500 0x55ba0a54d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadb7fff640 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a54b500 0x55ba0a54d920 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a224518d7448328 server_cookie=4494572f3b7f8d87 in_seq=0 out_seq=0 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fad977fe640 10 client.17223 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fad977fe640 1 -- 172.21.15.28:0/3949237912 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba0a432d70 con 0x55ba0a54b500 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fad977fe640 10 client.17223 handle_command_reply: tid=0 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.583+0000 7fadc071f740 1 client.17223 shutdown 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 msgr2=0x55ba0a4d4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d4610 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fadb000a7d0 tx=0x7fadb0009830 comp rx=0 tx=0).stop 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 shutdown_connections 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55ba0a54b500 0x55ba0a54d920 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba0a4d0370 0x55ba0a4d4610 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0a4c6b40 0x55ba0a4d4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fad977fe640 0 client.17223 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 --2- 172.21.15.28:0/3949237912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ba0a4c55d0 0x55ba0a4d8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 >> 172.21.15.28:0/3949237912 conn(0x55ba0a48e310 msgr2=0x55ba0a4cc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 shutdown_connections 2024-09-15T22:06:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 1 -- 172.21.15.28:0/3949237912 wait complete. 2024-09-15T22:06:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.584+0000 7fadc071f740 20 client.17223 trim_cache size 0 max 16384 2024-09-15T22:06:14.595 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fada40a33b0 msgr2=0x7fada40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fada40a33b0 0x7fada40a5870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009fd0 tx=0x7fadb0004a40 comp rx=0 tx=0).stop 2024-09-15T22:06:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 msgr2=0x7fadb810e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb810e3e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fada800b4a0 tx=0x7fada800b960 comp rx=0 tx=0).stop 2024-09-15T22:06:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 shutdown_connections 2024-09-15T22:06:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fada40a33b0 0x7fada40a5870 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fadb8106a80 0x7fadb810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fadb8101420 0x7fadb810e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 --2- 172.21.15.28:0/3264439436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadb8100a80 0x7fadb810e3e0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 >> 172.21.15.28:0/3264439436 conn(0x7fadb80fc7a0 msgr2=0x7fadb80fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 shutdown_connections 2024-09-15T22:06:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:14.585+0000 7fadbe346640 1 -- 172.21.15.28:0/3264439436 wait complete. 2024-09-15T22:06:16.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:15 smithi174 ceph-mon[35244]: pgmap v273: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:16.371 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T22:06:16.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T22:06:16.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:15 smithi028 ceph-mon[30969]: pgmap v273: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:16.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:15 smithi203 ceph-mon[35546]: pgmap v273: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 -- 172.21.15.28:0/2916491127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 msgr2=0x7fc3081067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 --2- 172.21.15.28:0/2916491127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc3081067e0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009900 tx=0x7fc2f002a410 comp rx=0 tx=0).stop 2024-09-15T22:06:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 -- 172.21.15.28:0/2916491127 shutdown_connections 2024-09-15T22:06:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 --2- 172.21.15.28:0/2916491127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc308106f10 0x7fc30810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 --2- 172.21.15.28:0/2916491127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc3081067e0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 --2- 172.21.15.28:0/2916491127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc308101de0 0x7fc3081021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 -- 172.21.15.28:0/2916491127 >> 172.21.15.28:0/2916491127 conn(0x7fc3080fd8c0 msgr2=0x7fc3080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.202+0000 7fc30d472640 1 -- 172.21.15.28:0/2916491127 shutdown_connections 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 -- 172.21.15.28:0/2916491127 wait complete. 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 Processor -- start 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 -- start start 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc308101de0 0x7fc3080744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc308074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc308106f10 0x7fc308074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc308111fc0 con 0x7fc308102790 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc308111e40 con 0x7fc308101de0 2024-09-15T22:06:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.203+0000 7fc30d472640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc308112140 con 0x7fc308106f10 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc308074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc308074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38596/0 (socket says 172.21.15.28:38596) 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 -- 172.21.15.28:0/151595732 learned_addr learned my addr 172.21.15.28:0/151595732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc306ffd640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc308101de0 0x7fc3080744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc3077fe640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc308106f10 0x7fc308074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 -- 172.21.15.28:0/151595732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc308106f10 msgr2=0x7fc308074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc308106f10 0x7fc308074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 -- 172.21.15.28:0/151595732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc308101de0 msgr2=0x7fc3080744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc308101de0 0x7fc3080744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 -- 172.21.15.28:0/151595732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3081a6990 con 0x7fc308102790 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc2fffff640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc308074a30 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0004770 tx=0x7fc2f0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc304ff9640 1 -- 172.21.15.28:0/151595732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f003f070 con 0x7fc308102790 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc304ff9640 1 -- 172.21.15.28:0/151595732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f0031ca0 con 0x7fc308102790 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3081a6c20 con 0x7fc308102790 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc304ff9640 1 -- 172.21.15.28:0/151595732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0035030 con 0x7fc308102790 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.204+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3081a7150 con 0x7fc308102790 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.206+0000 7fc304ff9640 1 -- 172.21.15.28:0/151595732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fc2f0042070 con 0x7fc308102790 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 Processor -- start 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 -- start start 2024-09-15T22:06:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4253f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf425e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf425ed40 0x55abf4261130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc304ff9640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc2f40a3430 0x7fc2f40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55abf41b9280 con 0x55abf4254480 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55abf4160b10 con 0x55abf425ed40 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.207+0000 7fc30f84b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55abf41b5240 con 0x55abf4253b60 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf425e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc3077fe640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc2f40a3430 0x7fc2f40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf425e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38598/0 (socket says 172.21.15.28:38598) 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 -- 172.21.15.28:0/4263691627 learned_addr learned my addr 172.21.15.28:0/4263691627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc304ff9640 1 -- 172.21.15.28:0/151595732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7fc2f00f08a0 con 0x7fc308102790 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc2fffff640 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf425ed40 0x55abf4261130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc3077fe640 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4253f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 -- 172.21.15.28:0/4263691627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 msgr2=0x55abf4253f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4253f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 -- 172.21.15.28:0/4263691627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf425ed40 msgr2=0x55abf4261130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf425ed40 0x55abf4261130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 -- 172.21.15.28:0/4263691627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abf416d1f0 con 0x55abf4254480 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc2fffff640 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf425ed40 0x55abf4261130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc3077fe640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc2f40a3430 0x7fc2f40a58f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc300005c40 tx=0x7fc300009c90 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc306ffd640 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf425e800 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800bd10 tx=0x7fc2f80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12751b0942e558c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc2fd7fa640 1 -- 172.21.15.28:0/4263691627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8004030 con 0x55abf4254480 2024-09-15T22:06:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc2fd7fa640 1 -- 172.21.15.28:0/4263691627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f80041d0 con 0x55abf4254480 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.208+0000 7fc2fd7fa640 1 -- 172.21.15.28:0/4263691627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8011840 con 0x55abf4254480 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 -- 172.21.15.28:0/4263691627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 msgr2=0x55abf425e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf425e800 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800bd10 tx=0x7fc2f80107a0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 -- 172.21.15.28:0/4263691627 shutdown_connections 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf425ed40 0x55abf4261130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf425e800 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 --2- 172.21.15.28:0/4263691627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4253f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 -- 172.21.15.28:0/4263691627 >> 172.21.15.28:0/4263691627 conn(0x55abf42500f0 msgr2=0x55abf42504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 -- 172.21.15.28:0/4263691627 shutdown_connections 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.209+0000 7fc30f84b740 1 -- 172.21.15.28:0/4263691627 wait complete. 2024-09-15T22:06:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 -- start start 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 Processor -- start 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4253b60 0x55abf425bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf425c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf425c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55abf4261910 con 0x55abf4253b60 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55abf4173f10 con 0x55abf4254480 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55abf4261a90 con 0x55abf425ed40 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc30f84b740 10 client.0 _ll_register_callbacks cb 0x55abf413ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc306ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4253b60 0x55abf425bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc306ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4253b60 0x55abf425bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38602/0 (socket says 172.21.15.28:38602) 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc306ffd640 1 -- 172.21.15.28:0/1153583430 learned_addr learned my addr 172.21.15.28:0/1153583430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc2fffff640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf425c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.210+0000 7fc3077fe640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf425c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc306ffd640 1 -- 172.21.15.28:0/1153583430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 msgr2=0x55abf425c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc306ffd640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf425c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc306ffd640 1 -- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 msgr2=0x55abf425c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc306ffd640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf425c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc306ffd640 1 -- 172.21.15.28:0/1153583430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abf3f2a6a0 con 0x55abf4253b60 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc3077fe640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf425c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc306ffd640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4253b60 0x55abf425bc00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005880 tx=0x7fc2f80058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945931fd065f4dd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc2ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/1153583430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8029060 con 0x55abf4253b60 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/1153583430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7fc2f8004380 con 0x55abf4253b60 2024-09-15T22:06:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f80041d0 con 0x55abf4253b60 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/1153583430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc2f8030050 con 0x55abf4253b60 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc30f84b740 10 client.17235 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55abf426d4f0 con 0x55abf4253b60 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.211+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/1153583430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f80041d0 con 0x55abf4253b60 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/1153583430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc2f80046c0 con 0x55abf4253b60 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc30f84b740 10 client.17235 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc30f84b740 10 client.17235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc30f84b740 20 client.17235 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf426e8a0 0x55abf4270c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc30f84b740 4 client.17235 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abf41c67f0 con 0x55abf426e8a0 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc2fffff640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf426e8a0 0x55abf4270c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc2ee7fc640 10 client.17235 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.212+0000 7fc2fffff640 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf426e8a0 0x55abf4270c90 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=882a4bdd810e9dbe server_cookie=f42b7e96d65da93e in_seq=0 out_seq=0 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/1153583430 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55abf41c67f0 con 0x55abf426e8a0 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc2ee7fc640 10 client.17235 handle_command_reply: tid=0 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 client.17235 shutdown 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4253b60 msgr2=0x55abf425bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4253b60 0x55abf425bc00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005880 tx=0x7fc2f80058b0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 shutdown_connections 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf426e8a0 0x55abf4270c90 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf425c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf425c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc2ee7fc640 0 client.17235 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1153583430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4253b60 0x55abf425bc00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 >> 172.21.15.28:0/1153583430 conn(0x55abf42500f0 msgr2=0x55abf4255dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 shutdown_connections 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 1 -- 172.21.15.28:0/1153583430 wait complete. 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.213+0000 7fc30f84b740 20 client.17235 trim_cache size 0 max 16384 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 Processor -- start 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 -- start start 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4253b60 0x55abf42565c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf4273cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf42741f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3081b12f0 con 0x55abf4254480 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3081b2e60 con 0x55abf4253b60 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc30f84b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3081b19c0 con 0x55abf425ed40 2024-09-15T22:06:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc2fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf4273cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc2fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf4273cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38604/0 (socket says 172.21.15.28:38604) 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.231+0000 7fc2fffff640 1 -- 172.21.15.28:0/1129522614 learned_addr learned my addr 172.21.15.28:0/1129522614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc306ffd640 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4253b60 0x55abf42565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fffff640 1 -- 172.21.15.28:0/1129522614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 msgr2=0x55abf42741f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fffff640 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf42741f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fffff640 1 -- 172.21.15.28:0/1129522614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4253b60 msgr2=0x55abf42565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fffff640 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4253b60 0x55abf42565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fffff640 1 -- 172.21.15.28:0/1129522614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abf425b2d0 con 0x55abf4254480 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc306ffd640 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4253b60 0x55abf42565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fffff640 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf4273cb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00ee1e0 tx=0x7fc2f004b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95743819ac782d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fcff9640 1 -- 172.21.15.28:0/1129522614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0049040 con 0x55abf4254480 2024-09-15T22:06:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fcff9640 1 -- 172.21.15.28:0/1129522614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f0031e50 con 0x55abf4254480 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.232+0000 7fc2fcff9640 1 -- 172.21.15.28:0/1129522614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00ede60 con 0x55abf4254480 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 -- 172.21.15.28:0/1129522614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 msgr2=0x55abf4273cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf4273cb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00ee1e0 tx=0x7fc2f004b000 comp rx=0 tx=0).stop 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 -- 172.21.15.28:0/1129522614 shutdown_connections 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf425ed40 0x55abf42741f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf4254480 0x55abf4273cb0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 --2- 172.21.15.28:0/1129522614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4253b60 0x55abf42565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 -- 172.21.15.28:0/1129522614 >> 172.21.15.28:0/1129522614 conn(0x55abf42500f0 msgr2=0x55abf425abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 -- 172.21.15.28:0/1129522614 shutdown_connections 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 -- 172.21.15.28:0/1129522614 wait complete. 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 -- start start 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 Processor -- start 2024-09-15T22:06:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4273e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf42743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf425ed40 0x55abf4272270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.233+0000 7fc30f84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3081b2790 con 0x55abf425ed40 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc30f84b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3081b20c0 con 0x55abf4254480 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc30f84b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3081af2f0 con 0x55abf4253b60 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc30f84b740 10 client.0 _ll_register_callbacks cb 0x7fc2f0005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf425ed40 0x55abf4272270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf425ed40 0x55abf4272270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38620/0 (socket says 172.21.15.28:38620) 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 -- 172.21.15.28:0/422218165 learned_addr learned my addr 172.21.15.28:0/422218165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc2fffff640 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf42743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc306ffd640 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4273e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 -- 172.21.15.28:0/422218165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 msgr2=0x55abf4273e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4273e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 -- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 msgr2=0x55abf42743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf42743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 -- 172.21.15.28:0/422218165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abf4272a70 con 0x55abf425ed40 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc3077fe640 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf425ed40 0x55abf4272270 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc300005ec0 tx=0x7fc300005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78473b2d37e12746 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc2ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/422218165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30001ebc0 con 0x55abf425ed40 2024-09-15T22:06:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f0031e50 con 0x55abf425ed40 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.234+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/422218165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7fc30001ed60 con 0x55abf425ed40 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/422218165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc300034050 con 0x55abf425ed40 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 10 client.17241 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55abf42d9cb0 con 0x55abf425ed40 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/422218165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f0031e50 con 0x55abf425ed40 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/422218165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc30001f070 con 0x55abf425ed40 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 10 client.17241 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 10 client.17241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 20 client.17241 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf42da400 0x55abf42dc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 4 client.17241 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T22:06:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55abf41c0d00 con 0x55abf42da400 2024-09-15T22:06:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.235+0000 7fc306ffd640 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf42da400 0x55abf42dc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.236+0000 7fc306ffd640 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf42da400 0x55abf42dc820 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba6fdc5d221fad7 server_cookie=25a29cf0975118ec in_seq=0 out_seq=0 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.236+0000 7fc2ee7fc640 10 client.17241 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.237+0000 7fc2ee7fc640 1 -- 172.21.15.28:0/422218165 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55abf41c0d00 con 0x55abf42da400 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.237+0000 7fc2ee7fc640 10 client.17241 handle_command_reply: tid=0 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.237+0000 7fc30f84b740 1 client.17241 shutdown 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.237+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf425ed40 msgr2=0x55abf4272270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.237+0000 7fc30f84b740 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf425ed40 0x55abf4272270 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc300005ec0 tx=0x7fc300005bd0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 shutdown_connections 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55abf42da400 0x55abf42dc820 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abf425ed40 0x55abf4272270 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55abf4254480 0x55abf42743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 --2- 172.21.15.28:0/422218165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55abf4253b60 0x55abf4273e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 >> 172.21.15.28:0/422218165 conn(0x55abf42500f0 msgr2=0x55abf425d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc2ee7fc640 0 client.17241 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 shutdown_connections 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 1 -- 172.21.15.28:0/422218165 wait complete. 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30f84b740 20 client.17241 trim_cache size 0 max 16384 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:17.256 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T22:06:17.257 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "b045eac2-80b1-4663-a28e-4ca2c166a7bf", 2024-09-15T22:06:17.257 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T22:06:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc2f40a3430 msgr2=0x7fc2f40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc2f40a3430 0x7fc2f40a58f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc300005c40 tx=0x7fc300009c90 comp rx=0 tx=0).stop 2024-09-15T22:06:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 msgr2=0x7fc308074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:17.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc308074a30 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0004770 tx=0x7fc2f0003c10 comp rx=0 tx=0).stop 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 shutdown_connections 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fc2f40a3430 0x7fc2f40a58f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc308106f10 0x7fc308074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc308102790 0x7fc308074a30 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 --2- 172.21.15.28:0/151595732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc308101de0 0x7fc3080744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.238+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 >> 172.21.15.28:0/151595732 conn(0x7fc3080fd8c0 msgr2=0x7fc3080fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.239+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 shutdown_connections 2024-09-15T22:06:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:17.239+0000 7fc30d472640 1 -- 172.21.15.28:0/151595732 wait complete. 2024-09-15T22:06:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:17 smithi174 ceph-mon[35244]: pgmap v274: 129 pgs: 7 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:17 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T22:06:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:17 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:18.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:17 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:06:18.261 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:17 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:18.270 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:17 smithi203 ceph-mon[35546]: pgmap v274: 129 pgs: 7 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:18.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:17 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T22:06:18.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:17 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:18.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:17 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:06:18.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:17 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:18.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:17 smithi028 ceph-mon[30969]: pgmap v274: 129 pgs: 7 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 11 op/s 2024-09-15T22:06:18.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:17 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T22:06:18.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:17 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:18.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:17 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:06:18.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:17 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:19.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:06:19.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:18 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:06:19.347 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:19.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:18 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:06:19.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:18 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:06:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 -- 172.21.15.28:0/1756795694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908102d30 msgr2=0x7f8908103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 --2- 172.21.15.28:0/1756795694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908102d30 0x7f8908103110 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009cd0 tx=0x7f88f802a8e0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 -- 172.21.15.28:0/1756795694 shutdown_connections 2024-09-15T22:06:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 --2- 172.21.15.28:0/1756795694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89081082a0 0x7f890810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 --2- 172.21.15.28:0/1756795694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89081036e0 0x7f8908107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 --2- 172.21.15.28:0/1756795694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908102d30 0x7f8908103110 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 -- 172.21.15.28:0/1756795694 >> 172.21.15.28:0/1756795694 conn(0x7f89080fc7d0 msgr2=0x7f89080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 -- 172.21.15.28:0/1756795694 shutdown_connections 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 -- 172.21.15.28:0/1756795694 wait complete. 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.847+0000 7f890d92c640 1 Processor -- start 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f890d92c640 1 -- start start 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f890d92c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8908102d30 0x7f89081a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f890d92c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89081036e0 0x7f89081a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f890d92c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89081082a0 0x7f890810ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f890d92c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89081120a0 con 0x7f89081082a0 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f890d92c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8908111f20 con 0x7f8908102d30 2024-09-15T22:06:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f890d92c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8908112220 con 0x7f89081036e0 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89081082a0 0x7f890810ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89081082a0 0x7f890810ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38636/0 (socket says 172.21.15.28:38636) 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 -- 172.21.15.28:0/3522113015 learned_addr learned my addr 172.21.15.28:0/3522113015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f8906ffd640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8908102d30 0x7f89081a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89067fc640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89081036e0 0x7f89081a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 -- 172.21.15.28:0/3522113015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89081036e0 msgr2=0x7f89081a4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89081036e0 0x7f89081a4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 -- 172.21.15.28:0/3522113015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8908102d30 msgr2=0x7f89081a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8908102d30 0x7f89081a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.848+0000 7f89077fe640 1 -- 172.21.15.28:0/3522113015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890810e670 con 0x7f89081082a0 2024-09-15T22:06:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.849+0000 7f89077fe640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89081082a0 0x7f890810ddb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f890000c970 tx=0x7f890000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.849+0000 7f890c92a640 1 -- 172.21.15.28:0/3522113015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900007bb0 con 0x7f89081082a0 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.849+0000 7f890c92a640 1 -- 172.21.15.28:0/3522113015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8900007d50 con 0x7f89081082a0 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.849+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f890810e960 con 0x7f89081082a0 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.849+0000 7f890c92a640 1 -- 172.21.15.28:0/3522113015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8900018800 con 0x7f89081082a0 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.849+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89081ab810 con 0x7f89081082a0 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.851+0000 7f890c92a640 1 -- 172.21.15.28:0/3522113015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f890000fa50 con 0x7f89081082a0 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.851+0000 7f890fd05740 1 Processor -- start 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890fd05740 1 -- start start 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890fd05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba3d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890fd05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba48140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890fd05740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 0x561ceba4aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890fd05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ceb9a32e0 con 0x561ceba3ddc0 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890fd05740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561ceb94ace0 con 0x561ceba3d4a0 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890fd05740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561ceb99f2a0 con 0x561ceba48680 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba48140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890c92a640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88f00a33b0 0x7f88f00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba48140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38640/0 (socket says 172.21.15.28:38640) 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 -- 172.21.15.28:0/3118728152 learned_addr learned my addr 172.21.15.28:0/3118728152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f89067fc640 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 0x561ceba4aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f89067fc640 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba3d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f890c92a640 1 -- 172.21.15.28:0/3522113015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f89000d0060 con 0x7f89081082a0 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f89077fe640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88f00a33b0 0x7f88f00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 -- 172.21.15.28:0/3118728152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 msgr2=0x561ceba4aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 0x561ceba4aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 -- 172.21.15.28:0/3118728152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 msgr2=0x561ceba3d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba3d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f8906ffd640 1 -- 172.21.15.28:0/3118728152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ceb92f440 con 0x561ceba3ddc0 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.852+0000 7f89067fc640 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba3d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.853+0000 7f8906ffd640 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba48140 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009cd0 tx=0x7f88f8031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826ea464d21d43e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.853+0000 7f89077fe640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88f00a33b0 0x7f88f00a5870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f89000cd400 tx=0x7f890000fd40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.853+0000 7f88edffb640 1 -- 172.21.15.28:0/3118728152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8048070 con 0x561ceba3ddc0 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.853+0000 7f88edffb640 1 -- 172.21.15.28:0/3118728152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f8036070 con 0x561ceba3ddc0 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.853+0000 7f88edffb640 1 -- 172.21.15.28:0/3118728152 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8004670 con 0x561ceba3ddc0 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.853+0000 7f890fd05740 1 -- 172.21.15.28:0/3118728152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 msgr2=0x561ceba48140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba48140 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009cd0 tx=0x7f88f8031700 comp rx=0 tx=0).stop 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- 172.21.15.28:0/3118728152 shutdown_connections 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 0x561ceba4aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba48140 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 --2- 172.21.15.28:0/3118728152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba3d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- 172.21.15.28:0/3118728152 >> 172.21.15.28:0/3118728152 conn(0x561ceba39a30 msgr2=0x561ceba39e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- 172.21.15.28:0/3118728152 shutdown_connections 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- 172.21.15.28:0/3118728152 wait complete. 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- start start 2024-09-15T22:06:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 Processor -- start 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba47f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3ddc0 0x561ceba451f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba48680 0x561ceba45730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ceba4b250 con 0x561ceba48680 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561ceb95e050 con 0x561ceba3d4a0 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.854+0000 7f890fd05740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561ceba4b3d0 con 0x561ceba3ddc0 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f890fd05740 10 client.0 _ll_register_callbacks cb 0x561ceb99e160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba48680 0x561ceba45730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba48680 0x561ceba45730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38656/0 (socket says 172.21.15.28:38656) 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 -- 172.21.15.28:0/184335677 learned_addr learned my addr 172.21.15.28:0/184335677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f8906ffd640 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3ddc0 0x561ceba451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 -- 172.21.15.28:0/184335677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3ddc0 msgr2=0x561ceba451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3ddc0 0x561ceba451f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 -- 172.21.15.28:0/184335677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 msgr2=0x561ceba47f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba47f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 -- 172.21.15.28:0/184335677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ceb7143c0 con 0x561ceba48680 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f89067fc640 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba48680 0x561ceba45730 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f89080697d0 tx=0x7f88fc010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63710d7d98d3521 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f88e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f88e6ffd640 1 -- 172.21.15.28:0/184335677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88fc01c020 con 0x561ceba48680 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f88e6ffd640 1 -- 172.21.15.28:0/184335677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f88fc005020 con 0x561ceba48680 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f8036070 con 0x561ceba48680 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.855+0000 7f88e6ffd640 1 -- 172.21.15.28:0/184335677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f88fc024050 con 0x561ceba48680 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 10 client.17253 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561ceba56ba0 con 0x561ceba48680 2024-09-15T22:06:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f88e6ffd640 1 -- 172.21.15.28:0/184335677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f8036070 con 0x561ceba48680 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f88e6ffd640 1 -- 172.21.15.28:0/184335677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f88fc018400 con 0x561ceba48680 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 10 client.17253 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 10 client.17253 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 20 client.17253 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561ceba58060 0x561ceba5a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 4 client.17253 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ceb9b0850 con 0x561ceba58060 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.856+0000 7f8906ffd640 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561ceba58060 0x561ceba5a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.857+0000 7f88e6ffd640 10 client.17253 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.857+0000 7f8906ffd640 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561ceba58060 0x561ceba5a450 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f570314f5badbd60 server_cookie=455bc1b700a7f1a4 in_seq=0 out_seq=0 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f88e6ffd640 1 -- 172.21.15.28:0/184335677 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ceb9b0850 con 0x561ceba58060 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f88e6ffd640 10 client.17253 handle_command_reply: tid=0 2024-09-15T22:06:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 client.17253 shutdown 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba48680 msgr2=0x561ceba45730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba48680 0x561ceba45730 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f89080697d0 tx=0x7f88fc010c10 comp rx=0 tx=0).stop 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 shutdown_connections 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561ceba58060 0x561ceba5a450 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba48680 0x561ceba45730 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3ddc0 0x561ceba451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 --2- 172.21.15.28:0/184335677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3d4a0 0x561ceba47f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 >> 172.21.15.28:0/184335677 conn(0x561ceba39a30 msgr2=0x561ceba38b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f88e6ffd640 0 client.17253 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 shutdown_connections 2024-09-15T22:06:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 1 -- 172.21.15.28:0/184335677 wait complete. 2024-09-15T22:06:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.858+0000 7f890fd05740 20 client.17253 trim_cache size 0 max 16384 2024-09-15T22:06:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.875+0000 7f890fd05740 1 Processor -- start 2024-09-15T22:06:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.875+0000 7f890fd05740 1 -- start start 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.875+0000 7f890fd05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3d4a0 0x561ceba544e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.875+0000 7f890fd05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3ddc0 0x561ceba5fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.875+0000 7f890fd05740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 0x561ceba60210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.875+0000 7f890fd05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89081b5750 con 0x561ceba3d4a0 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f890fd05740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89081b72c0 con 0x561ceba3ddc0 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f890fd05740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f89081b5e20 con 0x561ceba48680 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3d4a0 0x561ceba544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3d4a0 0x561ceba544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38668/0 (socket says 172.21.15.28:38668) 2024-09-15T22:06:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 -- 172.21.15.28:0/3841322959 learned_addr learned my addr 172.21.15.28:0/3841322959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f8906ffd640 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3ddc0 0x561ceba5fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 -- 172.21.15.28:0/3841322959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 msgr2=0x561ceba60210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 0x561ceba60210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 -- 172.21.15.28:0/3841322959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3ddc0 msgr2=0x561ceba5fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3ddc0 0x561ceba5fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 -- 172.21.15.28:0/3841322959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ceba3e330 con 0x561ceba3d4a0 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f8906ffd640 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3ddc0 0x561ceba5fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f89067fc640 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3d4a0 0x561ceba544e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00a3b0 tx=0x7f88fc010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0839652bbf85d2f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f88ed7fa640 1 -- 172.21.15.28:0/3841322959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88fc02f020 con 0x561ceba3d4a0 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f88ed7fa640 1 -- 172.21.15.28:0/3841322959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88fc018a40 con 0x561ceba3d4a0 2024-09-15T22:06:19.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.876+0000 7f88ed7fa640 1 -- 172.21.15.28:0/3841322959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88fc01c030 con 0x561ceba3d4a0 2024-09-15T22:06:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 -- 172.21.15.28:0/3841322959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3d4a0 msgr2=0x561ceba544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3d4a0 0x561ceba544e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00a3b0 tx=0x7f88fc010ec0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 -- 172.21.15.28:0/3841322959 shutdown_connections 2024-09-15T22:06:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba48680 0x561ceba60210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba3ddc0 0x561ceba5fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 --2- 172.21.15.28:0/3841322959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3d4a0 0x561ceba544e0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 -- 172.21.15.28:0/3841322959 >> 172.21.15.28:0/3841322959 conn(0x561ceba39a30 msgr2=0x561ceba407e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 -- 172.21.15.28:0/3841322959 shutdown_connections 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 -- 172.21.15.28:0/3841322959 wait complete. 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.877+0000 7f890fd05740 1 -- start start 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 1 Processor -- start 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3d4a0 0x561ceba50c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba4bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba48680 0x561ceba4c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89081b6bf0 con 0x561ceba3ddc0 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89081b6520 con 0x561ceba48680 2024-09-15T22:06:19.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f89081b3750 con 0x561ceba3d4a0 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f890fd05740 10 client.0 _ll_register_callbacks cb 0x7f88fc010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba4bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba4bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38670/0 (socket says 172.21.15.28:38670) 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 -- 172.21.15.28:0/1478028 learned_addr learned my addr 172.21.15.28:0/1478028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f89067fc640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3d4a0 0x561ceba50c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f89067fc640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba48680 0x561ceba4c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 -- 172.21.15.28:0/1478028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3d4a0 msgr2=0x561ceba50c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3d4a0 0x561ceba50c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 -- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba48680 msgr2=0x561ceba4c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba48680 0x561ceba4c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f8906ffd640 1 -- 172.21.15.28:0/1478028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ceba4cbe0 con 0x561ceba3ddc0 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.878+0000 7f89067fc640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba48680 0x561ceba4c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f8906ffd640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba4bfb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009a60 tx=0x7f88f80093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3649bf256e4f0cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f88e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f88e6ffd640 1 -- 172.21.15.28:0/1478028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8048040 con 0x561ceba3ddc0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f88e6ffd640 1 -- 172.21.15.28:0/1478028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9555+0+0 (secure 0 0 0) 0x7f88f8031c00 con 0x561ceba3ddc0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88fc018a40 con 0x561ceba3ddc0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f88e6ffd640 1 -- 172.21.15.28:0/1478028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f88f8009540 con 0x561ceba3ddc0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 10 client.17259 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561cebac2b90 con 0x561ceba3ddc0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f88e6ffd640 1 -- 172.21.15.28:0/1478028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88fc018a40 con 0x561ceba3ddc0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f88e6ffd640 1 -- 172.21.15.28:0/1478028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f88f8042030 con 0x561ceba3ddc0 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 10 client.17259 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 10 client.17259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 20 client.17259 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561cebac2f20 0x561cebac5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 4 client.17259 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.879+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561ceb9aad60 con 0x561cebac2f20 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.880+0000 7f89067fc640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561cebac2f20 0x561cebac5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.880+0000 7f88e6ffd640 10 client.17259 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.880+0000 7f89067fc640 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561cebac2f20 0x561cebac5310 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59ab12f376277776 server_cookie=8096d9efdd22dd2c in_seq=0 out_seq=0 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f88e6ffd640 1 -- 172.21.15.28:0/1478028 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561ceb9aad60 con 0x561cebac2f20 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f88e6ffd640 10 client.17259 handle_command_reply: tid=0 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 client.17259 shutdown 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 msgr2=0x561ceba4bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba4bfb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009a60 tx=0x7f88f80093c0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 shutdown_connections 2024-09-15T22:06:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x561cebac2f20 0x561cebac5310 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561ceba48680 0x561ceba4c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ceba3ddc0 0x561ceba4bfb0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f88e6ffd640 0 client.17259 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 --2- 172.21.15.28:0/1478028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561ceba3d4a0 0x561ceba50c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 >> 172.21.15.28:0/1478028 conn(0x561ceba39a30 msgr2=0x561ceba38350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 shutdown_connections 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 1 -- 172.21.15.28:0/1478028 wait complete. 2024-09-15T22:06:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.881+0000 7f890fd05740 20 client.17259 trim_cache size 0 max 16384 2024-09-15T22:06:19.890 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:06:19.890 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:06:19.890 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88f00a33b0 msgr2=0x7f88f00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88f00a33b0 0x7f88f00a5870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f89000cd400 tx=0x7f890000fd40 comp rx=0 tx=0).stop 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89081082a0 msgr2=0x7f890810ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89081082a0 0x7f890810ddb0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f890000c970 tx=0x7f890000ce30 comp rx=0 tx=0).stop 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 shutdown_connections 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88f00a33b0 0x7f88f00a5870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89081082a0 0x7f890810ddb0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89081036e0 0x7f89081a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 --2- 172.21.15.28:0/3522113015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8908102d30 0x7f89081a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 >> 172.21.15.28:0/3522113015 conn(0x7f89080fc7d0 msgr2=0x7f89080fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 shutdown_connections 2024-09-15T22:06:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:19.882+0000 7f890d92c640 1 -- 172.21.15.28:0/3522113015 wait complete. 2024-09-15T22:06:20.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:19 smithi028 ceph-mon[30969]: pgmap v275: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:06:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:19 smithi174 ceph-mon[35244]: pgmap v275: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:06:20.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:19 smithi203 ceph-mon[35546]: pgmap v275: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:06:20.725 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:06:20.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:06:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 -- 172.21.15.28:0/336864890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780106a80 msgr2=0x7f7780106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 --2- 172.21.15.28:0/336864890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780106a80 0x7f7780106e60 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7770009930 tx=0x7f777002a420 comp rx=0 tx=0).stop 2024-09-15T22:06:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 -- 172.21.15.28:0/336864890 shutdown_connections 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 --2- 172.21.15.28:0/336864890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7780101420 0x7f778010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 --2- 172.21.15.28:0/336864890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780100a80 0x7f7780100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 --2- 172.21.15.28:0/336864890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780106a80 0x7f7780106e60 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 -- 172.21.15.28:0/336864890 >> 172.21.15.28:0/336864890 conn(0x7f77800fc7a0 msgr2=0x7f77800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 -- 172.21.15.28:0/336864890 shutdown_connections 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.522+0000 7f7785584640 1 -- 172.21.15.28:0/336864890 wait complete. 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 Processor -- start 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 -- start start 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780100a80 0x7f7780102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780101420 0x7f7780102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7780106a80 0x7f77801030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:21.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7780110e80 con 0x7f7780101420 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7780110d00 con 0x7f7780106a80 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f7785584640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7780111000 con 0x7f7780100a80 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.523+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780101420 0x7f7780102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780101420 0x7f7780102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52092/0 (socket says 172.21.15.28:52092) 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 -- 172.21.15.28:0/501295236 learned_addr learned my addr 172.21.15.28:0/501295236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777f7fe640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7780106a80 0x7f77801030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777effd640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780100a80 0x7f7780102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 -- 172.21.15.28:0/501295236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780100a80 msgr2=0x7f7780102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780100a80 0x7f7780102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 -- 172.21.15.28:0/501295236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7780106a80 msgr2=0x7f77801030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7780106a80 0x7f77801030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 -- 172.21.15.28:0/501295236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77801a68d0 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f777e7fc640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780101420 0x7f7780102b90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f776800e9f0 tx=0x7f776800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f7777fff640 1 -- 172.21.15.28:0/501295236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776800cd80 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f7777fff640 1 -- 172.21.15.28:0/501295236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7768010910 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77801a6bc0 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f7777fff640 1 -- 172.21.15.28:0/501295236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768018990 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.524+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77801a7100 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.525+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7780100ee0 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.527+0000 7f7777fff640 1 -- 172.21.15.28:0/501295236 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f7768010ac0 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.527+0000 7f7777fff640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f776c0a33b0 0x7f776c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.527+0000 7f777effd640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f776c0a33b0 0x7f776c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.527+0000 7f7777fff640 1 -- 172.21.15.28:0/501295236 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f77680d0470 con 0x7f7780101420 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.528+0000 7f777effd640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f776c0a33b0 0x7f776c0a5870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7770000c00 tx=0x7f7770005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.532+0000 7f7777fff640 1 -- 172.21.15.28:0/501295236 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f776809caa0 con 0x7f7780101420 2024-09-15T22:06:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.742+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77801a72d0 con 0x7f7780101420 2024-09-15T22:06:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.743+0000 7f7777fff640 1 -- 172.21.15.28:0/501295236 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11988 (secure 0 0 0) 0x7f776809c3a0 con 0x7f7780101420 2024-09-15T22:06:21.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:06:21.745 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"btime":"2024-09-15T22:04:37:022811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:04:36.713632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:06:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-09-15T22:06:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f776c0a33b0 msgr2=0x7f776c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f776c0a33b0 0x7f776c0a5870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7770000c00 tx=0x7f7770005a20 comp rx=0 tx=0).stop 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780101420 msgr2=0x7f7780102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780101420 0x7f7780102b90 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f776800e9f0 tx=0x7f776800eeb0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 shutdown_connections 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f776c0a33b0 0x7f776c0a5870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7780106a80 0x7f77801030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7780101420 0x7f7780102b90 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 --2- 172.21.15.28:0/501295236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780100a80 0x7f7780102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 >> 172.21.15.28:0/501295236 conn(0x7f77800fc7a0 msgr2=0x7f778006a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 shutdown_connections 2024-09-15T22:06:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:21.745+0000 7f7785584640 1 -- 172.21.15.28:0/501295236 wait complete. 2024-09-15T22:06:22.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:21 smithi028 ceph-mon[30969]: pgmap v276: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 179 B/s wr, 9 op/s 2024-09-15T22:06:22.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:21 smithi028 ceph-mon[30969]: osdmap e97: 12 total, 12 up, 12 in 2024-09-15T22:06:22.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:21 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/501295236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:21 smithi203 ceph-mon[35546]: pgmap v276: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 179 B/s wr, 9 op/s 2024-09-15T22:06:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:21 smithi203 ceph-mon[35546]: osdmap e97: 12 total, 12 up, 12 in 2024-09-15T22:06:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:21 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/501295236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:21 smithi174 ceph-mon[35244]: pgmap v276: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 179 B/s wr, 9 op/s 2024-09-15T22:06:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:21 smithi174 ceph-mon[35244]: osdmap e97: 12 total, 12 up, 12 in 2024-09-15T22:06:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:21 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/501295236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:22.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:06:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 -- 172.21.15.28:0/3985671176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84106a80 msgr2=0x7fbb84106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3985671176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84106a80 0x7fbb84106e60 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009930 tx=0x7fbb7402a420 comp rx=0 tx=0).stop 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 -- 172.21.15.28:0/3985671176 shutdown_connections 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3985671176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb84101420 0x7fbb8410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3985671176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb84100a80 0x7fbb84100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3985671176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84106a80 0x7fbb84106e60 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 -- 172.21.15.28:0/3985671176 >> 172.21.15.28:0/3985671176 conn(0x7fbb840fc7a0 msgr2=0x7fbb840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 -- 172.21.15.28:0/3985671176 shutdown_connections 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.139+0000 7fbb8920f640 1 -- 172.21.15.28:0/3985671176 wait complete. 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 Processor -- start 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 -- start start 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84100a80 0x7fbb84198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb84101420 0x7fbb84198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb84106a80 0x7fbb8419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb84110e80 con 0x7fbb84100a80 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb84110d00 con 0x7fbb84101420 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.140+0000 7fbb8920f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb84111000 con 0x7fbb84106a80 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84100a80 0x7fbb84198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84100a80 0x7fbb84198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52124/0 (socket says 172.21.15.28:52124) 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3947334842 learned_addr learned my addr 172.21.15.28:0/3947334842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb8259c640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb84101420 0x7fbb84198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb84106a80 0x7fbb8419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3947334842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb84106a80 msgr2=0x7fbb8419cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb84106a80 0x7fbb8419cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3947334842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb84101420 msgr2=0x7fbb84198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb84101420 0x7fbb84198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3947334842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8419d4b0 con 0x7fbb84100a80 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb8259c640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb84101420 0x7fbb84198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84100a80 0x7fbb84198500 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009230 tx=0x7fbb74004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb6bfff640 1 -- 172.21.15.28:0/3947334842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7403f030 con 0x7fbb84100a80 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8419d740 con 0x7fbb84100a80 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb6bfff640 1 -- 172.21.15.28:0/3947334842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb74003ad0 con 0x7fbb84100a80 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb6bfff640 1 -- 172.21.15.28:0/3947334842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74035070 con 0x7fbb84100a80 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.141+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb841a5080 con 0x7fbb84100a80 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb6bfff640 1 -- 172.21.15.28:0/3947334842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fbb74042030 con 0x7fbb84100a80 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 Processor -- start 2024-09-15T22:06:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 -- start start 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d03b40 0x559580d02090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d02a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559580c682f0 con 0x559580d03b40 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559580c0fcf0 con 0x559580d0d370 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559580c642b0 con 0x559580d025d0 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d03b40 0x559580d02090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d03b40 0x559580d02090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52132/0 (socket says 172.21.15.28:52132) 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8359e640 1 -- 172.21.15.28:0/550972816 learned_addr learned my addr 172.21.15.28:0/550972816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb8259c640 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb6bfff640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb6c0a33b0 0x7fbb6c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.144+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d02a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 -- 172.21.15.28:0/550972816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 msgr2=0x559580d02a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d02a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 -- 172.21.15.28:0/550972816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 msgr2=0x559580d0f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 -- 172.21.15.28:0/550972816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559580bf43a0 con 0x559580d03b40 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8259c640 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb6c0a33b0 0x7fbb6c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb6bfff640 1 -- 172.21.15.28:0/3947334842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fbb740f0670 con 0x7fbb84100a80 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d03b40 0x559580d02090 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009fc0 tx=0x7fbb7c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451bd87739d2c4e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb697fa640 1 -- 172.21.15.28:0/550972816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c01c900 con 0x559580d03b40 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb697fa640 1 -- 172.21.15.28:0/550972816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb7c0113e0 con 0x559580d03b40 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb697fa640 1 -- 172.21.15.28:0/550972816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c013620 con 0x559580d03b40 2024-09-15T22:06:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.145+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb6c0a33b0 0x7fbb6c0a5870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c020000 tx=0x7fbb7c01d000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/550972816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d03b40 msgr2=0x559580d02090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d03b40 0x559580d02090 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009fc0 tx=0x7fbb7c010800 comp rx=0 tx=0).stop 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/550972816 shutdown_connections 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d02a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/550972816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d03b40 0x559580d02090 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/550972816 >> 172.21.15.28:0/550972816 conn(0x559580ccb310 msgr2=0x559580cfec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/550972816 shutdown_connections 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/550972816 wait complete. 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 -- start start 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.146+0000 7fbb8b5e8740 1 Processor -- start 2024-09-15T22:06:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d025d0 0x559580d0cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d09fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559580d10070 con 0x559580d025d0 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559580c23060 con 0x559580d0d370 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559580d101f0 con 0x559580d03b40 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8b5e8740 10 client.0 _ll_register_callbacks cb 0x5595809768c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d025d0 0x559580d0cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d025d0 0x559580d0cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52144/0 (socket says 172.21.15.28:52144) 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3432390848 learned_addr learned my addr 172.21.15.28:0/3432390848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8259c640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d09fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 msgr2=0x559580d09fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d09fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3432390848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 msgr2=0x559580d0a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 -- 172.21.15.28:0/3432390848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595809d93c0 con 0x559580d025d0 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8259c640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d09fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d025d0 0x559580d0cd10 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009900 tx=0x7fbb74009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0bee126a49cd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.147+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3432390848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74049050 con 0x559580d025d0 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3432390848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fbb740edc90 con 0x559580d025d0 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb7c0113e0 con 0x559580d025d0 2024-09-15T22:06:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3432390848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbb740bcca0 con 0x559580d025d0 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 10 client.17277 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559580d1bac0 con 0x559580d025d0 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3432390848 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb7c0113e0 con 0x559580d025d0 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3432390848 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fbb74003c80 con 0x559580d025d0 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 10 client.17277 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 10 client.17277 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 20 client.17277 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d1cfb0 0x559580d1f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 4 client.17277 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559580c75860 con 0x559580d1cfb0 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.148+0000 7fbb8259c640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d1cfb0 0x559580d1f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.149+0000 7fbb66ff5640 10 client.17277 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.149+0000 7fbb8259c640 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d1cfb0 0x559580d1f3d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ecd1517f113950c3 server_cookie=8bc064f45948a712 in_seq=0 out_seq=0 2024-09-15T22:06:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3432390848 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559580c75860 con 0x559580d1cfb0 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb66ff5640 10 client.17277 handle_command_reply: tid=0 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 client.17277 shutdown 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d025d0 msgr2=0x559580d0cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d025d0 0x559580d0cd10 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009900 tx=0x7fbb74009260 comp rx=0 tx=0).stop 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 shutdown_connections 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d1cfb0 0x559580d1f3d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d0d370 0x559580d0a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d09fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb66ff5640 0 client.17277 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3432390848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d025d0 0x559580d0cd10 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 >> 172.21.15.28:0/3432390848 conn(0x559580ccb310 msgr2=0x559580d04960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 shutdown_connections 2024-09-15T22:06:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3432390848 wait complete. 2024-09-15T22:06:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.150+0000 7fbb8b5e8740 20 client.17277 trim_cache size 0 max 16384 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.163+0000 7fbb8b5e8740 1 Processor -- start 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.163+0000 7fbb8b5e8740 1 -- start start 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.163+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d09880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d03b40 0x559580cfdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d05120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb841af050 con 0x559580d0d370 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb841b0bc0 con 0x559580d03b40 2024-09-15T22:06:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb841af720 con 0x559580d025d0 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d05120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d05120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52154/0 (socket says 172.21.15.28:52154) 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 -- 172.21.15.28:0/4181866271 learned_addr learned my addr 172.21.15.28:0/4181866271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d09880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8259c640 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d03b40 0x559580cfdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 -- 172.21.15.28:0/4181866271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 msgr2=0x559580d09880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d09880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 -- 172.21.15.28:0/4181866271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d03b40 msgr2=0x559580cfdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d03b40 0x559580cfdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 -- 172.21.15.28:0/4181866271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559580cff730 con 0x559580d0d370 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb8359e640 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d05120 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009760 tx=0x7fbb7c0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456467b37018222a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb68ff9640 1 -- 172.21.15.28:0/4181866271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c0135e0 con 0x559580d0d370 2024-09-15T22:06:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.164+0000 7fbb68ff9640 1 -- 172.21.15.28:0/4181866271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb7c01b3d0 con 0x559580d0d370 2024-09-15T22:06:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb68ff9640 1 -- 172.21.15.28:0/4181866271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c01bb10 con 0x559580d0d370 2024-09-15T22:06:23.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/4181866271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 msgr2=0x559580d05120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d05120 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009760 tx=0x7fbb7c0095a0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/4181866271 shutdown_connections 2024-09-15T22:06:23.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d05120 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d03b40 0x559580cfdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/4181866271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d025d0 0x559580d09880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/4181866271 >> 172.21.15.28:0/4181866271 conn(0x559580ccb310 msgr2=0x559580cfd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/4181866271 shutdown_connections 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/4181866271 wait complete. 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.165+0000 7fbb8b5e8740 1 -- start start 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 1 Processor -- start 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d025d0 0x559580d15b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d06ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb841b04f0 con 0x559580d0d370 2024-09-15T22:06:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb841afe20 con 0x559580d025d0 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb841ad050 con 0x559580d03b40 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8b5e8740 10 client.0 _ll_register_callbacks cb 0x7fbb74005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52160/0 (socket says 172.21.15.28:52160) 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 -- 172.21.15.28:0/3739665194 learned_addr learned my addr 172.21.15.28:0/3739665194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d025d0 0x559580d15b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8259c640 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d06ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 -- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 msgr2=0x559580d06ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d06ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 -- 172.21.15.28:0/3739665194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d025d0 msgr2=0x559580d15b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d025d0 0x559580d15b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 -- 172.21.15.28:0/3739665194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559580d07a10 con 0x559580d0d370 2024-09-15T22:06:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb8359e640 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d071e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c01b930 tx=0x7fbb7c01b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb684f288ce499 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.166+0000 7fbb66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3739665194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c009b10 con 0x559580d0d370 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3739665194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fbb7c02d050 con 0x559580d0d370 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb7c01b3d0 con 0x559580d0d370 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3739665194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbb7c0135e0 con 0x559580d0d370 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 10 client.17283 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559580d87c90 con 0x559580d0d370 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3739665194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb7c01b3d0 con 0x559580d0d370 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3739665194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fbb7c00f040 con 0x559580d0d370 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 10 client.17283 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 10 client.17283 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:06:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 20 client.17283 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d881a0 0x559580d8a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 4 client.17283 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.167+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559580c6fd70 con 0x559580d881a0 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.168+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d881a0 0x559580d8a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.168+0000 7fbb82d9d640 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d881a0 0x559580d8a590 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fbd42e875f289e1e server_cookie=90cd0bb5c845e4af in_seq=0 out_seq=0 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.168+0000 7fbb66ff5640 10 client.17283 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.168+0000 7fbb66ff5640 1 -- 172.21.15.28:0/3739665194 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559580c6fd70 con 0x559580d881a0 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.168+0000 7fbb66ff5640 10 client.17283 handle_command_reply: tid=0 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.168+0000 7fbb8b5e8740 1 client.17283 shutdown 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 msgr2=0x559580d071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d071e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c01b930 tx=0x7fbb7c01b850 comp rx=0 tx=0).stop 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 shutdown_connections 2024-09-15T22:06:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x559580d881a0 0x559580d8a590 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580d0d370 0x559580d071e0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559580d03b40 0x559580d06ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 --2- 172.21.15.28:0/3739665194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559580d025d0 0x559580d15b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 >> 172.21.15.28:0/3739665194 conn(0x559580ccb310 msgr2=0x559580cfdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb66ff5640 0 client.17283 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 shutdown_connections 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 1 -- 172.21.15.28:0/3739665194 wait complete. 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8b5e8740 20 client.17283 trim_cache size 0 max 16384 2024-09-15T22:06:23.176 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb6c0a33b0 msgr2=0x7fbb6c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb6c0a33b0 0x7fbb6c0a5870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c020000 tx=0x7fbb7c01d000 comp rx=0 tx=0).stop 2024-09-15T22:06:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84100a80 msgr2=0x7fbb84198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84100a80 0x7fbb84198500 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009230 tx=0x7fbb74004270 comp rx=0 tx=0).stop 2024-09-15T22:06:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 shutdown_connections 2024-09-15T22:06:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.169+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbb6c0a33b0 0x7fbb6c0a5870 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.170+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb84106a80 0x7fbb8419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.170+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb84101420 0x7fbb84198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.170+0000 7fbb8920f640 1 --2- 172.21.15.28:0/3947334842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb84100a80 0x7fbb84198500 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.170+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 >> 172.21.15.28:0/3947334842 conn(0x7fbb840fc7a0 msgr2=0x7fbb8406aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.170+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 shutdown_connections 2024-09-15T22:06:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:23.170+0000 7fbb8920f640 1 -- 172.21.15.28:0/3947334842 wait complete. 2024-09-15T22:06:23.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:06:24.136 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:24.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:23 smithi028 ceph-mon[30969]: pgmap v278: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:24.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:23 smithi203 ceph-mon[35546]: pgmap v278: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:24.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:23 smithi174 ceph-mon[35244]: pgmap v278: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 -- 172.21.15.28:0/2375049761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 msgr2=0x7f06c010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 --2- 172.21.15.28:0/2375049761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c010ea90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009ca0 tx=0x7f06b802a8c0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 -- 172.21.15.28:0/2375049761 shutdown_connections 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 --2- 172.21.15.28:0/2375049761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c010ea90 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 --2- 172.21.15.28:0/2375049761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c0102790 0x7f06c01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 --2- 172.21.15.28:0/2375049761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06c0101de0 0x7f06c01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 -- 172.21.15.28:0/2375049761 >> 172.21.15.28:0/2375049761 conn(0x7f06c00fd8c0 msgr2=0x7f06c00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 -- 172.21.15.28:0/2375049761 shutdown_connections 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.676+0000 7f06c5b34640 1 -- 172.21.15.28:0/2375049761 wait complete. 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 Processor -- start 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 -- start start 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06c0101de0 0x7f06c01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c0102790 0x7f06c0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0111f70 con 0x7f06c0106f10 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0111df0 con 0x7f06c0102790 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06c5b34640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01120f0 con 0x7f06c0101de0 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.677+0000 7f06bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52170/0 (socket says 172.21.15.28:52170) 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 -- 172.21.15.28:0/3484991657 learned_addr learned my addr 172.21.15.28:0/3484991657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06c0101de0 0x7f06c01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06beffd640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c0102790 0x7f06c0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 -- 172.21.15.28:0/3484991657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06c0101de0 msgr2=0x7f06c01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06c0101de0 0x7f06c01983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 -- 172.21.15.28:0/3484991657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c0102790 msgr2=0x7f06c0198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c0102790 0x7f06c0198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 -- 172.21.15.28:0/3484991657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c019d380 con 0x7f06c0106f10 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bffff640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c019cca0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f06b8031a20 tx=0x7f06b8031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bcff9640 1 -- 172.21.15.28:0/3484991657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8003c30 con 0x7f06c0106f10 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bcff9640 1 -- 172.21.15.28:0/3484991657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b8003dd0 con 0x7f06c0106f10 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c019d610 con 0x7f06c0106f10 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06bcff9640 1 -- 172.21.15.28:0/3484991657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b802fcf0 con 0x7f06c0106f10 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.678+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c01a4f50 con 0x7f06c0106f10 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06bcff9640 1 -- 172.21.15.28:0/3484991657 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f06b8004050 con 0x7f06c0106f10 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 Processor -- start 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 -- start start 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e9b40 0x5582810e8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55828104e2f0 con 0x5582810f3370 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558280ff5cf0 con 0x5582810e85d0 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55828104a2b0 con 0x5582810e9b40 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52186/0 (socket says 172.21.15.28:52186) 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06beffd640 1 -- 172.21.15.28:0/2321141248 learned_addr learned my addr 172.21.15.28:0/2321141248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06beffd640 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e9b40 0x5582810e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06bcff9640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f06a80a3430 0x7f06a80a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.681+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06beffd640 1 -- 172.21.15.28:0/2321141248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e9b40 msgr2=0x5582810e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06beffd640 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e9b40 0x5582810e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06beffd640 1 -- 172.21.15.28:0/2321141248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 msgr2=0x5582810e8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06beffd640 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810e8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06beffd640 1 -- 172.21.15.28:0/2321141248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558280fda3a0 con 0x5582810f3370 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06bffff640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f06a80a3430 0x7f06a80a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06beffd640 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f5760 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f06b400efd0 tx=0x7f06b400e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70aad41dfc2aa9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06bcff9640 1 -- 172.21.15.28:0/3484991657 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f06b80f1390 con 0x7f06c0106f10 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06a5ffb640 1 -- 172.21.15.28:0/2321141248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4018070 con 0x5582810f3370 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06a5ffb640 1 -- 172.21.15.28:0/2321141248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b4005500 con 0x5582810f3370 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.682+0000 7f06a5ffb640 1 -- 172.21.15.28:0/2321141248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4004d90 con 0x5582810f3370 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 -- 172.21.15.28:0/2321141248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 msgr2=0x5582810f5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06bffff640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f06a80a3430 0x7f06a80a58f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f06b80ee8f0 tx=0x7f06b80b91f0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f5760 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f06b400efd0 tx=0x7f06b400e5d0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 -- 172.21.15.28:0/2321141248 shutdown_connections 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f5760 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/2321141248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e9b40 0x5582810e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 -- 172.21.15.28:0/2321141248 >> 172.21.15.28:0/2321141248 conn(0x5582810b1310 msgr2=0x5582810e4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 -- 172.21.15.28:0/2321141248 shutdown_connections 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 -- 172.21.15.28:0/2321141248 wait complete. 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 -- start start 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.683+0000 7f06c7f0d740 1 Processor -- start 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e85d0 0x5582810f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e9b40 0x5582810f0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582810f6070 con 0x5582810f3370 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558281009060 con 0x5582810e9b40 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5582810f61f0 con 0x5582810e85d0 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06c7f0d740 10 client.0 _ll_register_callbacks cb 0x558280d5c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52188/0 (socket says 172.21.15.28:52188) 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 -- 172.21.15.28:0/1893336340 learned_addr learned my addr 172.21.15.28:0/1893336340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e85d0 0x5582810f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e9b40 0x5582810f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 -- 172.21.15.28:0/1893336340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e85d0 msgr2=0x5582810f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e85d0 0x5582810f2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 -- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e9b40 msgr2=0x5582810f0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e9b40 0x5582810f0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 -- 172.21.15.28:0/1893336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558280dbf3c0 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f06beffd640 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f07f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f06b400ee40 tx=0x7f06b4004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54a8019a7196ebb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.684+0000 7f069effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f069effd640 1 -- 172.21.15.28:0/1893336340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4018070 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f069effd640 1 -- 172.21.15.28:0/1893336340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f06b40106a0 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06b4005500 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f069effd640 1 -- 172.21.15.28:0/1893336340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f06b4035050 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 10 client.17295 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x558281101b60 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f069effd640 1 -- 172.21.15.28:0/1893336340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b4005500 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f069effd640 1 -- 172.21.15.28:0/1893336340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f06b401c7c0 con 0x5582810f3370 2024-09-15T22:06:24.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 10 client.17295 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 10 client.17295 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 20 client.17295 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558281102eb0 0x5582811052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 4 client.17295 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.685+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55828105b860 con 0x558281102eb0 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.686+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558281102eb0 0x5582811052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.686+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558281102eb0 0x5582811052a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9374791ab9a6359a server_cookie=322836681377a8cf in_seq=0 out_seq=0 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.686+0000 7f069effd640 10 client.17295 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f069effd640 1 -- 172.21.15.28:0/1893336340 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55828105b860 con 0x558281102eb0 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f069effd640 10 client.17295 handle_command_reply: tid=0 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 client.17295 shutdown 2024-09-15T22:06:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 msgr2=0x5582810f07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f07f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f06b400ee40 tx=0x7f06b4004970 comp rx=0 tx=0).stop 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 shutdown_connections 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x558281102eb0 0x5582811052a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810f3370 0x5582810f07f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e9b40 0x5582810f0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1893336340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810e85d0 0x5582810f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f069effd640 0 client.17295 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 >> 172.21.15.28:0/1893336340 conn(0x5582810b1310 msgr2=0x5582810ea960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 shutdown_connections 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1893336340 wait complete. 2024-09-15T22:06:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.687+0000 7f06c7f0d740 20 client.17295 trim_cache size 0 max 16384 2024-09-15T22:06:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 Processor -- start 2024-09-15T22:06:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 -- start start 2024-09-15T22:06:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810ef920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x55828110a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x55828110af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01aefb0 con 0x5582810e9b40 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01b0b20 con 0x5582810e85d0 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.700+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01af680 con 0x5582810f3370 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x55828110a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x55828110a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52200/0 (socket says 172.21.15.28:52200) 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 -- 172.21.15.28:0/3911888834 learned_addr learned my addr 172.21.15.28:0/3911888834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06beffd640 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x55828110af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 -- 172.21.15.28:0/3911888834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 msgr2=0x55828110af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x55828110af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06beffd640 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810ef920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 -- 172.21.15.28:0/3911888834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 msgr2=0x5582810ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810ef920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 -- 172.21.15.28:0/3911888834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582810e44f0 con 0x5582810e9b40 2024-09-15T22:06:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x55828110a9e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x5582810ef8f0 tx=0x7f06b0014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a134da02a4d8fec5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06a57fa640 1 -- 172.21.15.28:0/3911888834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0014190 con 0x5582810e9b40 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06a57fa640 1 -- 172.21.15.28:0/3911888834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b0012930 con 0x5582810e9b40 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.701+0000 7f06a57fa640 1 -- 172.21.15.28:0/3911888834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b00269e0 con 0x5582810e9b40 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 -- 172.21.15.28:0/3911888834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 msgr2=0x55828110a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x55828110a9e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x5582810ef8f0 tx=0x7f06b0014900 comp rx=0 tx=0).stop 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 -- 172.21.15.28:0/3911888834 shutdown_connections 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x55828110af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x55828110a9e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/3911888834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810ef920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 -- 172.21.15.28:0/3911888834 >> 172.21.15.28:0/3911888834 conn(0x5582810b1310 msgr2=0x5582810dfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 -- 172.21.15.28:0/3911888834 shutdown_connections 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 -- 172.21.15.28:0/3911888834 wait complete. 2024-09-15T22:06:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 -- start start 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.702+0000 7f06c7f0d740 1 Processor -- start 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x5582810f7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06c7f0d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x5582810fb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01b0450 con 0x5582810e9b40 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01afd80 con 0x5582810e85d0 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06c7f0d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01acfb0 con 0x5582810f3370 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06c7f0d740 10 client.0 _ll_register_callbacks cb 0x7f06b400ef00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x5582810f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x5582810f7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52210/0 (socket says 172.21.15.28:52210) 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 -- 172.21.15.28:0/1157293737 learned_addr learned my addr 172.21.15.28:0/1157293737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06beffd640 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x5582810fb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06beffd640 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 -- 172.21.15.28:0/1157293737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 msgr2=0x5582810fb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x5582810fb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 -- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 msgr2=0x5582810f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810f6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 -- 172.21.15.28:0/1157293737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582810fb810 con 0x5582810e9b40 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f06bf7fe640 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x5582810f7120 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f06b00266c0 tx=0x7f06b000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325b0492a1a700cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.703+0000 7f069effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f069effd640 1 -- 172.21.15.28:0/1157293737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0012ae0 con 0x5582810e9b40 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f069effd640 1 -- 172.21.15.28:0/1157293737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f06b0012c80 con 0x5582810e9b40 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06b0012930 con 0x5582810e9b40 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f069effd640 1 -- 172.21.15.28:0/1157293737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f06b003b050 con 0x5582810e9b40 2024-09-15T22:06:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 10 client.17301 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55828116d410 con 0x5582810e9b40 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f069effd640 1 -- 172.21.15.28:0/1157293737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b0012930 con 0x5582810e9b40 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f069effd640 1 -- 172.21.15.28:0/1157293737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f06b0032450 con 0x5582810e9b40 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 10 client.17301 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 10 client.17301 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 20 client.17301 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55828116dba0 0x55828116ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 4 client.17301 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.704+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558281055d70 con 0x55828116dba0 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06beffd640 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55828116dba0 0x55828116ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06beffd640 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55828116dba0 0x55828116ffc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fadb14579ec0bce4 server_cookie=f399cd019e60d2f1 in_seq=0 out_seq=0 2024-09-15T22:06:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f069effd640 10 client.17301 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f069effd640 1 -- 172.21.15.28:0/1157293737 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558281055d70 con 0x55828116dba0 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f069effd640 10 client.17301 handle_command_reply: tid=0 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 client.17301 shutdown 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 msgr2=0x5582810f7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x5582810f7120 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f06b00266c0 tx=0x7f06b000be60 comp rx=0 tx=0).stop 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 shutdown_connections 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5582810f3370 0x5582810fb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55828116dba0 0x55828116ffc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582810e9b40 0x5582810f7120 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f069effd640 0 client.17301 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 --2- 172.21.15.28:0/1157293737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582810e85d0 0x5582810f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.705+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 >> 172.21.15.28:0/1157293737 conn(0x5582810b1310 msgr2=0x5582810ef150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 shutdown_connections 2024-09-15T22:06:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c7f0d740 1 -- 172.21.15.28:0/1157293737 wait complete. 2024-09-15T22:06:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c7f0d740 20 client.17301 trim_cache size 0 max 16384 2024-09-15T22:06:24.719 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f06a80a3430 msgr2=0x7f06a80a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f06a80a3430 0x7f06a80a58f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f06b80ee8f0 tx=0x7f06b80b91f0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 msgr2=0x7f06c019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c019cca0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f06b8031a20 tx=0x7f06b8031a50 comp rx=0 tx=0).stop 2024-09-15T22:06:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 shutdown_connections 2024-09-15T22:06:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f06a80a3430 0x7f06a80a58f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06c0106f10 0x7f06c019cca0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c0102790 0x7f06c0198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 --2- 172.21.15.28:0/3484991657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06c0101de0 0x7f06c01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.706+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 >> 172.21.15.28:0/3484991657 conn(0x7f06c00fd8c0 msgr2=0x7f06c00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.707+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 shutdown_connections 2024-09-15T22:06:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:24.707+0000 7f06c5b34640 1 -- 172.21.15.28:0/3484991657 wait complete. 2024-09-15T22:06:25.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:06:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:26.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:25 smithi028 ceph-mon[30969]: pgmap v279: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 -- 172.21.15.28:0/2693566768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 msgr2=0x7fae901021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 --2- 172.21.15.28:0/2693566768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901021c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009900 tx=0x7fae8c02a410 comp rx=0 tx=0).stop 2024-09-15T22:06:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 -- 172.21.15.28:0/2693566768 shutdown_connections 2024-09-15T22:06:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 --2- 172.21.15.28:0/2693566768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae90106f10 0x7fae9010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 --2- 172.21.15.28:0/2693566768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae90102790 0x7fae901067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 --2- 172.21.15.28:0/2693566768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901021c0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 -- 172.21.15.28:0/2693566768 >> 172.21.15.28:0/2693566768 conn(0x7fae900fd8c0 msgr2=0x7fae900ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 -- 172.21.15.28:0/2693566768 shutdown_connections 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.185+0000 7fae98424640 1 -- 172.21.15.28:0/2693566768 wait complete. 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 Processor -- start 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 -- start start 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae90102790 0x7fae90198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae90106f10 0x7fae9019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae90111f70 con 0x7fae90101de0 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fae90111df0 con 0x7fae90106f10 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae98424640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fae901120f0 con 0x7fae90102790 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae961c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae961c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52216/0 (socket says 172.21.15.28:52216) 2024-09-15T22:06:26.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae961c0640 1 -- 172.21.15.28:0/1215589623 learned_addr learned my addr 172.21.15.28:0/1215589623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.186+0000 7fae959bf640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae90102790 0x7fae90198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae969c1640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae90106f10 0x7fae9019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae961c0640 1 -- 172.21.15.28:0/1215589623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae90102790 msgr2=0x7fae90198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae961c0640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae90102790 0x7fae90198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae961c0640 1 -- 172.21.15.28:0/1215589623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae90106f10 msgr2=0x7fae9019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae961c0640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae90106f10 0x7fae9019cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae961c0640 1 -- 172.21.15.28:0/1215589623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9019d480 con 0x7fae90101de0 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae959bf640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae90102790 0x7fae90198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae961c0640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901984d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fae8c006010 tx=0x7fae8c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae877fe640 1 -- 172.21.15.28:0/1215589623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c03f030 con 0x7fae90101de0 2024-09-15T22:06:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9019d710 con 0x7fae90101de0 2024-09-15T22:06:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae877fe640 1 -- 172.21.15.28:0/1215589623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8c003ce0 con 0x7fae90101de0 2024-09-15T22:06:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae877fe640 1 -- 172.21.15.28:0/1215589623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c035070 con 0x7fae90101de0 2024-09-15T22:06:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.187+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae901a5050 con 0x7fae90101de0 2024-09-15T22:06:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 Processor -- start 2024-09-15T22:06:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae877fe640 1 -- 172.21.15.28:0/1215589623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fae8c042030 con 0x7fae90101de0 2024-09-15T22:06:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 -- start start 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739371090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x556739371a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x55673937e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567392d72f0 con 0x55673937c370 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55673927ecf0 con 0x5567393715d0 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5567392d32b0 con 0x556739372b40 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae959bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x55673937e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae959bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x55673937e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52220/0 (socket says 172.21.15.28:52220) 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae959bf640 1 -- 172.21.15.28:0/1747452022 learned_addr learned my addr 172.21.15.28:0/1747452022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae969c1640 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739371090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.190+0000 7fae961c0640 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x556739371a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae877fe640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fae7c0a33b0 0x7fae7c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae959bf640 1 -- 172.21.15.28:0/1747452022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 msgr2=0x556739371090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae959bf640 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739371090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae959bf640 1 -- 172.21.15.28:0/1747452022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 msgr2=0x556739371a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae959bf640 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x556739371a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae959bf640 1 -- 172.21.15.28:0/1747452022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567392633a0 con 0x55673937c370 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae961c0640 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x556739371a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae969c1640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fae7c0a33b0 0x7fae7c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae877fe640 1 -- 172.21.15.28:0/1215589623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fae8c0f07b0 con 0x7fae90101de0 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae959bf640 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x55673937e760 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fae8000bd30 tx=0x7fae8000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433dd367930e5dfd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae84dea640 1 -- 172.21.15.28:0/1747452022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80015070 con 0x55673937c370 2024-09-15T22:06:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae84dea640 1 -- 172.21.15.28:0/1747452022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8000be20 con 0x55673937c370 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.191+0000 7fae84dea640 1 -- 172.21.15.28:0/1747452022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80011a40 con 0x55673937c370 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1747452022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 msgr2=0x55673937e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae969c1640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fae7c0a33b0 0x7fae7c0a5870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fae8800bd50 tx=0x7fae88004a40 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x55673937e760 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fae8000bd30 tx=0x7fae8000f730 comp rx=0 tx=0).stop 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1747452022 shutdown_connections 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x55673937e760 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x556739371a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1747452022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739371090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1747452022 >> 172.21.15.28:0/1747452022 conn(0x55673933a310 msgr2=0x55673936dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1747452022 shutdown_connections 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1747452022 wait complete. 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.192+0000 7fae9a7fd740 1 -- start start 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 1 Processor -- start 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5567393715d0 0x55673937bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556739372b40 0x556739378fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x5567393795d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55673937f070 con 0x556739372b40 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556739292060 con 0x55673937c370 2024-09-15T22:06:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55673937f1f0 con 0x5567393715d0 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae9a7fd740 10 client.0 _ll_register_callbacks cb 0x556738fe58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556739372b40 0x556739378fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556739372b40 0x556739378fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52224/0 (socket says 172.21.15.28:52224) 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 -- 172.21.15.28:0/1700836573 learned_addr learned my addr 172.21.15.28:0/1700836573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae961c0640 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5567393715d0 0x55673937bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 -- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5567393715d0 msgr2=0x55673937bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5567393715d0 0x55673937bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 -- 172.21.15.28:0/1700836573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 msgr2=0x5567393795d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x5567393795d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 -- 172.21.15.28:0/1700836573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567390483c0 con 0x556739372b40 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae959bf640 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556739372b40 0x556739378fe0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fae80005270 tx=0x7fae800052a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcbd40f7e71aa6f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.193+0000 7fae79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae79ffb640 1 -- 172.21.15.28:0/1700836573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80028060 con 0x556739372b40 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae79ffb640 1 -- 172.21.15.28:0/1700836573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fae80011820 con 0x556739372b40 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae8000be20 con 0x556739372b40 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae79ffb640 1 -- 172.21.15.28:0/1700836573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fae8002f050 con 0x556739372b40 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 10 client.17313 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55673938eee0 con 0x556739372b40 2024-09-15T22:06:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae79ffb640 1 -- 172.21.15.28:0/1700836573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8000be20 con 0x556739372b40 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae79ffb640 1 -- 172.21.15.28:0/1700836573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fae80004c00 con 0x556739372b40 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 10 client.17313 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 10 client.17313 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 20 client.17313 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x556739392620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 4 client.17313 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.194+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567392e4860 con 0x556739390230 2024-09-15T22:06:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.195+0000 7fae961c0640 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x556739392620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.195+0000 7fae79ffb640 10 client.17313 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.195+0000 7fae961c0640 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x556739392620 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec1fb2ea79bb7ba0 server_cookie=9fa3f9e68e49e112 in_seq=0 out_seq=0 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae79ffb640 1 -- 172.21.15.28:0/1700836573 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5567392e4860 con 0x556739390230 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae79ffb640 10 client.17313 handle_command_reply: tid=0 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 client.17313 shutdown 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556739372b40 msgr2=0x556739378fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556739372b40 0x556739378fe0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fae80005270 tx=0x7fae800052a0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 shutdown_connections 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x556739392620 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x5567393795d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556739372b40 0x556739378fe0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae79ffb640 0 client.17313 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/1700836573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5567393715d0 0x55673937bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 >> 172.21.15.28:0/1700836573 conn(0x55673933a310 msgr2=0x556739373960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 shutdown_connections 2024-09-15T22:06:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 1 -- 172.21.15.28:0/1700836573 wait complete. 2024-09-15T22:06:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.196+0000 7fae9a7fd740 20 client.17313 trim_cache size 0 max 16384 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.209+0000 7fae9a7fd740 1 Processor -- start 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.209+0000 7fae9a7fd740 1 -- start start 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x55673938c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739372300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x5567393778b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae901acee0 con 0x55673937c370 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fae901aea50 con 0x5567393715d0 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fae901ad5b0 con 0x556739372b40 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x5567393778b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x5567393778b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52234/0 (socket says 172.21.15.28:52234) 2024-09-15T22:06:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 -- 172.21.15.28:0/3169568160 learned_addr learned my addr 172.21.15.28:0/3169568160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae961c0640 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x55673938c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae959bf640 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739372300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 -- 172.21.15.28:0/3169568160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 msgr2=0x556739372300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739372300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 -- 172.21.15.28:0/3169568160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 msgr2=0x55673938c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x55673938c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 -- 172.21.15.28:0/3169568160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556739377530 con 0x55673937c370 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae961c0640 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x55673938c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.210+0000 7fae969c1640 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x5567393778b0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fae88011470 tx=0x7fae88013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b811624f1a57a1d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae7bfff640 1 -- 172.21.15.28:0/3169568160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8801d840 con 0x55673937c370 2024-09-15T22:06:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae7bfff640 1 -- 172.21.15.28:0/3169568160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae88021030 con 0x55673937c370 2024-09-15T22:06:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae7bfff640 1 -- 172.21.15.28:0/3169568160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8801ae10 con 0x55673937c370 2024-09-15T22:06:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 -- 172.21.15.28:0/3169568160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 msgr2=0x5567393778b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x5567393778b0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fae88011470 tx=0x7fae88013070 comp rx=0 tx=0).stop 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 -- 172.21.15.28:0/3169568160 shutdown_connections 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55673937c370 0x5567393778b0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x556739372300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/3169568160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567393715d0 0x55673938c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 -- 172.21.15.28:0/3169568160 >> 172.21.15.28:0/3169568160 conn(0x55673933a310 msgr2=0x556739368a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 -- 172.21.15.28:0/3169568160 shutdown_connections 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.211+0000 7fae9a7fd740 1 -- 172.21.15.28:0/3169568160 wait complete. 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 -- start start 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 Processor -- start 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567393715d0 0x55673937cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x55673937d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x55673937d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae901ae380 con 0x5567393715d0 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fae901adcb0 con 0x55673937c370 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fae901aaee0 con 0x556739372b40 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae9a7fd740 10 client.0 _ll_register_callbacks cb 0x7fae8000bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae961c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567393715d0 0x55673937cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae961c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567393715d0 0x55673937cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52240/0 (socket says 172.21.15.28:52240) 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.212+0000 7fae961c0640 1 -- 172.21.15.28:0/954041278 learned_addr learned my addr 172.21.15.28:0/954041278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae959bf640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x55673937d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae969c1640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x55673937d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae961c0640 1 -- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 msgr2=0x55673937d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae961c0640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x55673937d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae961c0640 1 -- 172.21.15.28:0/954041278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 msgr2=0x55673937d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae961c0640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x55673937d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae961c0640 1 -- 172.21.15.28:0/954041278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556739387bb0 con 0x5567393715d0 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae969c1640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x55673937d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae961c0640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567393715d0 0x55673937cf50 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0edbb0 tx=0x7fae8c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c77bfb581b8b069 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae79ffb640 1 -- 172.21.15.28:0/954041278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c049040 con 0x5567393715d0 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae79ffb640 1 -- 172.21.15.28:0/954041278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fae8c0b8e10 con 0x5567393715d0 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae88021030 con 0x5567393715d0 2024-09-15T22:06:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae79ffb640 1 -- 172.21.15.28:0/954041278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fae8c03bcb0 con 0x5567393715d0 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae9a7fd740 10 client.17319 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556739388f10 con 0x5567393715d0 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.213+0000 7fae79ffb640 1 -- 172.21.15.28:0/954041278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae88021030 con 0x5567393715d0 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae79ffb640 1 -- 172.21.15.28:0/954041278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fae8c0f03b0 con 0x5567393715d0 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 10 client.17319 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 10 client.17319 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 20 client.17319 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x5567393fccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 4 client.17319 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567392ded70 con 0x556739390230 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae959bf640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x5567393fccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae959bf640 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x5567393fccb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d896b45539db4fdf server_cookie=4d29f537469d5ca5 in_seq=0 out_seq=0 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae79ffb640 10 client.17319 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae79ffb640 1 -- 172.21.15.28:0/954041278 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567392ded70 con 0x556739390230 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae79ffb640 10 client.17319 handle_command_reply: tid=0 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 1 client.17319 shutdown 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567393715d0 msgr2=0x55673937cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.214+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567393715d0 0x55673937cf50 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0edbb0 tx=0x7fae8c02a410 comp rx=0 tx=0).stop 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 shutdown_connections 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x556739390230 0x5567393fccb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55673937c370 0x55673937d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556739372b40 0x55673937d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 --2- 172.21.15.28:0/954041278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567393715d0 0x55673937cf50 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 >> 172.21.15.28:0/954041278 conn(0x55673933a310 msgr2=0x556739368a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae79ffb640 0 client.17319 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 shutdown_connections 2024-09-15T22:06:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 1 -- 172.21.15.28:0/954041278 wait complete. 2024-09-15T22:06:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae9a7fd740 20 client.17319 trim_cache size 0 max 16384 2024-09-15T22:06:26.224 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fae7c0a33b0 msgr2=0x7fae7c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fae7c0a33b0 0x7fae7c0a5870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fae8800bd50 tx=0x7fae88004a40 comp rx=0 tx=0).stop 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 msgr2=0x7fae901984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901984d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fae8c006010 tx=0x7fae8c0042f0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 shutdown_connections 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fae7c0a33b0 0x7fae7c0a5870 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae90106f10 0x7fae9019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae90102790 0x7fae90198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 --2- 172.21.15.28:0/1215589623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae90101de0 0x7fae901984d0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 >> 172.21.15.28:0/1215589623 conn(0x7fae900fd8c0 msgr2=0x7fae900fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 shutdown_connections 2024-09-15T22:06:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:26.215+0000 7fae98424640 1 -- 172.21.15.28:0/1215589623 wait complete. 2024-09-15T22:06:26.226 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:25 smithi203 ceph-mon[35546]: pgmap v279: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:26.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:25 smithi174 ceph-mon[35244]: pgmap v279: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:26.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:06:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 -- 172.21.15.28:0/3005454495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e81036e0 msgr2=0x7f79e8107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 --2- 172.21.15.28:0/3005454495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e81036e0 0x7f79e8107b90 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009930 tx=0x7f79dc02a420 comp rx=0 tx=0).stop 2024-09-15T22:06:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 -- 172.21.15.28:0/3005454495 shutdown_connections 2024-09-15T22:06:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 --2- 172.21.15.28:0/3005454495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79e81082a0 0x7f79e810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 --2- 172.21.15.28:0/3005454495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e81036e0 0x7f79e8107b90 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 --2- 172.21.15.28:0/3005454495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79e8102d30 0x7f79e8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 -- 172.21.15.28:0/3005454495 >> 172.21.15.28:0/3005454495 conn(0x7f79e80fc7d0 msgr2=0x7f79e80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 -- 172.21.15.28:0/3005454495 shutdown_connections 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.637+0000 7f79f02dc640 1 -- 172.21.15.28:0/3005454495 wait complete. 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 Processor -- start 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 -- start start 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e8102d30 0x7f79e81a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79e81036e0 0x7f79e81a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79e81082a0 0x7f79e819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81120e0 con 0x7f79e8102d30 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8111f60 con 0x7f79e81036e0 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79f02dc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79e8112260 con 0x7f79e81082a0 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e8102d30 0x7f79e81a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e8102d30 0x7f79e81a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52252/0 (socket says 172.21.15.28:52252) 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 -- 172.21.15.28:0/1206258686 learned_addr learned my addr 172.21.15.28:0/1206258686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ed877640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79e81036e0 0x7f79e81a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee879640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79e81082a0 0x7f79e819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 -- 172.21.15.28:0/1206258686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79e81082a0 msgr2=0x7f79e819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79e81082a0 0x7f79e819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 -- 172.21.15.28:0/1206258686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79e81036e0 msgr2=0x7f79e81a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79e81036e0 0x7f79e81a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 -- 172.21.15.28:0/1206258686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e819af00 con 0x7f79e8102d30 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.638+0000 7f79ee078640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e8102d30 0x7f79e81a05c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f79e400ea80 tx=0x7f79e400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.639+0000 7f79db7fe640 1 -- 172.21.15.28:0/1206258686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e400ce40 con 0x7f79e8102d30 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.639+0000 7f79db7fe640 1 -- 172.21.15.28:0/1206258686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e4005020 con 0x7f79e8102d30 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.639+0000 7f79db7fe640 1 -- 172.21.15.28:0/1206258686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4011760 con 0x7f79e8102d30 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.639+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e819b1f0 con 0x7f79e8102d30 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.639+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e81a7370 con 0x7f79e8102d30 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.641+0000 7f79db7fe640 1 -- 172.21.15.28:0/1206258686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f79e40051d0 con 0x7f79e8102d30 2024-09-15T22:06:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.641+0000 7f79f26b5740 1 Processor -- start 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.641+0000 7f79f26b5740 1 -- start start 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab93b40 0x55b3bab92090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab92a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79db7fe640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f79d40a3430 0x7f79d40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab9f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79f26b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3baaf82f0 con 0x55b3bab9d370 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79f26b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b3baa9fcf0 con 0x55b3bab93b40 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79f26b5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3baaf42b0 con 0x55b3bab925d0 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee078640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f79d40a3430 0x7f79d40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab9f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab9f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52258/0 (socket says 172.21.15.28:52258) 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 -- 172.21.15.28:0/2711036511 learned_addr learned my addr 172.21.15.28:0/2711036511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab93b40 0x55b3bab92090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79db7fe640 1 -- 172.21.15.28:0/1206258686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f79e40d05c0 con 0x7f79e8102d30 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ed877640 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab92a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 -- 172.21.15.28:0/2711036511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 msgr2=0x55b3bab92a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab92a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 -- 172.21.15.28:0/2711036511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab93b40 msgr2=0x55b3bab92090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab93b40 0x55b3bab92090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee879640 1 -- 172.21.15.28:0/2711036511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3baa843a0 con 0x55b3bab9d370 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.642+0000 7f79ee078640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f79d40a3430 0x7f79d40a58f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f79e40cdf60 tx=0x7f79e4089040 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79ee879640 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab9f760 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f79e0017730 tx=0x7f79e00094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4430b795a407de42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79d8dea640 1 -- 172.21.15.28:0/2711036511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e00155e0 con 0x55b3bab9d370 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79d8dea640 1 -- 172.21.15.28:0/2711036511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e001d070 con 0x55b3bab9d370 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79d8dea640 1 -- 172.21.15.28:0/2711036511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0018af0 con 0x55b3bab9d370 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79f26b5740 1 -- 172.21.15.28:0/2711036511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 msgr2=0x55b3bab9f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79f26b5740 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab9f760 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f79e0017730 tx=0x7f79e00094a0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79f26b5740 1 -- 172.21.15.28:0/2711036511 shutdown_connections 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.643+0000 7f79f26b5740 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab9f760 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab92a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 --2- 172.21.15.28:0/2711036511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab93b40 0x55b3bab92090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 -- 172.21.15.28:0/2711036511 >> 172.21.15.28:0/2711036511 conn(0x55b3bab5b310 msgr2=0x55b3bab8ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 -- 172.21.15.28:0/2711036511 shutdown_connections 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 -- 172.21.15.28:0/2711036511 wait complete. 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 -- start start 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 Processor -- start 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab9cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab99ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab9d370 0x55b3bab9a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3baba0010 con 0x55b3bab93b40 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b3baab3060 con 0x55b3bab9d370 2024-09-15T22:06:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3baba0190 con 0x55b3bab925d0 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79f26b5740 10 client.0 _ll_register_callbacks cb 0x55b3ba8068c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79ed877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab99ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.644+0000 7f79ed877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab99ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52260/0 (socket says 172.21.15.28:52260) 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ed877640 1 -- 172.21.15.28:0/3215829610 learned_addr learned my addr 172.21.15.28:0/3215829610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ee879640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab9cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ee879640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab9d370 0x55b3bab9a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ed877640 1 -- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 msgr2=0x55b3bab9cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ed877640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab9cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ed877640 1 -- 172.21.15.28:0/3215829610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab9d370 msgr2=0x55b3bab9a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ed877640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab9d370 0x55b3bab9a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ed877640 1 -- 172.21.15.28:0/3215829610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ba8693c0 con 0x55b3bab93b40 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ee879640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab9d370 0x55b3bab9a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79ed877640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab99ff0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f79dc000c00 tx=0x7f79dc0315d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393a58d666542f81 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79d1ffb640 1 -- 172.21.15.28:0/3215829610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc035040 con 0x55b3bab93b40 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79e001d070 con 0x55b3bab93b40 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79d1ffb640 1 -- 172.21.15.28:0/3215829610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f79dc004040 con 0x55b3bab93b40 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79d1ffb640 1 -- 172.21.15.28:0/3215829610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f79dc04f050 con 0x55b3bab93b40 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79f26b5740 10 client.17331 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.645+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b3bababa80 con 0x55b3bab93b40 2024-09-15T22:06:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79d1ffb640 1 -- 172.21.15.28:0/3215829610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e001d070 con 0x55b3bab93b40 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79d1ffb640 1 -- 172.21.15.28:0/3215829610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f79dc004890 con 0x55b3bab93b40 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79f26b5740 10 client.17331 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79f26b5740 10 client.17331 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79f26b5740 20 client.17331 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79f26b5740 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3babacf40 0x55b3babaf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79f26b5740 4 client.17331 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3bab05860 con 0x55b3babacf40 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.646+0000 7f79ee879640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3babacf40 0x55b3babaf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.647+0000 7f79ee879640 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3babacf40 0x55b3babaf380 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ed041f1d9f443fd server_cookie=df248727b1bbfef1 in_seq=0 out_seq=0 2024-09-15T22:06:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.647+0000 7f79d1ffb640 10 client.17331 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.647+0000 7f79d1ffb640 1 -- 172.21.15.28:0/3215829610 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b3bab05860 con 0x55b3babacf40 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.647+0000 7f79d1ffb640 10 client.17331 handle_command_reply: tid=0 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.647+0000 7f79f26b5740 1 client.17331 shutdown 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 msgr2=0x55b3bab99ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab99ff0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f79dc000c00 tx=0x7f79dc0315d0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 shutdown_connections 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3babacf40 0x55b3babaf380 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab9d370 0x55b3bab9a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab99ff0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 --2- 172.21.15.28:0/3215829610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab925d0 0x55b3bab9cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79d1ffb640 0 client.17331 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 >> 172.21.15.28:0/3215829610 conn(0x55b3bab5b310 msgr2=0x55b3bab94960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 shutdown_connections 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 1 -- 172.21.15.28:0/3215829610 wait complete. 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.648+0000 7f79f26b5740 20 client.17331 trim_cache size 0 max 16384 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 Processor -- start 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 -- start start 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3bab8ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab93b40 0x55b3baba92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab98fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81b12e0 con 0x55b3bab9d370 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81b2e50 con 0x55b3bab925d0 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79f26b5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81b19b0 con 0x55b3bab93b40 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab98fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab98fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52272/0 (socket says 172.21.15.28:52272) 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 -- 172.21.15.28:0/594250621 learned_addr learned my addr 172.21.15.28:0/594250621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ed877640 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab93b40 0x55b3baba92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 -- 172.21.15.28:0/594250621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab93b40 msgr2=0x55b3baba92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab93b40 0x55b3baba92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 -- 172.21.15.28:0/594250621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 msgr2=0x55b3bab8ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3bab8ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.661+0000 7f79ee879640 1 -- 172.21.15.28:0/594250621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3bab95120 con 0x55b3bab9d370 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.662+0000 7f79ee879640 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab98fd0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f79e00187a0 tx=0x7f79e0018690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe4cb156c0f6924 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.662+0000 7f79d3fff640 1 -- 172.21.15.28:0/594250621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0032070 con 0x55b3bab9d370 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.662+0000 7f79d3fff640 1 -- 172.21.15.28:0/594250621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e001d040 con 0x55b3bab9d370 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.662+0000 7f79d3fff640 1 -- 172.21.15.28:0/594250621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e0015b90 con 0x55b3bab9d370 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.662+0000 7f79f26b5740 1 -- 172.21.15.28:0/594250621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 msgr2=0x55b3bab98fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.662+0000 7f79f26b5740 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab98fd0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f79e00187a0 tx=0x7f79e0018690 comp rx=0 tx=0).stop 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 -- 172.21.15.28:0/594250621 shutdown_connections 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab9d370 0x55b3bab98fd0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab93b40 0x55b3baba92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 --2- 172.21.15.28:0/594250621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3bab8ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 -- 172.21.15.28:0/594250621 >> 172.21.15.28:0/594250621 conn(0x55b3bab5b310 msgr2=0x55b3bab988c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 -- 172.21.15.28:0/594250621 shutdown_connections 2024-09-15T22:06:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 -- 172.21.15.28:0/594250621 wait complete. 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.663+0000 7f79f26b5740 1 -- start start 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 1 Processor -- start 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3baba58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab96a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab9d370 0x55b3bab96fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81b2780 con 0x55b3bab93b40 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81b20b0 con 0x55b3bab925d0 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79e81af2e0 con 0x55b3bab9d370 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79f26b5740 10 client.0 _ll_register_callbacks cb 0x7f79dc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab96a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab96a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52286/0 (socket says 172.21.15.28:52286) 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 -- 172.21.15.28:0/787759434 learned_addr learned my addr 172.21.15.28:0/787759434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ee879640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3baba58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ee879640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab9d370 0x55b3bab96fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 -- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab9d370 msgr2=0x55b3bab96fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab9d370 0x55b3bab96fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 -- 172.21.15.28:0/787759434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 msgr2=0x55b3baba58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3baba58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 -- 172.21.15.28:0/787759434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3bab974e0 con 0x55b3bab93b40 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ee879640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3baba58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.664+0000 7f79ed877640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab96a60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f79dc02a940 tx=0x7f79dc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d1bf53818b70ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79d1ffb640 1 -- 172.21.15.28:0/787759434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc035070 con 0x55b3bab93b40 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79d1ffb640 1 -- 172.21.15.28:0/787759434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f79dc004770 con 0x55b3bab93b40 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79e001d040 con 0x55b3bab93b40 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79d1ffb640 1 -- 172.21.15.28:0/787759434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f79dc060050 con 0x55b3bab93b40 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 10 client.17337 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55b3bac17970 con 0x55b3bab93b40 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79d1ffb640 1 -- 172.21.15.28:0/787759434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e001d040 con 0x55b3bab93b40 2024-09-15T22:06:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79d1ffb640 1 -- 172.21.15.28:0/787759434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f79dc05b030 con 0x55b3bab93b40 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 10 client.17337 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 10 client.17337 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 20 client.17337 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3bac18170 0x55b3bac1a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 4 client.17337 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.665+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3baaffd70 con 0x55b3bac18170 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79ee879640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3bac18170 0x55b3bac1a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79ee879640 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3bac18170 0x55b3bac1a590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=92a91625eb383af7 server_cookie=1a8f6a1410bd42ad in_seq=0 out_seq=0 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79d1ffb640 10 client.17337 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79d1ffb640 1 -- 172.21.15.28:0/787759434 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3baaffd70 con 0x55b3bac18170 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79d1ffb640 10 client.17337 handle_command_reply: tid=0 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 client.17337 shutdown 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 msgr2=0x55b3bab96a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab96a60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f79dc02a940 tx=0x7f79dc003820 comp rx=0 tx=0).stop 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 shutdown_connections 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55b3bac18170 0x55b3bac1a590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b3bab9d370 0x55b3bab96fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3bab93b40 0x55b3bab96a60 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79f26b5740 1 --2- 172.21.15.28:0/787759434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3bab925d0 0x55b3baba58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.666+0000 7f79d1ffb640 0 client.17337 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 >> 172.21.15.28:0/787759434 conn(0x55b3bab5b310 msgr2=0x55b3bab990e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 shutdown_connections 2024-09-15T22:06:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f26b5740 1 -- 172.21.15.28:0/787759434 wait complete. 2024-09-15T22:06:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f26b5740 20 client.17337 trim_cache size 0 max 16384 2024-09-15T22:06:27.736 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f79d40a3430 msgr2=0x7f79d40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f79d40a3430 0x7f79d40a58f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f79e40cdf60 tx=0x7f79e4089040 comp rx=0 tx=0).stop 2024-09-15T22:06:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e8102d30 msgr2=0x7f79e81a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e8102d30 0x7f79e81a05c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f79e400ea80 tx=0x7f79e400ef40 comp rx=0 tx=0).stop 2024-09-15T22:06:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 shutdown_connections 2024-09-15T22:06:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f79d40a3430 0x7f79d40a58f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79e81082a0 0x7f79e819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79e81036e0 0x7f79e81a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 --2- 172.21.15.28:0/1206258686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79e8102d30 0x7f79e81a05c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 >> 172.21.15.28:0/1206258686 conn(0x7f79e80fc7d0 msgr2=0x7f79e810cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.667+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 shutdown_connections 2024-09-15T22:06:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:27.668+0000 7f79f02dc640 1 -- 172.21.15.28:0/1206258686 wait complete. 2024-09-15T22:06:28.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:27 smithi028 ceph-mon[30969]: pgmap v280: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:28.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:27 smithi203 ceph-mon[35546]: pgmap v280: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:28.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:27 smithi174 ceph-mon[35244]: pgmap v280: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-09-15T22:06:28.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:06:28.656 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 -- 172.21.15.28:0/2267545947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f094410c170 msgr2=0x7f094410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 --2- 172.21.15.28:0/2267545947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f094410c170 0x7f094410e610 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f093c009ca0 tx=0x7f093c02a950 comp rx=0 tx=0).stop 2024-09-15T22:06:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 -- 172.21.15.28:0/2267545947 shutdown_connections 2024-09-15T22:06:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 --2- 172.21.15.28:0/2267545947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f094410c170 0x7f094410e610 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 --2- 172.21.15.28:0/2267545947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0944103660 0x7f094410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 --2- 172.21.15.28:0/2267545947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0944102d40 0x7f0944103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 -- 172.21.15.28:0/2267545947 >> 172.21.15.28:0/2267545947 conn(0x7f09440fc7a0 msgr2=0x7f09440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 -- 172.21.15.28:0/2267545947 shutdown_connections 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 -- 172.21.15.28:0/2267545947 wait complete. 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 Processor -- start 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.157+0000 7f094a943640 1 -- start start 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f094a943640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0944102d40 0x7f0944198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f094a943640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0944103660 0x7f09441989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f094a943640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f094410c170 0x7f094419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f094a943640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0944111fd0 con 0x7f0944103660 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f094a943640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0944111e50 con 0x7f094410c170 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f094a943640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0944112150 con 0x7f0944102d40 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0944103660 0x7f09441989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0944103660 0x7f09441989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52296/0 (socket says 172.21.15.28:52296) 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 -- 172.21.15.28:0/2784217334 learned_addr learned my addr 172.21.15.28:0/2784217334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f094410c170 0x7f094419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f0943fff640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0944102d40 0x7f0944198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 -- 172.21.15.28:0/2784217334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0944102d40 msgr2=0x7f0944198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0944102d40 0x7f0944198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 -- 172.21.15.28:0/2784217334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f094410c170 msgr2=0x7f094419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f094410c170 0x7f094419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 -- 172.21.15.28:0/2784217334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094419d420 con 0x7f0944103660 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f094410c170 0x7f094419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f0943fff640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0944102d40 0x7f0944198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09437fe640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0944103660 0x7f09441989b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f093800d750 tx=0x7f093800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09417fa640 1 -- 172.21.15.28:0/2784217334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938017070 con 0x7f0944103660 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094419d710 con 0x7f0944103660 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09417fa640 1 -- 172.21.15.28:0/2784217334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0938005020 con 0x7f0944103660 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.158+0000 7f09417fa640 1 -- 172.21.15.28:0/2784217334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938004d00 con 0x7f0944103660 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.159+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09441a4ff0 con 0x7f0944103660 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.161+0000 7f09417fa640 1 -- 172.21.15.28:0/2784217334 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f0938009a30 con 0x7f0944103660 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.161+0000 7f094cd1c740 1 Processor -- start 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.161+0000 7f094cd1c740 1 -- start start 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f09417fa640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f092c0a33b0 0x7f092c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93d8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93d8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93e5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f094cd1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5581b933e2f0 con 0x5581b93d85d0 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f094cd1c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5581b92e5cf0 con 0x5581b93d9b40 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f094cd1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5581b933a2b0 con 0x5581b93e3370 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f092c0a33b0 0x7f092c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93d8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:36856/0 (socket says 172.21.15.28:36856) 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f09437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93d8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 -- 172.21.15.28:0/2815533237 learned_addr learned my addr 172.21.15.28:0/2815533237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f09417fa640 1 -- 172.21.15.28:0/2784217334 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f09380cfc50 con 0x7f0944103660 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 -- 172.21.15.28:0/2815533237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 msgr2=0x5581b93e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 -- 172.21.15.28:0/2815533237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 msgr2=0x5581b93d8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93d8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0943fff640 1 -- 172.21.15.28:0/2815533237 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b92ca3a0 con 0x5581b93d9b40 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.162+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f092c0a33b0 0x7f092c0a5870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f093c005cc0 tx=0x7f093c02fb00 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.163+0000 7f0943fff640 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93d8090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0934007850 tx=0x7f0934007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bc804a92234d61e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.163+0000 7f092a7fc640 1 -- 172.21.15.28:0/2815533237 <== mon.2 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093400d950 con 0x5581b93d9b40 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.163+0000 7f092a7fc640 1 -- 172.21.15.28:0/2815533237 <== mon.2 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0934018070 con 0x5581b93d9b40 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.163+0000 7f092a7fc640 1 -- 172.21.15.28:0/2815533237 <== mon.2 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09340123e0 con 0x5581b93d9b40 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.163+0000 7f094cd1c740 1 -- 172.21.15.28:0/2815533237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 msgr2=0x5581b93d8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.163+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93d8090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0934007850 tx=0x7f0934007d10 comp rx=0 tx=0).stop 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- 172.21.15.28:0/2815533237 shutdown_connections 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2815533237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93d8090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- 172.21.15.28:0/2815533237 >> 172.21.15.28:0/2815533237 conn(0x5581b93a1310 msgr2=0x5581b93d4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- 172.21.15.28:0/2815533237 shutdown_connections 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- 172.21.15.28:0/2815533237 wait complete. 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- start start 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 Processor -- start 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d9b40 0x5581b93dffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93e0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5581b93e6070 con 0x5581b93d85d0 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5581b92f9060 con 0x5581b93e3370 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5581b93e61f0 con 0x5581b93d9b40 2024-09-15T22:06:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.164+0000 7f094cd1c740 10 client.0 _ll_register_callbacks cb 0x5581b904c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f0943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f0943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93e2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52310/0 (socket says 172.21.15.28:52310) 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f0943fff640 1 -- 172.21.15.28:0/2540471446 learned_addr learned my addr 172.21.15.28:0/2540471446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f0948ee0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93e0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f09437fe640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d9b40 0x5581b93dffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f0943fff640 1 -- 172.21.15.28:0/2540471446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d9b40 msgr2=0x5581b93dffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f0943fff640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d9b40 0x5581b93dffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.165+0000 7f0943fff640 1 -- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 msgr2=0x5581b93e0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f0943fff640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93e0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f0943fff640 1 -- 172.21.15.28:0/2540471446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b90af3c0 con 0x5581b93d85d0 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f09437fe640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d9b40 0x5581b93dffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93e0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f0943fff640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93e2d10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f0934000c00 tx=0x7f09340136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9752cac31bb2cc21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f09237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f09237fe640 1 -- 172.21.15.28:0/2540471446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093400dd60 con 0x5581b93d85d0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f09237fe640 1 -- 172.21.15.28:0/2540471446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f0934024420 con 0x5581b93d85d0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0934018070 con 0x5581b93d85d0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f09237fe640 1 -- 172.21.15.28:0/2540471446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f093402f050 con 0x5581b93d85d0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f094cd1c740 10 client.17349 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.166+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5581b93f5ed0 con 0x5581b93d85d0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f09237fe640 1 -- 172.21.15.28:0/2540471446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0934018070 con 0x5581b93d85d0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f09237fe640 1 -- 172.21.15.28:0/2540471446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f093401c5b0 con 0x5581b93d85d0 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f094cd1c740 10 client.17349 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f094cd1c740 10 client.17349 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f094cd1c740 20 client.17349 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b93f7330 0x5581b93f9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f094cd1c740 4 client.17349 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581b934b860 con 0x5581b93f7330 2024-09-15T22:06:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.167+0000 7f09437fe640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b93f7330 0x5581b93f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.168+0000 7f09437fe640 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b93f7330 0x5581b93f9720 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=410a44de0e36cb7c server_cookie=e82cabbaf8ce1558 in_seq=0 out_seq=0 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.168+0000 7f09237fe640 10 client.17349 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.168+0000 7f09237fe640 1 -- 172.21.15.28:0/2540471446 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581b934b860 con 0x5581b93f7330 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.168+0000 7f09237fe640 10 client.17349 handle_command_reply: tid=0 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.168+0000 7f094cd1c740 1 client.17349 shutdown 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 msgr2=0x5581b93e2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93e2d10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f0934000c00 tx=0x7f09340136b0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 shutdown_connections 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b93f7330 0x5581b93f9720 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93e0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d9b40 0x5581b93dffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f09237fe640 0 client.17349 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2540471446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93e2d10 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 >> 172.21.15.28:0/2540471446 conn(0x5581b93a1310 msgr2=0x5581b93da960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 shutdown_connections 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 1 -- 172.21.15.28:0/2540471446 wait complete. 2024-09-15T22:06:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.169+0000 7f094cd1c740 20 client.17349 trim_cache size 0 max 16384 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 Processor -- start 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 -- start start 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d85d0 0x5581b93f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d9b40 0x5581b93db120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93db660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09441ace70 con 0x5581b93d9b40 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09441ae9e0 con 0x5581b93e3370 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f094cd1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09441ad540 con 0x5581b93d85d0 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f09437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d9b40 0x5581b93db120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f09437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d9b40 0x5581b93db120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52322/0 (socket says 172.21.15.28:52322) 2024-09-15T22:06:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.182+0000 7f09437fe640 1 -- 172.21.15.28:0/2609828268 learned_addr learned my addr 172.21.15.28:0/2609828268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93db660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f0943fff640 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d85d0 0x5581b93f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f09437fe640 1 -- 172.21.15.28:0/2609828268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d85d0 msgr2=0x5581b93f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f09437fe640 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d85d0 0x5581b93f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f09437fe640 1 -- 172.21.15.28:0/2609828268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 msgr2=0x5581b93db660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f09437fe640 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93db660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f09437fe640 1 -- 172.21.15.28:0/2609828268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b93d3310 con 0x5581b93d9b40 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f0948ee0640 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f09437fe640 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d9b40 0x5581b93db120 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f0938097f80 tx=0x7f0938086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2a49aff3f22f9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f0929ffb640 1 -- 172.21.15.28:0/2609828268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938020040 con 0x5581b93d9b40 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f0929ffb640 1 -- 172.21.15.28:0/2609828268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0938099bc0 con 0x5581b93d9b40 2024-09-15T22:06:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.183+0000 7f0929ffb640 1 -- 172.21.15.28:0/2609828268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09380052f0 con 0x5581b93d9b40 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 -- 172.21.15.28:0/2609828268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d9b40 msgr2=0x5581b93db120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d9b40 0x5581b93db120 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f0938097f80 tx=0x7f0938086040 comp rx=0 tx=0).stop 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 -- 172.21.15.28:0/2609828268 shutdown_connections 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93e3370 0x5581b93db660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d9b40 0x5581b93db120 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 --2- 172.21.15.28:0/2609828268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93d85d0 0x5581b93f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 -- 172.21.15.28:0/2609828268 >> 172.21.15.28:0/2609828268 conn(0x5581b93a1310 msgr2=0x5581b93df0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:29.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 -- 172.21.15.28:0/2609828268 shutdown_connections 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 -- 172.21.15.28:0/2609828268 wait complete. 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 -- start start 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.184+0000 7f094cd1c740 1 Processor -- start 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93eff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93eb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09441ae310 con 0x5581b93d85d0 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09441adc40 con 0x5581b93d9b40 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09441aae70 con 0x5581b93e3370 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 10 client.0 _ll_register_callbacks cb 0x7f0938004ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93eff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52326/0 (socket says 172.21.15.28:52326) 2024-09-15T22:06:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 -- 172.21.15.28:0/4003434987 learned_addr learned my addr 172.21.15.28:0/4003434987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f09437fe640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0948ee0640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 -- 172.21.15.28:0/4003434987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 msgr2=0x5581b93eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 -- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 msgr2=0x5581b93eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 -- 172.21.15.28:0/4003434987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b93ebef0 con 0x5581b93d85d0 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0948ee0640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f0943fff640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93eff40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f0934007d50 tx=0x7f093401cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892cf75f299776b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f09237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f09237fe640 1 -- 172.21.15.28:0/4003434987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934037070 con 0x5581b93d85d0 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f09237fe640 1 -- 172.21.15.28:0/4003434987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f093400dd60 con 0x5581b93d85d0 2024-09-15T22:06:29.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.185+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0938099bc0 con 0x5581b93d85d0 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f09237fe640 1 -- 172.21.15.28:0/4003434987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0934040050 con 0x5581b93d85d0 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 10 client.17355 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5581b9461bd0 con 0x5581b93d85d0 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f09237fe640 1 -- 172.21.15.28:0/4003434987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0938099bc0 con 0x5581b93d85d0 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f09237fe640 1 -- 172.21.15.28:0/4003434987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f093403a020 con 0x5581b93d85d0 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 10 client.17355 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 10 client.17355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 20 client.17355 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b94622e0 0x5581b9464700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 4 client.17355 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:29.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5581b9345d70 con 0x5581b94622e0 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.186+0000 7f09437fe640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b94622e0 0x5581b9464700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f09437fe640 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b94622e0 0x5581b9464700 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed5ca78906b42eb server_cookie=f74d1659eebc53d2 in_seq=0 out_seq=0 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f09237fe640 10 client.17355 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f09237fe640 1 -- 172.21.15.28:0/4003434987 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5581b9345d70 con 0x5581b94622e0 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f09237fe640 10 client.17355 handle_command_reply: tid=0 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 client.17355 shutdown 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 msgr2=0x5581b93eff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93eff40 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f0934007d50 tx=0x7f093401cea0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 shutdown_connections 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5581b94622e0 0x5581b9464700 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5581b93e3370 0x5581b93eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581b93d9b40 0x5581b93eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 --2- 172.21.15.28:0/4003434987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b93d85d0 0x5581b93eff40 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 >> 172.21.15.28:0/4003434987 conn(0x5581b93a1310 msgr2=0x5581b93e1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:29.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f09237fe640 0 client.17355 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:29.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 shutdown_connections 2024-09-15T22:06:29.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 1 -- 172.21.15.28:0/4003434987 wait complete. 2024-09-15T22:06:29.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.187+0000 7f094cd1c740 20 client.17355 trim_cache size 0 max 16384 2024-09-15T22:06:29.198 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f092c0a33b0 msgr2=0x7f092c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f092c0a33b0 0x7f092c0a5870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f093c005cc0 tx=0x7f093c02fb00 comp rx=0 tx=0).stop 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0944103660 msgr2=0x7f09441989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0944103660 0x7f09441989b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f093800d750 tx=0x7f093800dc10 comp rx=0 tx=0).stop 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 shutdown_connections 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f092c0a33b0 0x7f092c0a5870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f094410c170 0x7f094419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0944103660 0x7f09441989b0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 --2- 172.21.15.28:0/2784217334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0944102d40 0x7f0944198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:29.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 >> 172.21.15.28:0/2784217334 conn(0x7f09440fc7a0 msgr2=0x7f09440fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:29.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 shutdown_connections 2024-09-15T22:06:29.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:29.188+0000 7f094a943640 1 -- 172.21.15.28:0/2784217334 wait complete. 2024-09-15T22:06:30.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:29 smithi028 ceph-mon[30969]: pgmap v281: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-09-15T22:06:30.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:29 smithi203 ceph-mon[35546]: pgmap v281: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-09-15T22:06:30.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:29 smithi174 ceph-mon[35244]: pgmap v281: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-09-15T22:06:30.969 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T22:06:30.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T22:06:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 -- 172.21.15.28:0/637981854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 msgr2=0x7f0334101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 --2- 172.21.15.28:0/637981854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334101200 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f0324009900 tx=0x7f032402a480 comp rx=0 tx=0).stop 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 -- 172.21.15.28:0/637981854 shutdown_connections 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 --2- 172.21.15.28:0/637981854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334102ef0 0x7f033410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 --2- 172.21.15.28:0/637981854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0334102570 0x7f03341029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 --2- 172.21.15.28:0/637981854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334101200 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 -- 172.21.15.28:0/637981854 >> 172.21.15.28:0/637981854 conn(0x7f03340fc7d0 msgr2=0x7f03340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 -- 172.21.15.28:0/637981854 shutdown_connections 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 -- 172.21.15.28:0/637981854 wait complete. 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.737+0000 7f033a6d6640 1 Processor -- start 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f033a6d6640 1 -- start start 2024-09-15T22:06:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f033a6d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f033a6d6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0334102570 0x7f0334198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f033a6d6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334102ef0 0x7f033419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f033a6d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0334110f50 con 0x7f0334100e20 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f033a6d6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0334110dd0 con 0x7f0334102570 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f033a6d6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03341110d0 con 0x7f0334102ef0 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45024/0 (socket says 172.21.15.28:45024) 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 -- 172.21.15.28:0/1696944641 learned_addr learned my addr 172.21.15.28:0/1696944641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0338c73640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334102ef0 0x7f033419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 -- 172.21.15.28:0/1696944641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334102ef0 msgr2=0x7f033419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334102ef0 0x7f033419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 -- 172.21.15.28:0/1696944641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0334102570 msgr2=0x7f0334198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0334102570 0x7f0334198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0333fff640 1 -- 172.21.15.28:0/1696944641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033419d3f0 con 0x7f0334100e20 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.738+0000 7f0338c73640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334102ef0 0x7f033419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.739+0000 7f0333fff640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334198440 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f0324006010 tx=0x7f0324003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.739+0000 7f0331ffb640 1 -- 172.21.15.28:0/1696944641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032403f070 con 0x7f0334100e20 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.739+0000 7f0331ffb640 1 -- 172.21.15.28:0/1696944641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0324031e30 con 0x7f0334100e20 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.739+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f033419d680 con 0x7f0334100e20 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.739+0000 7f0331ffb640 1 -- 172.21.15.28:0/1696944641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324035030 con 0x7f0334100e20 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.739+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03341a4fc0 con 0x7f0334100e20 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.741+0000 7f0331ffb640 1 -- 172.21.15.28:0/1696944641 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f0324042070 con 0x7f0334100e20 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.741+0000 7f033caaf740 1 Processor -- start 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.741+0000 7f033caaf740 1 -- start start 2024-09-15T22:06:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f033caaf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3e8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f033caaf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f033caaf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3fa540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f033caaf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a60c34e280 con 0x55a60c3e8a60 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f033caaf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a60c2f5b10 con 0x55a60c3e9380 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f033caaf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a60c34a240 con 0x55a60c3f8150 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3e8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3e8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45040/0 (socket says 172.21.15.28:45040) 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 -- 172.21.15.28:0/2413142929 learned_addr learned my addr 172.21.15.28:0/2413142929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0331ffb640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f032bfff640 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0333fff640 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0331ffb640 1 -- 172.21.15.28:0/1696944641 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f03240f0ab0 con 0x7f0334100e20 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 -- 172.21.15.28:0/2413142929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 msgr2=0x55a60c3fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3fa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 -- 172.21.15.28:0/2413142929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 msgr2=0x55a60c3f7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.742+0000 7f0338c73640 1 -- 172.21.15.28:0/2413142929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a60c3021f0 con 0x55a60c3e8a60 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.743+0000 7f0338c73640 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3e8e40 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f032c012940 tx=0x7f032c012d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b5627ddfd6a719 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.743+0000 7f0338c73640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f032c019000 tx=0x7f032c01b000 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.743+0000 7f032a7fc640 1 -- 172.21.15.28:0/2413142929 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c004d20 con 0x55a60c3e8a60 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.743+0000 7f032a7fc640 1 -- 172.21.15.28:0/2413142929 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f032c005510 con 0x55a60c3e8a60 2024-09-15T22:06:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.743+0000 7f032a7fc640 1 -- 172.21.15.28:0/2413142929 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032c01c810 con 0x55a60c3e8a60 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.743+0000 7f033caaf740 1 -- 172.21.15.28:0/2413142929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 msgr2=0x55a60c3e8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.743+0000 7f033caaf740 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3e8e40 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f032c012940 tx=0x7f032c012d40 comp rx=0 tx=0).stop 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 -- 172.21.15.28:0/2413142929 shutdown_connections 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 --2- 172.21.15.28:0/2413142929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3e8e40 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 -- 172.21.15.28:0/2413142929 >> 172.21.15.28:0/2413142929 conn(0x55a60c3e50f0 msgr2=0x55a60c3e54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 -- 172.21.15.28:0/2413142929 shutdown_connections 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 -- 172.21.15.28:0/2413142929 wait complete. 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 -- start start 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 Processor -- start 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.744+0000 7f033caaf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f033caaf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a60c3facc0 con 0x55a60c3e8a60 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f033caaf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a60c308f10 con 0x55a60c3e9380 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f033caaf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a60c3fae40 con 0x55a60c3f8150 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f033caaf740 10 client.0 _ll_register_callbacks cb 0x55a60c2d1a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3f50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45056/0 (socket says 172.21.15.28:45056) 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 -- 172.21.15.28:0/2368449466 learned_addr learned my addr 172.21.15.28:0/2368449466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f032bfff640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0338c73640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 -- 172.21.15.28:0/2368449466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 msgr2=0x55a60c3f5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3f5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 -- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 msgr2=0x55a60c3f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 -- 172.21.15.28:0/2368449466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a60c0bf6a0 con 0x55a60c3e8a60 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f032bfff640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f0333fff640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3f50b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f03240bc680 tx=0x7f032402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2db15abaf6ec0e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f03137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f03137fe640 1 -- 172.21.15.28:0/2368449466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324049030 con 0x55a60c3e8a60 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f03137fe640 1 -- 172.21.15.28:0/2368449466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f03240ee070 con 0x55a60c3e8a60 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f032c005510 con 0x55a60c3e8a60 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f03137fe640 1 -- 172.21.15.28:0/2368449466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f03240a70d0 con 0x55a60c3e8a60 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f033caaf740 10 client.17367 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.745+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a60c406980 con 0x55a60c3e8a60 2024-09-15T22:06:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f03137fe640 1 -- 172.21.15.28:0/2368449466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f032c005510 con 0x55a60c3e8a60 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f03137fe640 1 -- 172.21.15.28:0/2368449466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f03240c5160 con 0x55a60c3e8a60 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f033caaf740 10 client.17367 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f033caaf740 10 client.17367 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f033caaf740 20 client.17367 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f033caaf740 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c407d70 0x55a60c40a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f033caaf740 4 client.17367 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.746+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a60c35b7f0 con 0x55a60c407d70 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.747+0000 7f032bfff640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c407d70 0x55a60c40a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.748+0000 7f032bfff640 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c407d70 0x55a60c40a200 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92c545145f61d785 server_cookie=74814fe1d9437d14 in_seq=0 out_seq=0 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.748+0000 7f03137fe640 10 client.17367 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.748+0000 7f03137fe640 1 -- 172.21.15.28:0/2368449466 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a60c35b7f0 con 0x55a60c407d70 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.748+0000 7f03137fe640 10 client.17367 handle_command_reply: tid=0 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.748+0000 7f033caaf740 1 client.17367 shutdown 2024-09-15T22:06:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 msgr2=0x55a60c3f50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3f50b0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f03240bc680 tx=0x7f032402a480 comp rx=0 tx=0).stop 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 shutdown_connections 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c407d70 0x55a60c40a200 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 --2- 172.21.15.28:0/2368449466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3f50b0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f03137fe640 0 client.17367 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 >> 172.21.15.28:0/2368449466 conn(0x55a60c3e50f0 msgr2=0x55a60c3eae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 shutdown_connections 2024-09-15T22:06:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 1 -- 172.21.15.28:0/2368449466 wait complete. 2024-09-15T22:06:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.749+0000 7f033caaf740 20 client.17367 trim_cache size 0 max 16384 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.766+0000 7f033caaf740 1 Processor -- start 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.766+0000 7f033caaf740 1 -- start start 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.766+0000 7f033caaf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c40fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.766+0000 7f033caaf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c410010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f033caaf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3f2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f033caaf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03341acc60 con 0x55a60c3e8a60 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f033caaf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03341ae7d0 con 0x55a60c3e9380 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f033caaf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03341ad330 con 0x55a60c3f8150 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c40fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c40fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45068/0 (socket says 172.21.15.28:45068) 2024-09-15T22:06:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 -- 172.21.15.28:0/3372294813 learned_addr learned my addr 172.21.15.28:0/3372294813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 -- 172.21.15.28:0/3372294813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 msgr2=0x55a60c3f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 --2- 172.21.15.28:0/3372294813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 -- 172.21.15.28:0/3372294813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 msgr2=0x55a60c410010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 --2- 172.21.15.28:0/3372294813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c410010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 -- 172.21.15.28:0/3372294813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a60c3f6080 con 0x55a60c3e8a60 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0333fff640 1 --2- 172.21.15.28:0/3372294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c40fad0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f03240d2040 tx=0x7f0324005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dedc7cfb41ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0329ffb640 1 -- 172.21.15.28:0/3372294813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324049070 con 0x55a60c3e8a60 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0329ffb640 1 -- 172.21.15.28:0/3372294813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03240c55f0 con 0x55a60c3e8a60 2024-09-15T22:06:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.767+0000 7f0329ffb640 1 -- 172.21.15.28:0/3372294813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240ee070 con 0x55a60c3e8a60 2024-09-15T22:06:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 -- 172.21.15.28:0/3372294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 msgr2=0x55a60c40fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 --2- 172.21.15.28:0/3372294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c40fad0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f03240d2040 tx=0x7f0324005c00 comp rx=0 tx=0).stop 2024-09-15T22:06:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 -- 172.21.15.28:0/3372294813 shutdown_connections 2024-09-15T22:06:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 --2- 172.21.15.28:0/3372294813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c3f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 --2- 172.21.15.28:0/3372294813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c410010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 --2- 172.21.15.28:0/3372294813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c40fad0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 -- 172.21.15.28:0/3372294813 >> 172.21.15.28:0/3372294813 conn(0x55a60c3e50f0 msgr2=0x55a60c3eae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 -- 172.21.15.28:0/3372294813 shutdown_connections 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 -- 172.21.15.28:0/3372294813 wait complete. 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.768+0000 7f033caaf740 1 -- start start 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 1 Processor -- start 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3fc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3fca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c471aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03341ae100 con 0x55a60c3e8a60 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03341ada30 con 0x55a60c3e9380 2024-09-15T22:06:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03341aac60 con 0x55a60c3f8150 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f033caaf740 10 client.0 _ll_register_callbacks cb 0x7f0324003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45074/0 (socket says 172.21.15.28:45074) 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 -- 172.21.15.28:0/3010440896 learned_addr learned my addr 172.21.15.28:0/3010440896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f032bfff640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3fca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0338c73640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c471aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 -- 172.21.15.28:0/3010440896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 msgr2=0x55a60c471aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c471aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 -- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 msgr2=0x55a60c3fca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3fca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 -- 172.21.15.28:0/3010440896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a60c471fe0 con 0x55a60c3e8a60 2024-09-15T22:06:31.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0338c73640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c471aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f032bfff640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3fca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f0333fff640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3fc510 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f03240d2010 tx=0x7f03240a7270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37b58785fefaf91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.769+0000 7f03137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f03137fe640 1 -- 172.21.15.28:0/3010440896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324049070 con 0x55a60c3e8a60 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f03137fe640 1 -- 172.21.15.28:0/3010440896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f03240d5b80 con 0x55a60c3e8a60 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03240c55f0 con 0x55a60c3e8a60 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f03137fe640 1 -- 172.21.15.28:0/3010440896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f03240bc2e0 con 0x55a60c3e8a60 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 10 client.17373 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a60c471fe0 con 0x55a60c3e8a60 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f03137fe640 1 -- 172.21.15.28:0/3010440896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03240c55f0 con 0x55a60c3e8a60 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f03137fe640 1 -- 172.21.15.28:0/3010440896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f03240b8870 con 0x55a60c3e8a60 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 10 client.17373 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:31.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 10 client.17373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 20 client.17373 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c473610 0x55a60c475a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 4 client.17373 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T22:06:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.770+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a60c355d00 con 0x55a60c473610 2024-09-15T22:06:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.771+0000 7f032bfff640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c473610 0x55a60c475a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.771+0000 7f032bfff640 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c473610 0x55a60c475a00 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6b2415f26fddec3 server_cookie=87a9e8cba93fc406 in_seq=0 out_seq=0 2024-09-15T22:06:31.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.771+0000 7f03137fe640 10 client.17373 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:31.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f03137fe640 1 -- 172.21.15.28:0/3010440896 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a60c355d00 con 0x55a60c473610 2024-09-15T22:06:31.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f03137fe640 10 client.17373 handle_command_reply: tid=0 2024-09-15T22:06:31.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 client.17373 shutdown 2024-09-15T22:06:31.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 msgr2=0x55a60c3fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3fc510 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f03240d2010 tx=0x7f03240a7270 comp rx=0 tx=0).stop 2024-09-15T22:06:31.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 shutdown_connections 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55a60c473610 0x55a60c475a00 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a60c3f8150 0x55a60c471aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a60c3e9380 0x55a60c3fca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 --2- 172.21.15.28:0/3010440896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a60c3e8a60 0x55a60c3fc510 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 >> 172.21.15.28:0/3010440896 conn(0x55a60c3e50f0 msgr2=0x55a60c3f6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f03137fe640 0 client.17373 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.772+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 shutdown_connections 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033caaf740 1 -- 172.21.15.28:0/3010440896 wait complete. 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033caaf740 20 client.17373 trim_cache size 0 max 16384 2024-09-15T22:06:31.780 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:31.781 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T22:06:31.781 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d7104244-dc4e-49c3-8b95-4070ca1ea7a7", 2024-09-15T22:06:31.781 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T22:06:31.781 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 msgr2=0x7f03200a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f032c019000 tx=0x7f032c01b000 comp rx=0 tx=0).stop 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 msgr2=0x7f0334198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334198440 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f0324006010 tx=0x7f0324003c80 comp rx=0 tx=0).stop 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 shutdown_connections 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f03200a3430 0x7f03200a58f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0334102ef0 0x7f033419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0334102570 0x7f0334198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 --2- 172.21.15.28:0/1696944641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0334100e20 0x7f0334198440 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 >> 172.21.15.28:0/1696944641 conn(0x7f03340fc7d0 msgr2=0x7f033406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 shutdown_connections 2024-09-15T22:06:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:31.773+0000 7f033a6d6640 1 -- 172.21.15.28:0/1696944641 wait complete. 2024-09-15T22:06:31.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:31 smithi028 ceph-mon[30969]: pgmap v282: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-09-15T22:06:32.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:31 smithi203 ceph-mon[35546]: pgmap v282: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-09-15T22:06:32.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:31 smithi174 ceph-mon[35244]: pgmap v282: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-09-15T22:06:32.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:06:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:32 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T22:06:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:32 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:32 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:06:33.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:32 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:32 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T22:06:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:32 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:32 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:06:33.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:32 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:32 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T22:06:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:32 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:32 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:06:33.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:32 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:33.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 -- 172.21.15.28:0/2719313683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 msgr2=0x7fcfc410eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/2719313683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc410eaf0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009cd0 tx=0x7fcfbc02a8e0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 -- 172.21.15.28:0/2719313683 shutdown_connections 2024-09-15T22:06:33.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/2719313683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc410eaf0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/2719313683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc4100d00 0x7fcfc4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/2719313683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc4100350 0x7fcfc4100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 -- 172.21.15.28:0/2719313683 >> 172.21.15.28:0/2719313683 conn(0x7fcfc4077210 msgr2=0x7fcfc40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 -- 172.21.15.28:0/2719313683 shutdown_connections 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.321+0000 7fcfc99ed640 1 -- 172.21.15.28:0/2719313683 wait complete. 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 Processor -- start 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 -- start start 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc4100350 0x7fcfc41983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc4100d00 0x7fcfc41988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc4111fa0 con 0x7fcfc4106f50 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc4111e20 con 0x7fcfc4100350 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.322+0000 7fcfc99ed640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc4112120 con 0x7fcfc4100d00 2024-09-15T22:06:33.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45102/0 (socket says 172.21.15.28:45102) 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 -- 172.21.15.28:0/3275585987 learned_addr learned my addr 172.21.15.28:0/3275585987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc4100350 0x7fcfc41983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc4100d00 0x7fcfc41988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 -- 172.21.15.28:0/3275585987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc4100d00 msgr2=0x7fcfc41988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc4100d00 0x7fcfc41988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 -- 172.21.15.28:0/3275585987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc4100350 msgr2=0x7fcfc41983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc4100350 0x7fcfc41983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 -- 172.21.15.28:0/3275585987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc419d360 con 0x7fcfc4106f50 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc37fe640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc419cc80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc005cc0 tx=0x7fcfbc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc89eb640 1 -- 172.21.15.28:0/3275585987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc004040 con 0x7fcfc4106f50 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc89eb640 1 -- 172.21.15.28:0/3275585987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfbc0046c0 con 0x7fcfc4106f50 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc419d5f0 con 0x7fcfc4106f50 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.323+0000 7fcfc89eb640 1 -- 172.21.15.28:0/3275585987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc046480 con 0x7fcfc4106f50 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.324+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc41a4f30 con 0x7fcfc4106f50 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.326+0000 7fcfcbdc6740 1 Processor -- start 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.326+0000 7fcfc89eb640 1 -- 172.21.15.28:0/3275585987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fcfbc0305a0 con 0x7fcfc4106f50 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfcbdc6740 1 -- start start 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972069880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972074140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 0x55d972076a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d971fcf2e0 con 0x55d972069dc0 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d971f76ce0 con 0x55d972074680 2024-09-15T22:06:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d971fcb2a0 con 0x55d9720694a0 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972074140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45114/0 (socket says 172.21.15.28:45114) 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/292911325 learned_addr learned my addr 172.21.15.28:0/292911325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 0x55d972076a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/292911325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 msgr2=0x55d972069880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972069880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972069880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/292911325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 msgr2=0x55d972076a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 0x55d972076a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/292911325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d971f5b440 con 0x55d972069dc0 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972069880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.327+0000 7fcfc89eb640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfb00a33b0 0x7fcfb00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.328+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfb00a33b0 0x7fcfb00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.328+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972074140 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cc60 tx=0x7fcfb400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939ba8b4525584a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.328+0000 7fcfc89eb640 1 -- 172.21.15.28:0/3275585987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fcfbc02e030 con 0x7fcfc4106f50 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.328+0000 7fcfb9ffb640 1 -- 172.21.15.28:0/292911325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4014070 con 0x55d972069dc0 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.328+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfb00a33b0 0x7fcfb00a5870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55d972066db0 tx=0x7fcfac0046c0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.328+0000 7fcfb9ffb640 1 -- 172.21.15.28:0/292911325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb40047a0 con 0x55d972069dc0 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.328+0000 7fcfb9ffb640 1 -- 172.21.15.28:0/292911325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4010790 con 0x55d972069dc0 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/292911325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 msgr2=0x55d972074140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972074140 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400cc60 tx=0x7fcfb400e720 comp rx=0 tx=0).stop 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/292911325 shutdown_connections 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 0x55d972076a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972074140 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/292911325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972069880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/292911325 >> 172.21.15.28:0/292911325 conn(0x55d972065a30 msgr2=0x55d972065e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/292911325 shutdown_connections 2024-09-15T22:06:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/292911325 wait complete. 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.329+0000 7fcfcbdc6740 1 -- start start 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 1 Processor -- start 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972073fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972071250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 0x55d972071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d972077250 con 0x55d972069dc0 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d971f8a050 con 0x55d972074680 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d9720773d0 con 0x55d9720694a0 2024-09-15T22:06:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfcbdc6740 10 client.0 _ll_register_callbacks cb 0x55d971fca160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972071250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45122/0 (socket says 172.21.15.28:45122) 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 -- 172.21.15.28:0/2793341916 learned_addr learned my addr 172.21.15.28:0/2793341916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972073fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 -- 172.21.15.28:0/2793341916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 msgr2=0x55d972073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972073fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 -- 172.21.15.28:0/2793341916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 msgr2=0x55d972071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 0x55d972071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 -- 172.21.15.28:0/2793341916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d971d403c0 con 0x55d972069dc0 2024-09-15T22:06:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972071250 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcfac011c20 tx=0x7fcfac00c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5324414415e20589 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.330+0000 7fcfa2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2793341916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfac011e50 con 0x55d972069dc0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2793341916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fcfac027400 con 0x55d972069dc0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb40047a0 con 0x55d972069dc0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2793341916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcfac030050 con 0x55d972069dc0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 10 client.17385 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d972082ba0 con 0x55d972069dc0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2793341916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb40047a0 con 0x55d972069dc0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2793341916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fcfac01e410 con 0x55d972069dc0 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 10 client.17385 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 10 client.17385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 20 client.17385 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720840b0 0x55d972086570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 4 client.17385 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.331+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d971fdc850 con 0x55d9720840b0 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.332+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720840b0 0x55d972086570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.332+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720840b0 0x55d972086570 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f5e325d85027411 server_cookie=59ec862d8709747d in_seq=0 out_seq=0 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.332+0000 7fcfa2ffd640 10 client.17385 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2793341916 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d971fdc850 con 0x55d9720840b0 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfa2ffd640 10 client.17385 handle_command_reply: tid=0 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 client.17385 shutdown 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 msgr2=0x55d972071250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972071250 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcfac011c20 tx=0x7fcfac00c3b0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 shutdown_connections 2024-09-15T22:06:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720840b0 0x55d972086570 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972074680 0x55d972071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972069dc0 0x55d972071250 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2793341916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d972073fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 >> 172.21.15.28:0/2793341916 conn(0x55d972065a30 msgr2=0x55d97206b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfa2ffd640 0 client.17385 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 shutdown_connections 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2793341916 wait complete. 2024-09-15T22:06:33.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.333+0000 7fcfcbdc6740 20 client.17385 trim_cache size 0 max 16384 2024-09-15T22:06:33.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 Processor -- start 2024-09-15T22:06:33.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 -- start start 2024-09-15T22:06:33.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9720694a0 0x55d9720804b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d9720860c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d972074680 0x55d97206c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41af000 con 0x55d9720694a0 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41b0b70 con 0x55d972069dc0 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41af6d0 con 0x55d972074680 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9720694a0 0x55d9720804b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9720694a0 0x55d9720804b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45132/0 (socket says 172.21.15.28:45132) 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/896896876 learned_addr learned my addr 172.21.15.28:0/896896876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d9720860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/896896876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d972074680 msgr2=0x55d97206c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d972074680 0x55d97206c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/896896876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 msgr2=0x55d9720860c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d9720860c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 -- 172.21.15.28:0/896896876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d972070200 con 0x55d9720694a0 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d9720860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.351+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9720694a0 0x55d9720804b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c7e0 tx=0x7fcfb4010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fb894d45f8c391 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfb97fa640 1 -- 172.21.15.28:0/896896876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4014040 con 0x55d9720694a0 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfb97fa640 1 -- 172.21.15.28:0/896896876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb40048c0 con 0x55d9720694a0 2024-09-15T22:06:33.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfb97fa640 1 -- 172.21.15.28:0/896896876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40249b0 con 0x55d9720694a0 2024-09-15T22:06:33.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/896896876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9720694a0 msgr2=0x55d9720804b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9720694a0 0x55d9720804b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c7e0 tx=0x7fcfb4010430 comp rx=0 tx=0).stop 2024-09-15T22:06:33.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/896896876 shutdown_connections 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d972074680 0x55d97206c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d9720860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/896896876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d9720694a0 0x55d9720804b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/896896876 >> 172.21.15.28:0/896896876 conn(0x55d972065a30 msgr2=0x55d97206c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/896896876 shutdown_connections 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.352+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/896896876 wait complete. 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 -- start start 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 Processor -- start 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d97207cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d972077f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972074680 0x55d9720784b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41b04a0 con 0x55d972074680 2024-09-15T22:06:33.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41afdd0 con 0x55d972069dc0 2024-09-15T22:06:33.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc41ad000 con 0x55d9720694a0 2024-09-15T22:06:33.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfcbdc6740 10 client.0 _ll_register_callbacks cb 0x7fcfac00c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:33.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972074680 0x55d9720784b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d972077f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfba7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d972077f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.28:34170/0 (socket says 172.21.15.28:34170) 2024-09-15T22:06:33.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfba7fc640 1 -- 172.21.15.28:0/2550506533 learned_addr learned my addr 172.21.15.28:0/2550506533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:33.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972074680 0x55d9720784b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45148/0 (socket says 172.21.15.28:45148) 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.353+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d97207cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfc37fe640 1 -- 172.21.15.28:0/2550506533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 msgr2=0x55d97207cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfc37fe640 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d97207cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfc37fe640 1 -- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 msgr2=0x55d972077f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfc37fe640 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d972077f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfc37fe640 1 -- 172.21.15.28:0/2550506533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d972078af0 con 0x55d972074680 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfba7fc640 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d972077f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:33.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfc37fe640 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972074680 0x55d9720784b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0eed80 tx=0x7fcfbc0bd7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4beeeb4c35793c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfa2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2550506533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc031e60 con 0x55d972074680 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2550506533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fcfbc0bcd60 con 0x55d972074680 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb40048c0 con 0x55d972074680 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2550506533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcfbc0be010 con 0x55d972074680 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfcbdc6740 10 client.17391 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d9720eec20 con 0x55d972074680 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.354+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2550506533 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb40048c0 con 0x55d972074680 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2550506533 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fcfbc04e070 con 0x55d972074680 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfcbdc6740 10 client.17391 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfcbdc6740 10 client.17391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfcbdc6740 20 client.17391 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720ef340 0x55d9720f1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfcbdc6740 4 client.17391 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d971fd6d60 con 0x55d9720ef340 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720ef340 0x55d9720f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfa2ffd640 10 client.17391 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.355+0000 7fcfc2ffd640 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720ef340 0x55d9720f1760 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f804db12ad85dd09 server_cookie=e38c9676b7a5dd4c in_seq=0 out_seq=0 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfa2ffd640 1 -- 172.21.15.28:0/2550506533 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d971fd6d60 con 0x55d9720ef340 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfa2ffd640 10 client.17391 handle_command_reply: tid=0 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 client.17391 shutdown 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972074680 msgr2=0x55d9720784b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972074680 0x55d9720784b0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0eed80 tx=0x7fcfbc0bd7a0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 shutdown_connections 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55d9720ef340 0x55d9720f1760 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d972074680 0x55d9720784b0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d972069dc0 0x55d972077f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 --2- 172.21.15.28:0/2550506533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d9720694a0 0x55d97207cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfa2ffd640 0 client.17391 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 >> 172.21.15.28:0/2550506533 conn(0x55d972065a30 msgr2=0x55d97206b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 shutdown_connections 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 1 -- 172.21.15.28:0/2550506533 wait complete. 2024-09-15T22:06:33.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.356+0000 7fcfcbdc6740 20 client.17391 trim_cache size 0 max 16384 2024-09-15T22:06:33.366 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:06:33.366 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:06:33.366 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:33.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfb00a33b0 msgr2=0x7fcfb00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfb00a33b0 0x7fcfb00a5870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55d972066db0 tx=0x7fcfac0046c0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 msgr2=0x7fcfc419cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc419cc80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc005cc0 tx=0x7fcfbc003970 comp rx=0 tx=0).stop 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 shutdown_connections 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fcfb00a33b0 0x7fcfb00a5870 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfc4106f50 0x7fcfc419cc80 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcfc4100d00 0x7fcfc41988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 --2- 172.21.15.28:0/3275585987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc4100350 0x7fcfc41983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 >> 172.21.15.28:0/3275585987 conn(0x7fcfc4077210 msgr2=0x7fcfc41057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:33.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 shutdown_connections 2024-09-15T22:06:33.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:33.357+0000 7fcfc99ed640 1 -- 172.21.15.28:0/3275585987 wait complete. 2024-09-15T22:06:34.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:33 smithi028 ceph-mon[30969]: pgmap v283: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.2 KiB/s rd, 13 op/s 2024-09-15T22:06:34.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:33 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:06:34.090 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:06:34.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:06:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:33 smithi203 ceph-mon[35546]: pgmap v283: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.2 KiB/s rd, 13 op/s 2024-09-15T22:06:34.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:33 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:06:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:33 smithi174 ceph-mon[35244]: pgmap v283: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 8.2 KiB/s rd, 13 op/s 2024-09-15T22:06:34.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:33 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:06:34.347 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 -- 172.21.15.28:0/917792256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 msgr2=0x7f88d8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/917792256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d8106e60 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009930 tx=0x7f88c802a420 comp rx=0 tx=0).stop 2024-09-15T22:06:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 -- 172.21.15.28:0/917792256 shutdown_connections 2024-09-15T22:06:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/917792256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88d8101420 0x7f88d810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/917792256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88d8100a80 0x7f88d8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/917792256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d8106e60 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 -- 172.21.15.28:0/917792256 >> 172.21.15.28:0/917792256 conn(0x7f88d80fc7a0 msgr2=0x7f88d80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 -- 172.21.15.28:0/917792256 shutdown_connections 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.829+0000 7f88dd3a4640 1 -- 172.21.15.28:0/917792256 wait complete. 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 Processor -- start 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 -- start start 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88d8100a80 0x7f88d81a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88d8101420 0x7f88d81a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8110f20 con 0x7f88d8106a80 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8110da0 con 0x7f88d8101420 2024-09-15T22:06:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88dd3a4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81110a0 con 0x7f88d8100a80 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45174/0 (socket says 172.21.15.28:45174) 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88d77fe640 1 -- 172.21.15.28:0/3274826263 learned_addr learned my addr 172.21.15.28:0/3274826263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88d6ffd640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88d8100a80 0x7f88d81a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.830+0000 7f88d67fc640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88d8101420 0x7f88d81a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d77fe640 1 -- 172.21.15.28:0/3274826263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88d8100a80 msgr2=0x7f88d81a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d77fe640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88d8100a80 0x7f88d81a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d77fe640 1 -- 172.21.15.28:0/3274826263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88d8101420 msgr2=0x7f88d81a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:34.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d77fe640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88d8101420 0x7f88d81a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d77fe640 1 -- 172.21.15.28:0/3274826263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d819ad20 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d67fc640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88d8101420 0x7f88d81a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d6ffd640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88d8100a80 0x7f88d81a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88d77fe640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d819a5f0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f88d000b970 tx=0x7f88d000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88bffff640 1 -- 172.21.15.28:0/3274826263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0019070 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d819b010 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88bffff640 1 -- 172.21.15.28:0/3274826263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88d000cb40 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88bffff640 1 -- 172.21.15.28:0/3274826263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0007650 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.831+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d81a7300 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.832+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d8100f60 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.833+0000 7f88bffff640 1 -- 172.21.15.28:0/3274826263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f88d001c070 con 0x7f88d8106a80 2024-09-15T22:06:34.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.834+0000 7f88bffff640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88c00a33b0 0x7f88c00a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.834+0000 7f88bffff640 1 -- 172.21.15.28:0/3274826263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f88d00cf660 con 0x7f88d8106a80 2024-09-15T22:06:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.834+0000 7f88d6ffd640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88c00a33b0 0x7f88c00a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.835+0000 7f88d6ffd640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88c00a33b0 0x7f88c00a5870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f88c8000c00 tx=0x7f88c8005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:34.839+0000 7f88bffff640 1 -- 172.21.15.28:0/3274826263 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f88d009bda0 con 0x7f88d8106a80 2024-09-15T22:06:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.052+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88d81a7530 con 0x7f88d8106a80 2024-09-15T22:06:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.053+0000 7f88bffff640 1 -- 172.21.15.28:0/3274826263 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11988 (secure 0 0 0) 0x7f88d009b6a0 con 0x7f88d8106a80 2024-09-15T22:06:35.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:06:35.055 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"btime":"2024-09-15T22:04:37:022811+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:04:36.713632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:06:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-09-15T22:06:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88c00a33b0 msgr2=0x7f88c00a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88c00a33b0 0x7f88c00a5870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f88c8000c00 tx=0x7f88c8005a20 comp rx=0 tx=0).stop 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 msgr2=0x7f88d819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d819a5f0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f88d000b970 tx=0x7f88d000be30 comp rx=0 tx=0).stop 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 shutdown_connections 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f88c00a33b0 0x7f88c00a5870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8106a80 0x7f88d819a5f0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88d8101420 0x7f88d81a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 --2- 172.21.15.28:0/3274826263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88d8100a80 0x7f88d81a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 >> 172.21.15.28:0/3274826263 conn(0x7f88d80fc7a0 msgr2=0x7f88d806a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 shutdown_connections 2024-09-15T22:06:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:35.055+0000 7f88dd3a4640 1 -- 172.21.15.28:0/3274826263 wait complete. 2024-09-15T22:06:35.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:06:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:36.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:35 smithi028 ceph-mon[30969]: pgmap v284: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-09-15T22:06:36.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:35 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3274826263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:36.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:35 smithi203 ceph-mon[35546]: pgmap v284: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-09-15T22:06:36.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:35 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3274826263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:36.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:35 smithi174 ceph-mon[35244]: pgmap v284: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-09-15T22:06:36.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:35 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3274826263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:06:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 -- 172.21.15.28:0/1261316679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103660 msgr2=0x7fefc010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 --2- 172.21.15.28:0/1261316679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103660 0x7fefc010bb50 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009900 tx=0x7fefa802a480 comp rx=0 tx=0).stop 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 -- 172.21.15.28:0/1261316679 shutdown_connections 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 --2- 172.21.15.28:0/1261316679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc010c170 0x7fefc010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 --2- 172.21.15.28:0/1261316679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103660 0x7fefc010bb50 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 --2- 172.21.15.28:0/1261316679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc0102d40 0x7fefc0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 -- 172.21.15.28:0/1261316679 >> 172.21.15.28:0/1261316679 conn(0x7fefc00fc7a0 msgr2=0x7fefc00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 -- 172.21.15.28:0/1261316679 shutdown_connections 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.441+0000 7fefc486e640 1 -- 172.21.15.28:0/1261316679 wait complete. 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 Processor -- start 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 -- start start 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc0102d40 0x7fefc01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc0103660 0x7fefc0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc010c170 0x7fefc0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111f70 con 0x7fefc010c170 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111df0 con 0x7fefc0103660 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefc486e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01120f0 con 0x7fefc0102d40 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefbed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc010c170 0x7fefc0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefbed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc010c170 0x7fefc0101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45190/0 (socket says 172.21.15.28:45190) 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefbed9d640 1 -- 172.21.15.28:0/2297187054 learned_addr learned my addr 172.21.15.28:0/2297187054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc0103660 0x7fefc0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.442+0000 7fefbe59c640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc0102d40 0x7fefc01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefbed9d640 1 -- 172.21.15.28:0/2297187054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc0102d40 msgr2=0x7fefc01009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefbed9d640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc0102d40 0x7fefc01009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefbed9d640 1 -- 172.21.15.28:0/2297187054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc0103660 msgr2=0x7fefc0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefbed9d640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc0103660 0x7fefc0100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefbed9d640 1 -- 172.21.15.28:0/2297187054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc01a6930 con 0x7fefc010c170 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefbed9d640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc010c170 0x7fefc0101420 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fefb800cad0 tx=0x7fefb800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefb7fff640 1 -- 172.21.15.28:0/2297187054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb800ae20 con 0x7fefc010c170 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefb7fff640 1 -- 172.21.15.28:0/2297187054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb8005020 con 0x7fefc010c170 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc01a6c20 con 0x7fefc010c170 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefb7fff640 1 -- 172.21.15.28:0/2297187054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb800f6f0 con 0x7fefc010c170 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.443+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc01a70e0 con 0x7fefc010c170 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.445+0000 7fefb7fff640 1 -- 172.21.15.28:0/2297187054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fefb80051d0 con 0x7fefc010c170 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 Processor -- start 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 -- start start 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a2f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a2f5d0 0x562169a2fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a3a370 0x562169a3c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621699952f0 con 0x562169a3a370 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56216993ccf0 con 0x562169a30b40 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefc6c47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5621699912b0 con 0x562169a2f5d0 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefb7fff640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fefac0a33b0 0x7fefac0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefb5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a3a370 0x562169a3c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefb5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a3a370 0x562169a3c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45194/0 (socket says 172.21.15.28:45194) 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2277500118 learned_addr learned my addr 172.21.15.28:0/2277500118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefbe59c640 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a2f5d0 0x562169a2fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a2f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.446+0000 7fefbed9d640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fefac0a33b0 0x7fefac0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb7fff640 1 -- 172.21.15.28:0/2297187054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fefb80d0070 con 0x7fefc010c170 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2277500118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a2f5d0 msgr2=0x562169a2fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a2f5d0 0x562169a2fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2277500118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 msgr2=0x562169a2f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a2f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2277500118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621699213a0 con 0x562169a3a370 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a3a370 0x562169a3c760 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009900 tx=0x7fefa8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9be1a3a9862e3d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefbed9d640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fefac0a33b0 0x7fefac0a5870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fefb809bf80 tx=0x7fefb8091480 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb4d99640 1 -- 172.21.15.28:0/2277500118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8035070 con 0x562169a3a370 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb4d99640 1 -- 172.21.15.28:0/2277500118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa8005020 con 0x562169a3a370 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.447+0000 7fefb4d99640 1 -- 172.21.15.28:0/2277500118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8004a00 con 0x562169a3a370 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 -- 172.21.15.28:0/2277500118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a3a370 msgr2=0x562169a3c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a3a370 0x562169a3c760 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009900 tx=0x7fefa8000bc0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 -- 172.21.15.28:0/2277500118 shutdown_connections 2024-09-15T22:06:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a3a370 0x562169a3c760 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a2f5d0 0x562169a2fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2277500118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a2f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 -- 172.21.15.28:0/2277500118 >> 172.21.15.28:0/2277500118 conn(0x5621699f8310 msgr2=0x562169a2bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 -- 172.21.15.28:0/2277500118 shutdown_connections 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 -- 172.21.15.28:0/2277500118 wait complete. 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 -- start start 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.448+0000 7fefc6c47740 1 Processor -- start 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a2f5d0 0x562169a39d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a30b40 0x562169a36fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefc6c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562169a3d010 con 0x562169a30b40 2024-09-15T22:06:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefc6c47740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562169950060 con 0x562169a2f5d0 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefc6c47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562169a3d190 con 0x562169a3a370 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefc6c47740 10 client.0 _ll_register_callbacks cb 0x5621696a38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a30b40 0x562169a36fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a30b40 0x562169a36fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45206/0 (socket says 172.21.15.28:45206) 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 -- 172.21.15.28:0/3948412876 learned_addr learned my addr 172.21.15.28:0/3948412876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a2f5d0 0x562169a39d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 -- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 msgr2=0x562169a375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 -- 172.21.15.28:0/3948412876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a2f5d0 msgr2=0x562169a39d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a2f5d0 0x562169a39d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 -- 172.21.15.28:0/3948412876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621697063c0 con 0x562169a30b40 2024-09-15T22:06:36.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefbe59c640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a30b40 0x562169a36fe0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fefb000bfc0 tx=0x7fefb00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55273ef9b2cdf81 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.449+0000 7fefa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefa5ffb640 1 -- 172.21.15.28:0/3948412876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0004020 con 0x562169a30b40 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefa5ffb640 1 -- 172.21.15.28:0/3948412876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fefb00041c0 con 0x562169a30b40 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefa8005020 con 0x562169a30b40 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefa5ffb640 1 -- 172.21.15.28:0/3948412876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fefb0025050 con 0x562169a30b40 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 10 client.17409 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562169a48aa0 con 0x562169a30b40 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefa5ffb640 1 -- 172.21.15.28:0/3948412876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa8005020 con 0x562169a30b40 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefa5ffb640 1 -- 172.21.15.28:0/3948412876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fefb0003950 con 0x562169a30b40 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 10 client.17409 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 10 client.17409 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 20 client.17409 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169a4a000 0x562169a4c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 4 client.17409 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.450+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621699a2860 con 0x562169a4a000 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.451+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169a4a000 0x562169a4c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.452+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169a4a000 0x562169a4c420 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=279b7bba03429db1 server_cookie=ed283afda3d48b2d in_seq=0 out_seq=0 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.452+0000 7fefa5ffb640 10 client.17409 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefa5ffb640 1 -- 172.21.15.28:0/3948412876 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5621699a2860 con 0x562169a4a000 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefa5ffb640 10 client.17409 handle_command_reply: tid=0 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 client.17409 shutdown 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a30b40 msgr2=0x562169a36fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a30b40 0x562169a36fe0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fefb000bfc0 tx=0x7fefb00118d0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 shutdown_connections 2024-09-15T22:06:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169a4a000 0x562169a4c420 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a30b40 0x562169a36fe0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 --2- 172.21.15.28:0/3948412876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a2f5d0 0x562169a39d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefa5ffb640 0 client.17409 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 >> 172.21.15.28:0/3948412876 conn(0x5621699f8310 msgr2=0x562169a31960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 shutdown_connections 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 1 -- 172.21.15.28:0/3948412876 wait complete. 2024-09-15T22:06:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.453+0000 7fefc6c47740 20 client.17409 trim_cache size 0 max 16384 2024-09-15T22:06:36.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 Processor -- start 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 -- start start 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a2be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a368a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01b1250 con 0x562169a2f5d0 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01b2dc0 con 0x562169a30b40 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.466+0000 7fefc6c47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01b1920 con 0x562169a3a370 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a2be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a2be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45210/0 (socket says 172.21.15.28:45210) 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2136570023 learned_addr learned my addr 172.21.15.28:0/2136570023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefbe59c640 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2136570023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 msgr2=0x562169a368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2136570023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 msgr2=0x562169a462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 -- 172.21.15.28:0/2136570023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562169a2c960 con 0x562169a2f5d0 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a2be10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009900 tx=0x7fefa80314c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe40a1e011c1b2f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefa7fff640 1 -- 172.21.15.28:0/2136570023 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8035040 con 0x562169a2f5d0 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefa7fff640 1 -- 172.21.15.28:0/2136570023 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa8004c30 con 0x562169a2f5d0 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.467+0000 7fefa7fff640 1 -- 172.21.15.28:0/2136570023 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8005430 con 0x562169a2f5d0 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 -- 172.21.15.28:0/2136570023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 msgr2=0x562169a2be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a2be10 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fefa8009900 tx=0x7fefa80314c0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 -- 172.21.15.28:0/2136570023 shutdown_connections 2024-09-15T22:06:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 --2- 172.21.15.28:0/2136570023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a2be10 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 -- 172.21.15.28:0/2136570023 >> 172.21.15.28:0/2136570023 conn(0x5621699f8310 msgr2=0x562169a2b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 -- 172.21.15.28:0/2136570023 shutdown_connections 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 -- 172.21.15.28:0/2136570023 wait complete. 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 -- start start 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.468+0000 7fefc6c47740 1 Processor -- start 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a42630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a42b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefc6c47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a33810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefc6c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01b26f0 con 0x562169a2f5d0 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefc6c47740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01b2020 con 0x562169a30b40 2024-09-15T22:06:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefc6c47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01af250 con 0x562169a3a370 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefc6c47740 10 client.0 _ll_register_callbacks cb 0x7fefb000bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a42630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a42630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45218/0 (socket says 172.21.15.28:45218) 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 -- 172.21.15.28:0/712808449 learned_addr learned my addr 172.21.15.28:0/712808449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefbe59c640 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a42b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a33810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 -- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 msgr2=0x562169a33810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a33810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 -- 172.21.15.28:0/712808449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 msgr2=0x562169a42b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a42b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 -- 172.21.15.28:0/712808449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562169a33e80 con 0x562169a2f5d0 2024-09-15T22:06:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefb5d9b640 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a42630 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fefa8031470 tx=0x7fefa8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c6862bf00ca62c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.469+0000 7fefa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefa5ffb640 1 -- 172.21.15.28:0/712808449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa805a020 con 0x562169a2f5d0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefa8004c30 con 0x562169a2f5d0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefa5ffb640 1 -- 172.21.15.28:0/712808449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fefa80051b0 con 0x562169a2f5d0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefa5ffb640 1 -- 172.21.15.28:0/712808449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fefa8065050 con 0x562169a2f5d0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 10 client.17415 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562169ab4620 con 0x562169a2f5d0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefa5ffb640 1 -- 172.21.15.28:0/712808449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa8004c30 con 0x562169a2f5d0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefa5ffb640 1 -- 172.21.15.28:0/712808449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fefa803db80 con 0x562169a2f5d0 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 10 client.17415 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 10 client.17415 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 20 client.17415 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169ab47f0 0x562169ab6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 4 client.17415 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.470+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56216999cd70 con 0x562169ab47f0 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefbe59c640 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169ab47f0 0x562169ab6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefbe59c640 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169ab47f0 0x562169ab6be0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d5b1c49fbd05abe server_cookie=111785927a26e708 in_seq=0 out_seq=0 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefa5ffb640 10 client.17415 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefa5ffb640 1 -- 172.21.15.28:0/712808449 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56216999cd70 con 0x562169ab47f0 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefa5ffb640 10 client.17415 handle_command_reply: tid=0 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefc6c47740 1 client.17415 shutdown 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 msgr2=0x562169a42630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefc6c47740 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a42630 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fefa8031470 tx=0x7fefa8031400 comp rx=0 tx=0).stop 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 shutdown_connections 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.471+0000 7fefc6c47740 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x562169ab47f0 0x562169ab6be0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc6c47740 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562169a3a370 0x562169a33810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc6c47740 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562169a30b40 0x562169a42b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc6c47740 1 --2- 172.21.15.28:0/712808449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562169a2f5d0 0x562169a42630 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 >> 172.21.15.28:0/712808449 conn(0x5621699f8310 msgr2=0x562169a368a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefa5ffb640 0 client.17415 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 shutdown_connections 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc6c47740 1 -- 172.21.15.28:0/712808449 wait complete. 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc6c47740 20 client.17415 trim_cache size 0 max 16384 2024-09-15T22:06:36.480 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fefac0a33b0 msgr2=0x7fefac0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fefac0a33b0 0x7fefac0a5870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fefb809bf80 tx=0x7fefb8091480 comp rx=0 tx=0).stop 2024-09-15T22:06:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc010c170 msgr2=0x7fefc0101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc010c170 0x7fefc0101420 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fefb800cad0 tx=0x7fefb800cf90 comp rx=0 tx=0).stop 2024-09-15T22:06:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 shutdown_connections 2024-09-15T22:06:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fefac0a33b0 0x7fefac0a5870 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc010c170 0x7fefc0101420 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc0103660 0x7fefc0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 --2- 172.21.15.28:0/2297187054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc0102d40 0x7fefc01009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 >> 172.21.15.28:0/2297187054 conn(0x7fefc00fc7a0 msgr2=0x7fefc00fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 shutdown_connections 2024-09-15T22:06:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:36.472+0000 7fefc486e640 1 -- 172.21.15.28:0/2297187054 wait complete. 2024-09-15T22:06:37.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:06:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:37.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 -- 172.21.15.28:0/1852008361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4101440 msgr2=0x7f8df410df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/1852008361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4101440 0x7f8df410df30 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009900 tx=0x7f8de402a410 comp rx=0 tx=0).stop 2024-09-15T22:06:37.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 -- 172.21.15.28:0/1852008361 shutdown_connections 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/1852008361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df410e500 0x7f8df4110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/1852008361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4101440 0x7f8df410df30 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/1852008361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8df4100b20 0x7f8df4100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 -- 172.21.15.28:0/1852008361 >> 172.21.15.28:0/1852008361 conn(0x7f8df40fc7d0 msgr2=0x7f8df40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 -- 172.21.15.28:0/1852008361 shutdown_connections 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.918+0000 7f8dfbffe640 1 -- 172.21.15.28:0/1852008361 wait complete. 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 Processor -- start 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 -- start start 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4100b20 0x7f8df41a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8df4101440 0x7f8df41a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df410e500 0x7f8df419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41142c0 con 0x7f8df4100b20 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8df4114140 con 0x7f8df4101440 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfbffe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8df4114440 con 0x7f8df410e500 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8df9d9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4100b20 0x7f8df41a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8df9d9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4100b20 0x7f8df41a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45232/0 (socket says 172.21.15.28:45232) 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8df9d9a640 1 -- 172.21.15.28:0/4267523707 learned_addr learned my addr 172.21.15.28:0/4267523707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.919+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df410e500 0x7f8df419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8df9599640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8df4101440 0x7f8df41a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8df9d9a640 1 -- 172.21.15.28:0/4267523707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df410e500 msgr2=0x7f8df419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df410e500 0x7f8df419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8df9d9a640 1 -- 172.21.15.28:0/4267523707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8df4101440 msgr2=0x7f8df41a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8df4101440 0x7f8df41a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8df9d9a640 1 -- 172.21.15.28:0/4267523707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df419ace0 con 0x7f8df4100b20 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df410e500 0x7f8df419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4100b20 0x7f8df41a03f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8df000d950 tx=0x7f8df000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8deaffd640 1 -- 172.21.15.28:0/4267523707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0017030 con 0x7f8df4100b20 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8deaffd640 1 -- 172.21.15.28:0/4267523707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df0004020 con 0x7f8df4100b20 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df419afd0 con 0x7f8df4100b20 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8deaffd640 1 -- 172.21.15.28:0/4267523707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df00047a0 con 0x7f8df4100b20 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.920+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df41a72c0 con 0x7f8df4100b20 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.922+0000 7f8deaffd640 1 -- 172.21.15.28:0/4267523707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f8df00041d0 con 0x7f8df4100b20 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 Processor -- start 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 -- start start 2024-09-15T22:06:37.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de18664b40 0x55de18663090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de186635d0 0x55de18663a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18670760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55de185c92f0 con 0x55de1866e370 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55de18570cf0 con 0x55de186635d0 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55de185c52b0 con 0x55de18664b40 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8df9599640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18670760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8df9599640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18670760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45234/0 (socket says 172.21.15.28:45234) 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8df9599640 1 -- 172.21.15.28:0/86393847 learned_addr learned my addr 172.21.15.28:0/86393847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8deaffd640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8de00a3430 0x7f8de00a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de18664b40 0x55de18663090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.923+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de186635d0 0x55de18663a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8de00a3430 0x7f8de00a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8df9599640 1 -- 172.21.15.28:0/86393847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de18664b40 msgr2=0x55de18663090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8df9599640 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de18664b40 0x55de18663090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8df9599640 1 -- 172.21.15.28:0/86393847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de186635d0 msgr2=0x55de18663a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8df9599640 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de186635d0 0x55de18663a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8df9599640 1 -- 172.21.15.28:0/86393847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de185553a0 con 0x55de1866e370 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8deaffd640 1 -- 172.21.15.28:0/4267523707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f8df00cfb40 con 0x7f8df4100b20 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8df9599640 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18670760 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8de4005ab0 tx=0x7f8de4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716c1a179a9b4de1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8de00a3430 0x7f8de00a58f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8dec005c40 tx=0x7f8dec009280 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8ddffff640 1 -- 172.21.15.28:0/86393847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de403f070 con 0x55de1866e370 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8ddffff640 1 -- 172.21.15.28:0/86393847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de4004430 con 0x55de1866e370 2024-09-15T22:06:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.924+0000 7f8ddffff640 1 -- 172.21.15.28:0/86393847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de4035030 con 0x55de1866e370 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/86393847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 msgr2=0x55de18670760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18670760 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8de4005ab0 tx=0x7f8de4004040 comp rx=0 tx=0).stop 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/86393847 shutdown_connections 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18670760 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de186635d0 0x55de18663a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/86393847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de18664b40 0x55de18663090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/86393847 >> 172.21.15.28:0/86393847 conn(0x55de1862c310 msgr2=0x55de1865fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/86393847 shutdown_connections 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/86393847 wait complete. 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 -- start start 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.925+0000 7f8dfe3d7740 1 Processor -- start 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1866afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de1866b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55de18671010 con 0x55de1866e370 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55de18584060 con 0x55de18664b40 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55de18671190 con 0x55de186635d0 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfe3d7740 10 client.0 _ll_register_callbacks cb 0x55de182d78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de1866b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de1866b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45248/0 (socket says 172.21.15.28:45248) 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 -- 172.21.15.28:0/186600788 learned_addr learned my addr 172.21.15.28:0/186600788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8df9599640 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1866afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 -- 172.21.15.28:0/186600788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 msgr2=0x55de1866dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 -- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 msgr2=0x55de1866afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1866afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 -- 172.21.15.28:0/186600788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de1833a3c0 con 0x55de1866e370 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de1866b530 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8dec010990 tx=0x7f8dec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0308e40c6d723b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.926+0000 7f8ddd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/186600788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec00fb50 con 0x55de1866e370 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/186600788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f8dec00fcf0 con 0x55de1866e370 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de4004430 con 0x55de1866e370 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/186600788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8dec004b20 con 0x55de1866e370 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 10 client.17427 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55de1867cad0 con 0x55de1866e370 2024-09-15T22:06:37.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/186600788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de4004430 con 0x55de1866e370 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/186600788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f8dec027420 con 0x55de1866e370 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 10 client.17427 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 10 client.17427 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 20 client.17427 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de1867df90 0x55de18680400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 4 client.17427 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.927+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de185d6860 con 0x55de1867df90 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.928+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de1867df90 0x55de18680400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.928+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de1867df90 0x55de18680400 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2e72d0362dbc848d server_cookie=bb316786e444cf3a in_seq=0 out_seq=0 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.928+0000 7f8ddd7fa640 10 client.17427 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/186600788 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55de185d6860 con 0x55de1867df90 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8ddd7fa640 10 client.17427 handle_command_reply: tid=0 2024-09-15T22:06:37.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 client.17427 shutdown 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 msgr2=0x55de1866b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de1866b530 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8dec010990 tx=0x7f8dec004a40 comp rx=0 tx=0).stop 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 shutdown_connections 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de1867df90 0x55de18680400 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de1866b530 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1866afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8ddd7fa640 0 client.17427 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/186600788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 >> 172.21.15.28:0/186600788 conn(0x55de1862c310 msgr2=0x55de18665990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.929+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 shutdown_connections 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.930+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/186600788 wait complete. 2024-09-15T22:06:37.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.930+0000 7f8dfe3d7740 20 client.17427 trim_cache size 0 max 16384 2024-09-15T22:06:37.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 Processor -- start 2024-09-15T22:06:37.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 -- start start 2024-09-15T22:06:37.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1865ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18666180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b1310 con 0x55de1866e370 2024-09-15T22:06:37.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b2e80 con 0x55de18664b40 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b19e0 con 0x55de186635d0 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfa59b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18666180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfa59b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18666180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45262/0 (socket says 172.21.15.28:45262) 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8dfa59b640 1 -- 172.21.15.28:0/2416113313 learned_addr learned my addr 172.21.15.28:0/2416113313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.943+0000 7f8df9599640 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1865ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8dfa59b640 1 -- 172.21.15.28:0/2416113313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 msgr2=0x55de1866a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8dfa59b640 1 -- 172.21.15.28:0/2416113313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 msgr2=0x55de1865ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1865ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8dfa59b640 1 -- 172.21.15.28:0/2416113313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de18669f80 con 0x55de1866e370 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18666180 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8dec012e60 tx=0x7f8dec009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8747f66f6deaa10a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8ddf7fe640 1 -- 172.21.15.28:0/2416113313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec010520 con 0x55de1866e370 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8ddf7fe640 1 -- 172.21.15.28:0/2416113313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec023070 con 0x55de1866e370 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.944+0000 7f8ddf7fe640 1 -- 172.21.15.28:0/2416113313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0346b0 con 0x55de1866e370 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/2416113313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 msgr2=0x55de18666180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18666180 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8dec012e60 tx=0x7f8dec009aa0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/2416113313 shutdown_connections 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de1866e370 0x55de18666180 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de1865ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/2416113313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de186635d0 0x55de1866a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/2416113313 >> 172.21.15.28:0/2416113313 conn(0x55de1862c310 msgr2=0x55de1865e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/2416113313 shutdown_connections 2024-09-15T22:06:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/2416113313 wait complete. 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 -- start start 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.945+0000 7f8dfe3d7740 1 Processor -- start 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de186635d0 0x55de18667f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de18668490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de1866e370 0x55de186689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b27b0 con 0x55de186635d0 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41b20e0 con 0x55de18664b40 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41af310 con 0x55de1866e370 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de186635d0 0x55de18667f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de186635d0 0x55de18667f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45270/0 (socket says 172.21.15.28:45270) 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 10 client.0 _ll_register_callbacks cb 0x7f8dec012f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 -- 172.21.15.28:0/85549116 learned_addr learned my addr 172.21.15.28:0/85549116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfa59b640 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de1866e370 0x55de186689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9599640 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de18668490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 -- 172.21.15.28:0/85549116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de1866e370 msgr2=0x55de186689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de1866e370 0x55de186689d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 -- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 msgr2=0x55de18668490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de18668490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 -- 172.21.15.28:0/85549116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de186e7ca0 con 0x55de186635d0 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8df9d9a640 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de186635d0 0x55de18667f50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8df0004770 tx=0x7f8df006a700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b177062334cc82 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8ddd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/85549116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df009fdc0 con 0x55de186635d0 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/85549116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f8df0098080 con 0x55de186635d0 2024-09-15T22:06:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.946+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dec023070 con 0x55de186635d0 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/85549116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8df00a4680 con 0x55de186635d0 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 10 client.17433 fetch_fsmap learned FSMap version 34 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55de186e8f00 con 0x55de186635d0 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/85549116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec023070 con 0x55de186635d0 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/85549116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f8df009ae60 con 0x55de186635d0 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 10 client.17433 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 10 client.17433 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 20 client.17433 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de186e95b0 0x55de186eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 4 client.17433 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55de185d0d70 con 0x55de186e95b0 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.947+0000 7f8df9599640 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de186e95b0 0x55de186eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8df9599640 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de186e95b0 0x55de186eb9d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4756d54e0e41e26 server_cookie=1040d3e55f2c032a in_seq=0 out_seq=0 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8ddd7fa640 10 client.17433 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8ddd7fa640 1 -- 172.21.15.28:0/85549116 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55de185d0d70 con 0x55de186e95b0 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8ddd7fa640 10 client.17433 handle_command_reply: tid=0 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 client.17433 shutdown 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de186635d0 msgr2=0x55de18667f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de186635d0 0x55de18667f50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8df0004770 tx=0x7f8df006a700 comp rx=0 tx=0).stop 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 shutdown_connections 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55de1866e370 0x55de186689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de18664b40 0x55de18668490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55de186e95b0 0x55de186eb9d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 --2- 172.21.15.28:0/85549116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de186635d0 0x55de18667f50 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 >> 172.21.15.28:0/85549116 conn(0x55de1862c310 msgr2=0x55de186792d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8ddd7fa640 0 client.17433 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 shutdown_connections 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.948+0000 7f8dfe3d7740 1 -- 172.21.15.28:0/85549116 wait complete. 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfe3d7740 20 client.17433 trim_cache size 0 max 16384 2024-09-15T22:06:37.958 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:37.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8de00a3430 msgr2=0x7f8de00a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8de00a3430 0x7f8de00a58f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8dec005c40 tx=0x7f8dec009280 comp rx=0 tx=0).stop 2024-09-15T22:06:37.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4100b20 msgr2=0x7f8df41a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:37.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4100b20 0x7f8df41a03f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f8df000d950 tx=0x7f8df000de10 comp rx=0 tx=0).stop 2024-09-15T22:06:37.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 shutdown_connections 2024-09-15T22:06:37.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f8de00a3430 0x7f8de00a58f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8df410e500 0x7f8df419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8df4101440 0x7f8df41a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 --2- 172.21.15.28:0/4267523707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df4100b20 0x7f8df41a03f0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:37.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 >> 172.21.15.28:0/4267523707 conn(0x7f8df40fc7d0 msgr2=0x7f8df40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:37.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 shutdown_connections 2024-09-15T22:06:37.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:37.949+0000 7f8dfbffe640 1 -- 172.21.15.28:0/4267523707 wait complete. 2024-09-15T22:06:38.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:37 smithi028 ceph-mon[30969]: pgmap v285: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-09-15T22:06:38.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:37 smithi203 ceph-mon[35546]: pgmap v285: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-09-15T22:06:38.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:37 smithi174 ceph-mon[35244]: pgmap v285: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-09-15T22:06:38.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:06:38.982 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.477+0000 7f874f46d640 1 -- 172.21.15.28:0/2588844498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 msgr2=0x7f8748100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.477+0000 7f874f46d640 1 --2- 172.21.15.28:0/2588844498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f8748100090 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8744009900 tx=0x7f874402a400 comp rx=0 tx=0).stop 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- 172.21.15.28:0/2588844498 shutdown_connections 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 --2- 172.21.15.28:0/2588844498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874810e810 0x7f8748110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 --2- 172.21.15.28:0/2588844498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8748102fc0 0x7f87481005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 --2- 172.21.15.28:0/2588844498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f8748100090 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- 172.21.15.28:0/2588844498 >> 172.21.15.28:0/2588844498 conn(0x7f8748077210 msgr2=0x7f87480fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- 172.21.15.28:0/2588844498 shutdown_connections 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- 172.21.15.28:0/2588844498 wait complete. 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 Processor -- start 2024-09-15T22:06:39.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- start start 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f87481a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874810e810 0x7f87481a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f874819a5c0 0x7f874819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8748114290 con 0x7f8748102b50 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8748114110 con 0x7f874819a5c0 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.478+0000 7f874f46d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8748114410 con 0x7f874810e810 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f87481a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f87481a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45282/0 (socket says 172.21.15.28:45282) 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 -- 172.21.15.28:0/2006702997 learned_addr learned my addr 172.21.15.28:0/2006702997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874ca08640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874810e810 0x7f87481a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874da0a640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f874819a5c0 0x7f874819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 -- 172.21.15.28:0/2006702997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874810e810 msgr2=0x7f87481a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874810e810 0x7f87481a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 -- 172.21.15.28:0/2006702997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f874819a5c0 msgr2=0x7f874819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f874819a5c0 0x7f874819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 -- 172.21.15.28:0/2006702997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874819b0e0 con 0x7f8748102b50 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874d209640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f87481a0400 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f8744005ab0 tx=0x7f87440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f873e7fc640 1 -- 172.21.15.28:0/2006702997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874403f030 con 0x7f8748102b50 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874819b370 con 0x7f8748102b50 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f873e7fc640 1 -- 172.21.15.28:0/2006702997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8744003ce0 con 0x7f8748102b50 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.479+0000 7f873e7fc640 1 -- 172.21.15.28:0/2006702997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744035070 con 0x7f8748102b50 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.480+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87481a7630 con 0x7f8748102b50 2024-09-15T22:06:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.482+0000 7f873e7fc640 1 -- 172.21.15.28:0/2006702997 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f8744042030 con 0x7f8748102b50 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.482+0000 7f8751846740 1 Processor -- start 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.482+0000 7f8751846740 1 -- start start 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.482+0000 7f8751846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b7ab40 0x55c575b79090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f8751846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b79a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f8751846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575b86760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f8751846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c575adf2f0 con 0x55c575b84370 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f8751846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c575a86cf0 con 0x55c575b795d0 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f8751846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c575adb2b0 con 0x55c575b7ab40 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f873e7fc640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87340a33e0 0x7f87340a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575b86760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575b86760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45286/0 (socket says 172.21.15.28:45286) 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 -- 172.21.15.28:0/2864310874 learned_addr learned my addr 172.21.15.28:0/2864310874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874da0a640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87340a33e0 0x7f87340a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f873e7fc640 1 -- 172.21.15.28:0/2006702997 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f87440f06b0 con 0x7f8748102b50 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874da0a640 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b7ab40 0x55c575b79090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874d209640 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b79a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 -- 172.21.15.28:0/2864310874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b7ab40 msgr2=0x55c575b79090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b7ab40 0x55c575b79090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 -- 172.21.15.28:0/2864310874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 msgr2=0x55c575b79a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b79a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 -- 172.21.15.28:0/2864310874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c575a6b3a0 con 0x55c575b84370 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874d209640 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b79a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874da0a640 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b7ab40 0x55c575b79090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874ca08640 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575b86760 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f873800b760 tx=0x7f873800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a3ad543b04a88 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.483+0000 7f874da0a640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87340a33e0 0x7f87340a58a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8740009fd0 tx=0x7f8740009540 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f87337fe640 1 -- 172.21.15.28:0/2864310874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873800d950 con 0x55c575b84370 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f87337fe640 1 -- 172.21.15.28:0/2864310874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8738019070 con 0x55c575b84370 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f87337fe640 1 -- 172.21.15.28:0/2864310874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8738014e00 con 0x55c575b84370 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 -- 172.21.15.28:0/2864310874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 msgr2=0x55c575b86760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575b86760 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f873800b760 tx=0x7f873800bc20 comp rx=0 tx=0).stop 2024-09-15T22:06:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 -- 172.21.15.28:0/2864310874 shutdown_connections 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575b86760 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b79a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 --2- 172.21.15.28:0/2864310874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b7ab40 0x55c575b79090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 -- 172.21.15.28:0/2864310874 >> 172.21.15.28:0/2864310874 conn(0x55c575b42310 msgr2=0x55c575b75c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 -- 172.21.15.28:0/2864310874 shutdown_connections 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.484+0000 7f8751846740 1 -- 172.21.15.28:0/2864310874 wait complete. 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 -- start start 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 Processor -- start 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b795d0 0x55c575b83d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b80fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b81530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c575b87070 con 0x55c575b795d0 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c575a9a060 con 0x55c575b7ab40 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c575b871f0 con 0x55c575b84370 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f8751846740 10 client.0 _ll_register_callbacks cb 0x55c5757ed8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f874d209640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b795d0 0x55c575b83d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f874d209640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b795d0 0x55c575b83d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45290/0 (socket says 172.21.15.28:45290) 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f874d209640 1 -- 172.21.15.28:0/1505912732 learned_addr learned my addr 172.21.15.28:0/1505912732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.485+0000 7f874ca08640 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b80fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f874da0a640 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b81530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f874d209640 1 -- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 msgr2=0x55c575b81530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f874d209640 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b81530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f874d209640 1 -- 172.21.15.28:0/1505912732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 msgr2=0x55c575b80fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f874d209640 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b80fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f874d209640 1 -- 172.21.15.28:0/1505912732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5758503c0 con 0x55c575b795d0 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f874d209640 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b795d0 0x55c575b83d00 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f87440bc510 tx=0x7f87440bc410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a395bd03e904227f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8730ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8730ff9640 1 -- 172.21.15.28:0/1505912732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744049070 con 0x55c575b795d0 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8730ff9640 1 -- 172.21.15.28:0/1505912732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f874403b460 con 0x55c575b795d0 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8738019070 con 0x55c575b795d0 2024-09-15T22:06:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8730ff9640 1 -- 172.21.15.28:0/1505912732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f87440bc580 con 0x55c575b795d0 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8751846740 10 client.17445 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c575b92aa0 con 0x55c575b795d0 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8730ff9640 1 -- 172.21.15.28:0/1505912732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8738019070 con 0x55c575b795d0 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.486+0000 7f8730ff9640 1 -- 172.21.15.28:0/1505912732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f87440edc70 con 0x55c575b795d0 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f8751846740 10 client.17445 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f8751846740 10 client.17445 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f8751846740 20 client.17445 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f8751846740 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575b93fa0 0x55c575b963f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f8751846740 4 client.17445 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c575aec860 con 0x55c575b93fa0 2024-09-15T22:06:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f874ca08640 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575b93fa0 0x55c575b963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f874ca08640 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575b93fa0 0x55c575b963f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a75ef1b9772684fe server_cookie=6a9f88e5752cb874 in_seq=0 out_seq=0 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.487+0000 7f8730ff9640 10 client.17445 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8730ff9640 1 -- 172.21.15.28:0/1505912732 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c575aec860 con 0x55c575b93fa0 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8730ff9640 10 client.17445 handle_command_reply: tid=0 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 client.17445 shutdown 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b795d0 msgr2=0x55c575b83d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b795d0 0x55c575b83d00 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f87440bc510 tx=0x7f87440bc410 comp rx=0 tx=0).stop 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 shutdown_connections 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575b93fa0 0x55c575b963f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b81530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b80fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 --2- 172.21.15.28:0/1505912732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b795d0 0x55c575b83d00 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8730ff9640 0 client.17445 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.488+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 >> 172.21.15.28:0/1505912732 conn(0x55c575b42310 msgr2=0x55c575b7b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.489+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 shutdown_connections 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.489+0000 7f8751846740 1 -- 172.21.15.28:0/1505912732 wait complete. 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.489+0000 7f8751846740 20 client.17445 trim_cache size 0 max 16384 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 Processor -- start 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 -- start start 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b7f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b7ab40 0x55c575b7c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b7c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87481b1280 con 0x55c575b7ab40 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f87481b2dc0 con 0x55c575b795d0 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f8751846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87481b1950 con 0x55c575b84370 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f874ca08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b7ab40 0x55c575b7c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f874ca08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b7ab40 0x55c575b7c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45306/0 (socket says 172.21.15.28:45306) 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f874ca08640 1 -- 172.21.15.28:0/1608044741 learned_addr learned my addr 172.21.15.28:0/1608044741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f874da0a640 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b7c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.502+0000 7f874d209640 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b7f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874da0a640 1 -- 172.21.15.28:0/1608044741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 msgr2=0x55c575b7f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874da0a640 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b7f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874da0a640 1 -- 172.21.15.28:0/1608044741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b7ab40 msgr2=0x55c575b7c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874da0a640 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b7ab40 0x55c575b7c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874da0a640 1 -- 172.21.15.28:0/1608044741 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c575b7ff60 con 0x55c575b84370 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874d209640 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b7f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874ca08640 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b7ab40 0x55c575b7c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.503+0000 7f874da0a640 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b7c660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8740009440 tx=0x7f87400093d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=167d613628fd55fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8732ffd640 1 -- 172.21.15.28:0/1608044741 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740013040 con 0x55c575b84370 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8732ffd640 1 -- 172.21.15.28:0/1608044741 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8740027070 con 0x55c575b84370 2024-09-15T22:06:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8732ffd640 1 -- 172.21.15.28:0/1608044741 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87400229a0 con 0x55c575b84370 2024-09-15T22:06:39.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 -- 172.21.15.28:0/1608044741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 msgr2=0x55c575b7c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b7c660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8740009440 tx=0x7f87400093d0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 -- 172.21.15.28:0/1608044741 shutdown_connections 2024-09-15T22:06:39.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b84370 0x55c575b7c660 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b7ab40 0x55c575b7c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 --2- 172.21.15.28:0/1608044741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b795d0 0x55c575b7f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 -- 172.21.15.28:0/1608044741 >> 172.21.15.28:0/1608044741 conn(0x55c575b42310 msgr2=0x55c575b83030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.504+0000 7f8751846740 1 -- 172.21.15.28:0/1608044741 shutdown_connections 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 -- 172.21.15.28:0/1608044741 wait complete. 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 -- start start 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 Processor -- start 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b795d0 0x55c575b88960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b88ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575bfdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87481b26f0 con 0x55c575b84370 2024-09-15T22:06:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f87481b2020 con 0x55c575b7ab40 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87481af280 con 0x55c575b795d0 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f8751846740 10 client.0 _ll_register_callbacks cb 0x7f87440044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.505+0000 7f874da0a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575bfdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575bfdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45314/0 (socket says 172.21.15.28:45314) 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 -- 172.21.15.28:0/2083198276 learned_addr learned my addr 172.21.15.28:0/2083198276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874d209640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b795d0 0x55c575b88960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874ca08640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b88ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 -- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b795d0 msgr2=0x55c575b88960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b795d0 0x55c575b88960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 -- 172.21.15.28:0/2083198276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 msgr2=0x55c575b88ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:39.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b88ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 -- 172.21.15.28:0/2083198276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c575bfe1b0 con 0x55c575b84370 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874ca08640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b88ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f874da0a640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575bfdc70 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8740023dd0 tx=0x7f8740023e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f36a8a51e31bbc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f8730ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f8730ff9640 1 -- 172.21.15.28:0/2083198276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874002f070 con 0x55c575b84370 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f8730ff9640 1 -- 172.21.15.28:0/2083198276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f8740018820 con 0x55c575b84370 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8740027070 con 0x55c575b84370 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f8730ff9640 1 -- 172.21.15.28:0/2083198276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f874003d050 con 0x55c575b84370 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f8751846740 10 client.17451 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.506+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c575bff4b0 con 0x55c575b84370 2024-09-15T22:06:39.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8730ff9640 1 -- 172.21.15.28:0/2083198276 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8740027070 con 0x55c575b84370 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8730ff9640 1 -- 172.21.15.28:0/2083198276 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f8740034760 con 0x55c575b84370 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8751846740 10 client.17451 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8751846740 10 client.17451 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8751846740 20 client.17451 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8751846740 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575bffb70 0x55c575c01f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8751846740 4 client.17451 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c575ae6d70 con 0x55c575bffb70 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.507+0000 7f874d209640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575bffb70 0x55c575c01f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f874d209640 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575bffb70 0x55c575c01f60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=35886bafc88aeb16 server_cookie=3f9f93658c8e920c in_seq=0 out_seq=0 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8730ff9640 10 client.17451 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8730ff9640 1 -- 172.21.15.28:0/2083198276 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c575ae6d70 con 0x55c575bffb70 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8730ff9640 10 client.17451 handle_command_reply: tid=0 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 client.17451 shutdown 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 msgr2=0x55c575bfdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575bfdc70 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8740023dd0 tx=0x7f8740023e00 comp rx=0 tx=0).stop 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 shutdown_connections 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55c575bffb70 0x55c575c01f60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c575b84370 0x55c575bfdc70 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c575b7ab40 0x55c575b88ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 --2- 172.21.15.28:0/2083198276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c575b795d0 0x55c575b88960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 >> 172.21.15.28:0/2083198276 conn(0x55c575b42310 msgr2=0x55c575b83030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8730ff9640 0 client.17451 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:06:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 shutdown_connections 2024-09-15T22:06:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 1 -- 172.21.15.28:0/2083198276 wait complete. 2024-09-15T22:06:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.508+0000 7f8751846740 20 client.17451 trim_cache size 0 max 16384 2024-09-15T22:06:39.529 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87340a33e0 msgr2=0x7f87340a58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87340a33e0 0x7f87340a58a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8740009fd0 tx=0x7f8740009540 comp rx=0 tx=0).stop 2024-09-15T22:06:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 msgr2=0x7f87481a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f87481a0400 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f8744005ab0 tx=0x7f87440042f0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 shutdown_connections 2024-09-15T22:06:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f87340a33e0 0x7f87340a58a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f874819a5c0 0x7f874819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874810e810 0x7f87481a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 --2- 172.21.15.28:0/2006702997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8748102b50 0x7f87481a0400 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 >> 172.21.15.28:0/2006702997 conn(0x7f8748077210 msgr2=0x7f874810fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 shutdown_connections 2024-09-15T22:06:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:39.509+0000 7f874f46d640 1 -- 172.21.15.28:0/2006702997 wait complete. 2024-09-15T22:06:40.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:39 smithi028 ceph-mon[30969]: pgmap v286: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:06:40.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:39 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:06:40.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:39 smithi203 ceph-mon[35546]: pgmap v286: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:06:40.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:39 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:06:40.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:39 smithi174 ceph-mon[35244]: pgmap v286: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:06:40.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:39 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:06:40.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:06:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 -- 172.21.15.28:0/1625330223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074eb0 msgr2=0x7fd8a8112660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 --2- 172.21.15.28:0/1625330223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074eb0 0x7fd8a8112660 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd898009900 tx=0x7fd89802a480 comp rx=0 tx=0).stop 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 -- 172.21.15.28:0/1625330223 shutdown_connections 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 --2- 172.21.15.28:0/1625330223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8a8112ba0 0x7fd8a8114f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 --2- 172.21.15.28:0/1625330223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074eb0 0x7fd8a8112660 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 --2- 172.21.15.28:0/1625330223 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a8074590 0x7fd8a8074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 -- 172.21.15.28:0/1625330223 >> 172.21.15.28:0/1625330223 conn(0x7fd8a80fd8b0 msgr2=0x7fd8a80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 -- 172.21.15.28:0/1625330223 shutdown_connections 2024-09-15T22:06:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 -- 172.21.15.28:0/1625330223 wait complete. 2024-09-15T22:06:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 Processor -- start 2024-09-15T22:06:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.046+0000 7fd8afe60640 1 -- start start 2024-09-15T22:06:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8afe60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074590 0x7fd8a81a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8afe60640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a8074eb0 0x7fd8a81a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8afe60640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8a8112ba0 0x7fd8a819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8afe60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a8118550 con 0x7fd8a8074590 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8afe60640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81183d0 con 0x7fd8a8112ba0 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8afe60640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81186d0 con 0x7fd8a8074eb0 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074590 0x7fd8a81a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074590 0x7fd8a81a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50016/0 (socket says 172.21.15.28:50016) 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 -- 172.21.15.28:0/670078388 learned_addr learned my addr 172.21.15.28:0/670078388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8a8112ba0 0x7fd8a819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a8074eb0 0x7fd8a81a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 -- 172.21.15.28:0/670078388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a8074eb0 msgr2=0x7fd8a81a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a8074eb0 0x7fd8a81a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 -- 172.21.15.28:0/670078388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8a8112ba0 msgr2=0x7fd8a819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8a8112ba0 0x7fd8a819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 -- 172.21.15.28:0/670078388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a819f010 con 0x7fd8a8074590 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8a8112ba0 0x7fd8a819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a8074eb0 0x7fd8a81a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.047+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074590 0x7fd8a81a4720 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400d950 tx=0x7fd8a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.048+0000 7fd89effd640 1 -- 172.21.15.28:0/670078388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4017030 con 0x7fd8a8074590 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.048+0000 7fd89effd640 1 -- 172.21.15.28:0/670078388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8a4004020 con 0x7fd8a8074590 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.048+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a819f300 con 0x7fd8a8074590 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.048+0000 7fd89effd640 1 -- 172.21.15.28:0/670078388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40047a0 con 0x7fd8a8074590 2024-09-15T22:06:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.048+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a81ab5f0 con 0x7fd8a8074590 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.050+0000 7fd89effd640 1 -- 172.21.15.28:0/670078388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd8a40041d0 con 0x7fd8a8074590 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.050+0000 7fd8b2239740 1 Processor -- start 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8b2239740 1 -- start start 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328bb40 0x55c22328a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328a5d0 0x55c22328aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223297760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8b2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2231f02f0 con 0x55c22328a5d0 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8b2239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c223197cf0 con 0x55c223295370 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8b2239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c2231ec2b0 con 0x55c22328bb40 2024-09-15T22:06:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328a5d0 0x55c22328aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328a5d0 0x55c22328aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50022/0 (socket says 172.21.15.28:50022) 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 -- 172.21.15.28:0/3999796083 learned_addr learned my addr 172.21.15.28:0/3999796083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223297760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328bb40 0x55c22328a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd89effd640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd8940a33b0 0x7fd8940a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 -- 172.21.15.28:0/3999796083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328bb40 msgr2=0x55c22328a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328bb40 0x55c22328a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 -- 172.21.15.28:0/3999796083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 msgr2=0x55c223297760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223297760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.051+0000 7fd8adbfc640 1 -- 172.21.15.28:0/3999796083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c22317c3a0 con 0x55c22328a5d0 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.052+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd8940a33b0 0x7fd8940a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.052+0000 7fd89effd640 1 -- 172.21.15.28:0/670078388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fd8a40cfba0 con 0x7fd8a8074590 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.052+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328a5d0 0x55c22328aa10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40cceb0 tx=0x7fd8a40afc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6356a75e0eb32950 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.052+0000 7fd893fff640 1 -- 172.21.15.28:0/3999796083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a409b610 con 0x55c22328a5d0 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.052+0000 7fd893fff640 1 -- 172.21.15.28:0/3999796083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8a409c1d0 con 0x55c22328a5d0 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.052+0000 7fd893fff640 1 -- 172.21.15.28:0/3999796083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40a1010 con 0x55c22328a5d0 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 -- 172.21.15.28:0/3999796083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328a5d0 msgr2=0x55c22328aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328a5d0 0x55c22328aa10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40cceb0 tx=0x7fd8a40afc30 comp rx=0 tx=0).stop 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd8940a33b0 0x7fd8940a5870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8076600 tx=0x7fd8a0004ab0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 -- 172.21.15.28:0/3999796083 shutdown_connections 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223297760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328a5d0 0x55c22328aa10 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 --2- 172.21.15.28:0/3999796083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328bb40 0x55c22328a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 -- 172.21.15.28:0/3999796083 >> 172.21.15.28:0/3999796083 conn(0x55c223253310 msgr2=0x55c223286c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 -- 172.21.15.28:0/3999796083 shutdown_connections 2024-09-15T22:06:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 -- 172.21.15.28:0/3999796083 wait complete. 2024-09-15T22:06:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 -- start start 2024-09-15T22:06:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.053+0000 7fd8b2239740 1 Processor -- start 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c223294d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223292040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223292650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8b2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c223298070 con 0x55c22328bb40 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8b2239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c2231ab060 con 0x55c223295370 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8b2239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c2232981f0 con 0x55c22328a5d0 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8b2239740 10 client.0 _ll_register_callbacks cb 0x55c222efe8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223292040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223292040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50026/0 (socket says 172.21.15.28:50026) 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1547067865 learned_addr learned my addr 172.21.15.28:0/1547067865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c223294d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223292650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 msgr2=0x55c223294d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c223294d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1547067865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 msgr2=0x55c223292650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223292650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1547067865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c222f613c0 con 0x55c22328bb40 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223292040 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd898009230 tx=0x7fd8980315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685505672a58f809 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.054+0000 7fd8917fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:41.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8917fa640 1 -- 172.21.15.28:0/1547067865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898040070 con 0x55c22328bb40 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8917fa640 1 -- 172.21.15.28:0/1547067865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fd898004240 con 0x55c22328bb40 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8a409c1d0 con 0x55c22328bb40 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8917fa640 1 -- 172.21.15.28:0/1547067865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd89804d050 con 0x55c22328bb40 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 10 client.17463 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c2232a3b40 con 0x55c22328bb40 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8917fa640 1 -- 172.21.15.28:0/1547067865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8a409c1d0 con 0x55c22328bb40 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8917fa640 1 -- 172.21.15.28:0/1547067865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7fd8980049c0 con 0x55c22328bb40 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 10 client.17463 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 10 client.17463 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 20 client.17463 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c2232a5000 0x55c2232a7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 4 client.17463 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:41.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.055+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2231fd860 con 0x55c2232a5000 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.056+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c2232a5000 0x55c2232a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.056+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c2232a5000 0x55c2232a7460 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8df3c05943e581e0 server_cookie=fbb873e026a23b6 in_seq=0 out_seq=0 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.056+0000 7fd8917fa640 10 client.17463 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8917fa640 1 -- 172.21.15.28:0/1547067865 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c2231fd860 con 0x55c2232a5000 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8917fa640 10 client.17463 handle_command_reply: tid=0 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 client.17463 shutdown 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 msgr2=0x55c223292040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223292040 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fd898009230 tx=0x7fd8980315b0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 shutdown_connections 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c2232a5000 0x55c2232a7460 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c223292650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223292040 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1547067865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c223294d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 >> 172.21.15.28:0/1547067865 conn(0x55c223253310 msgr2=0x55c22328c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:41.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8917fa640 0 client.17463 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:41.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 shutdown_connections 2024-09-15T22:06:41.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 1 -- 172.21.15.28:0/1547067865 wait complete. 2024-09-15T22:06:41.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.057+0000 7fd8b2239740 20 client.17463 trim_cache size 0 max 16384 2024-09-15T22:06:41.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 Processor -- start 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 -- start start 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c2232a1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223291900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c22328d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b5880 con 0x55c22328bb40 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b73c0 con 0x55c223295370 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8b2239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b5f50 con 0x55c22328a5d0 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8ad3fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223291900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8ad3fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223291900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50030/0 (socket says 172.21.15.28:50030) 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/599681868 learned_addr learned my addr 172.21.15.28:0/599681868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c22328d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.071+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c2232a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/599681868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 msgr2=0x55c2232a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c2232a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/599681868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 msgr2=0x55c22328d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c22328d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/599681868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c223292fe0 con 0x55c22328bb40 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223291900 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd898009900 tx=0x7fd89802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5940e9ea053061 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8937fe640 1 -- 172.21.15.28:0/599681868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898056070 con 0x55c22328bb40 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8937fe640 1 -- 172.21.15.28:0/599681868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd898004ba0 con 0x55c22328bb40 2024-09-15T22:06:41.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.072+0000 7fd8937fe640 1 -- 172.21.15.28:0/599681868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898040070 con 0x55c22328bb40 2024-09-15T22:06:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 -- 172.21.15.28:0/599681868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 msgr2=0x55c223291900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223291900 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd898009900 tx=0x7fd89802a480 comp rx=0 tx=0).stop 2024-09-15T22:06:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 -- 172.21.15.28:0/599681868 shutdown_connections 2024-09-15T22:06:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c223295370 0x55c22328d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c223291900 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 --2- 172.21.15.28:0/599681868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c22328a5d0 0x55c2232a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 -- 172.21.15.28:0/599681868 >> 172.21.15.28:0/599681868 conn(0x55c223253310 msgr2=0x55c223291130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 -- 172.21.15.28:0/599681868 shutdown_connections 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 -- 172.21.15.28:0/599681868 wait complete. 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 -- start start 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 Processor -- start 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.073+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c22328a5d0 0x55c223298b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c2232990c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8b2239740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c223295370 0x55c22329d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8b2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b6cf0 con 0x55c22328bb40 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8b2239740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b6620 con 0x55c22328a5d0 2024-09-15T22:06:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8b2239740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a81b3880 con 0x55c223295370 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8b2239740 10 client.0 _ll_register_callbacks cb 0x7fd898009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c2232990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c2232990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50044/0 (socket says 172.21.15.28:50044) 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1087023747 learned_addr learned my addr 172.21.15.28:0/1087023747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c223295370 0x55c22329d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c22328a5d0 0x55c223298b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c223295370 msgr2=0x55c22329d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c223295370 0x55c22329d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1087023747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c22328a5d0 msgr2=0x55c223298b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c22328a5d0 0x55c223298b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 -- 172.21.15.28:0/1087023747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c22329d7e0 con 0x55c22328bb40 2024-09-15T22:06:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ae3fd640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c223295370 0x55c22329d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c22328a5d0 0x55c223298b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8ad3fb640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c2232990c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd898009850 tx=0x7fd898005260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd6ac429037356 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8917fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8917fa640 1 -- 172.21.15.28:0/1087023747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898056040 con 0x55c22328bb40 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8917fa640 1 -- 172.21.15.28:0/1087023747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7fd8980046c0 con 0x55c22328bb40 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.074+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd898004ba0 con 0x55c22328bb40 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8917fa640 1 -- 172.21.15.28:0/1087023747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd898040040 con 0x55c22328bb40 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 10 client.17469 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c22330f750 con 0x55c22328bb40 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8917fa640 1 -- 172.21.15.28:0/1087023747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd898004ba0 con 0x55c22328bb40 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8917fa640 1 -- 172.21.15.28:0/1087023747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7fd89803cb90 con 0x55c22328bb40 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 10 client.17469 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 10 client.17469 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:06:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 20 client.17469 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c22330fe00 0x55c223312220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 4 client.17469 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2231f7d70 con 0x55c22330fe00 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.075+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c22330fe00 0x55c223312220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8917fa640 10 client.17469 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8adbfc640 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c22330fe00 0x55c223312220 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=363e36da840e6599 server_cookie=3e645ac1e3744560 in_seq=0 out_seq=0 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8917fa640 1 -- 172.21.15.28:0/1087023747 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2231f7d70 con 0x55c22330fe00 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8917fa640 10 client.17469 handle_command_reply: tid=0 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 client.17469 shutdown 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 msgr2=0x55c2232990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c2232990c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd898009850 tx=0x7fd898005260 comp rx=0 tx=0).stop 2024-09-15T22:06:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 shutdown_connections 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55c22330fe00 0x55c223312220 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c223295370 0x55c22329d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c22328bb40 0x55c2232990c0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8917fa640 0 client.17469 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 --2- 172.21.15.28:0/1087023747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c22328a5d0 0x55c223298b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 >> 172.21.15.28:0/1087023747 conn(0x55c223253310 msgr2=0x55c223291130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 shutdown_connections 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 1 -- 172.21.15.28:0/1087023747 wait complete. 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.076+0000 7fd8b2239740 20 client.17469 trim_cache size 0 max 16384 2024-09-15T22:06:41.084 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd8940a33b0 msgr2=0x7fd8940a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd8940a33b0 0x7fd8940a5870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8076600 tx=0x7fd8a0004ab0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074590 msgr2=0x7fd8a81a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074590 0x7fd8a81a4720 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400d950 tx=0x7fd8a400de10 comp rx=0 tx=0).stop 2024-09-15T22:06:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 shutdown_connections 2024-09-15T22:06:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd8940a33b0 0x7fd8940a5870 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8a8112ba0 0x7fd8a819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8a8074eb0 0x7fd8a81a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 --2- 172.21.15.28:0/670078388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8a8074590 0x7fd8a81a4720 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 >> 172.21.15.28:0/670078388 conn(0x7fd8a80fd8b0 msgr2=0x7fd8a80ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 shutdown_connections 2024-09-15T22:06:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:41.077+0000 7fd8afe60640 1 -- 172.21.15.28:0/670078388 wait complete. 2024-09-15T22:06:41.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:06:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:42.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:41 smithi028 ceph-mon[30969]: pgmap v287: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:06:42.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:41 smithi203 ceph-mon[35546]: pgmap v287: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:06:42.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:41 smithi174 ceph-mon[35244]: pgmap v287: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:06:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 -- 172.21.15.28:0/3396700430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98101055b0 msgr2=0x7f981010d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 --2- 172.21.15.28:0/3396700430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98101055b0 0x7f981010d130 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9808009ca0 tx=0x7f980802a8c0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 -- 172.21.15.28:0/3396700430 shutdown_connections 2024-09-15T22:06:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 --2- 172.21.15.28:0/3396700430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98101055b0 0x7f981010d130 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 --2- 172.21.15.28:0/3396700430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9810100e30 0x7f9810104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 --2- 172.21.15.28:0/3396700430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9810100480 0x7f9810100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 -- 172.21.15.28:0/3396700430 >> 172.21.15.28:0/3396700430 conn(0x7f9810077210 msgr2=0x7f98100fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 -- 172.21.15.28:0/3396700430 shutdown_connections 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.556+0000 7f9817e84640 1 -- 172.21.15.28:0/3396700430 wait complete. 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 Processor -- start 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 -- start start 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9810100480 0x7f9810198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9810100e30 0x7f9810198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98101055b0 0x7f981019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f981010fd90 con 0x7f9810100480 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f981010fc10 con 0x7f98101055b0 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9817e84640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f981010ff10 con 0x7f9810100e30 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9810100480 0x7f9810198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9810100480 0x7f9810198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50074/0 (socket says 172.21.15.28:50074) 2024-09-15T22:06:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 -- 172.21.15.28:0/497536314 learned_addr learned my addr 172.21.15.28:0/497536314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f981541f640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9810100e30 0x7f9810198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9816421640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98101055b0 0x7f981019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 -- 172.21.15.28:0/497536314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9810100e30 msgr2=0x7f9810198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9810100e30 0x7f9810198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 -- 172.21.15.28:0/497536314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98101055b0 msgr2=0x7f981019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98101055b0 0x7f981019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.557+0000 7f9815c20640 1 -- 172.21.15.28:0/497536314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981019d3c0 con 0x7f9810100480 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.558+0000 7f9815c20640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9810100480 0x7f9810198410 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f980c00ddc0 tx=0x7f980c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.558+0000 7f9806ffd640 1 -- 172.21.15.28:0/497536314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c01b070 con 0x7f9810100480 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.558+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981019d6b0 con 0x7f9810100480 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.558+0000 7f9806ffd640 1 -- 172.21.15.28:0/497536314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f980c00be40 con 0x7f9810100480 2024-09-15T22:06:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.558+0000 7f9806ffd640 1 -- 172.21.15.28:0/497536314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c016bb0 con 0x7f9810100480 2024-09-15T22:06:42.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.558+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98101a4f90 con 0x7f9810100480 2024-09-15T22:06:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.560+0000 7f9806ffd640 1 -- 172.21.15.28:0/497536314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f980c01e070 con 0x7f9810100480 2024-09-15T22:06:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.560+0000 7f981a25d740 1 Processor -- start 2024-09-15T22:06:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981a25d740 1 -- start start 2024-09-15T22:06:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981a25d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e17090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981a25d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e175d0 0x55fb27e17a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981a25d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e24760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981a25d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb27d7d2f0 con 0x55fb27e18b40 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981a25d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fb27d24cf0 con 0x55fb27e22370 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981a25d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fb27d792b0 con 0x55fb27e175d0 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f9816421640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e17090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f9816421640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e17090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50090/0 (socket says 172.21.15.28:50090) 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f9816421640 1 -- 172.21.15.28:0/2858513739 learned_addr learned my addr 172.21.15.28:0/2858513739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f981541f640 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e24760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f9815c20640 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e175d0 0x55fb27e17a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f9816421640 1 -- 172.21.15.28:0/2858513739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e175d0 msgr2=0x55fb27e17a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.561+0000 7f9816421640 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e175d0 0x55fb27e17a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f9816421640 1 -- 172.21.15.28:0/2858513739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 msgr2=0x55fb27e24760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f9806ffd640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f97fc0a33b0 0x7f97fc0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f9816421640 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e24760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f9816421640 1 -- 172.21.15.28:0/2858513739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb27d093a0 con 0x55fb27e18b40 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f9806ffd640 1 -- 172.21.15.28:0/497536314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f980c0cfe20 con 0x7f9810100480 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f9816421640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f97fc0a33b0 0x7f97fc0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f9816421640 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e17090 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f9808005cc0 tx=0x7f9808005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809023c46bf9f212 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f97fbfff640 1 -- 172.21.15.28:0/2858513739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980804c070 con 0x55fb27e18b40 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f97fbfff640 1 -- 172.21.15.28:0/2858513739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9808036070 con 0x55fb27e18b40 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.562+0000 7f97fbfff640 1 -- 172.21.15.28:0/2858513739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9808004530 con 0x55fb27e18b40 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 -- 172.21.15.28:0/2858513739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 msgr2=0x55fb27e17090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f9816421640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f97fc0a33b0 0x7f97fc0a5870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9808009a60 tx=0x7f980802f9e0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e17090 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f9808005cc0 tx=0x7f9808005f70 comp rx=0 tx=0).stop 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 -- 172.21.15.28:0/2858513739 shutdown_connections 2024-09-15T22:06:42.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e24760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e175d0 0x55fb27e17a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 --2- 172.21.15.28:0/2858513739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e17090 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 -- 172.21.15.28:0/2858513739 >> 172.21.15.28:0/2858513739 conn(0x55fb27de0310 msgr2=0x55fb27e13c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 -- 172.21.15.28:0/2858513739 shutdown_connections 2024-09-15T22:06:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 -- 172.21.15.28:0/2858513739 wait complete. 2024-09-15T22:06:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.563+0000 7f981a25d740 1 -- start start 2024-09-15T22:06:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 1 Processor -- start 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e175d0 0x55fb27e21d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e18b40 0x55fb27e1efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e1f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb27e25070 con 0x55fb27e175d0 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fb27d38060 con 0x55fb27e22370 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fb27e251f0 con 0x55fb27e18b40 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981a25d740 10 client.0 _ll_register_callbacks cb 0x55fb27a8b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e175d0 0x55fb27e21d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e175d0 0x55fb27e21d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50100/0 (socket says 172.21.15.28:50100) 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 -- 172.21.15.28:0/1002417163 learned_addr learned my addr 172.21.15.28:0/1002417163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f981541f640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e18b40 0x55fb27e1efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9816421640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e1f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 -- 172.21.15.28:0/1002417163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e18b40 msgr2=0x55fb27e1efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e18b40 0x55fb27e1efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 -- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 msgr2=0x55fb27e1f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e1f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.564+0000 7f9815c20640 1 -- 172.21.15.28:0/1002417163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb27aee3c0 con 0x55fb27e175d0 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f9816421640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e1f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f9815c20640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e175d0 0x55fb27e21d30 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f980c09bb40 tx=0x7f980c016650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312e9bea9142c708 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f97f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f97f97fa640 1 -- 172.21.15.28:0/1002417163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c09e070 con 0x55fb27e175d0 2024-09-15T22:06:42.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f97f97fa640 1 -- 172.21.15.28:0/1002417163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f980c0a4620 con 0x55fb27e175d0 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9808036070 con 0x55fb27e175d0 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f97f97fa640 1 -- 172.21.15.28:0/1002417163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f980c0d3540 con 0x55fb27e175d0 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f981a25d740 10 client.17481 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fb27e30ad0 con 0x55fb27e175d0 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f97f97fa640 1 -- 172.21.15.28:0/1002417163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9808036070 con 0x55fb27e175d0 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f97f97fa640 1 -- 172.21.15.28:0/1002417163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f980c00b040 con 0x55fb27e175d0 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f981a25d740 10 client.17481 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.565+0000 7f981a25d740 10 client.17481 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.566+0000 7f981a25d740 20 client.17481 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.566+0000 7f981a25d740 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27e31f90 0x55fb27e343e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.566+0000 7f981a25d740 4 client.17481 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.566+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb27d8a860 con 0x55fb27e31f90 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.566+0000 7f981541f640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27e31f90 0x55fb27e343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.566+0000 7f981541f640 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27e31f90 0x55fb27e343e0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12879fb541ee5fa0 server_cookie=43ee20f3fdb3aecf in_seq=0 out_seq=0 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.566+0000 7f97f97fa640 10 client.17481 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f97f97fa640 1 -- 172.21.15.28:0/1002417163 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fb27d8a860 con 0x55fb27e31f90 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f97f97fa640 10 client.17481 handle_command_reply: tid=0 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 client.17481 shutdown 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e175d0 msgr2=0x55fb27e21d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e175d0 0x55fb27e21d30 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f980c09bb40 tx=0x7f980c016650 comp rx=0 tx=0).stop 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 shutdown_connections 2024-09-15T22:06:42.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27e31f90 0x55fb27e343e0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e22370 0x55fb27e1f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f97f97fa640 0 client.17481 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e18b40 0x55fb27e1efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 --2- 172.21.15.28:0/1002417163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e175d0 0x55fb27e21d30 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 >> 172.21.15.28:0/1002417163 conn(0x55fb27de0310 msgr2=0x55fb27e19960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 shutdown_connections 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.567+0000 7f981a25d740 1 -- 172.21.15.28:0/1002417163 wait complete. 2024-09-15T22:06:42.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.568+0000 7f981a25d740 20 client.17481 trim_cache size 0 max 16384 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.580+0000 7f981a25d740 1 Processor -- start 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.580+0000 7f981a25d740 1 -- start start 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.580+0000 7f981a25d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e20f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981a25d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e21490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981a25d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e1a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981a25d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98101af000 con 0x55fb27e18b40 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981a25d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98101b0b40 con 0x55fb27e175d0 2024-09-15T22:06:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981a25d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98101af6d0 con 0x55fb27e22370 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e21490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e21490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50108/0 (socket says 172.21.15.28:50108) 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 -- 172.21.15.28:0/4008048286 learned_addr learned my addr 172.21.15.28:0/4008048286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f9816421640 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e1a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f9815c20640 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e20f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 -- 172.21.15.28:0/4008048286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 msgr2=0x55fb27e1a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e1a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 -- 172.21.15.28:0/4008048286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 msgr2=0x55fb27e20f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e20f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 -- 172.21.15.28:0/4008048286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb27e1d530 con 0x55fb27e18b40 2024-09-15T22:06:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f9815c20640 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e20f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f981541f640 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e21490 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f98100ff630 tx=0x7f9800013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62486b23f7c41177 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f97fb7fe640 1 -- 172.21.15.28:0/4008048286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9800022070 con 0x55fb27e18b40 2024-09-15T22:06:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f97fb7fe640 1 -- 172.21.15.28:0/4008048286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9800011920 con 0x55fb27e18b40 2024-09-15T22:06:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.581+0000 7f97fb7fe640 1 -- 172.21.15.28:0/4008048286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980001ca50 con 0x55fb27e18b40 2024-09-15T22:06:42.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 -- 172.21.15.28:0/4008048286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 msgr2=0x55fb27e21490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e21490 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f98100ff630 tx=0x7f9800013830 comp rx=0 tx=0).stop 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 -- 172.21.15.28:0/4008048286 shutdown_connections 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e1a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e21490 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 --2- 172.21.15.28:0/4008048286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e20f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 -- 172.21.15.28:0/4008048286 >> 172.21.15.28:0/4008048286 conn(0x55fb27de0310 msgr2=0x55fb27e20840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 -- 172.21.15.28:0/4008048286 shutdown_connections 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 -- 172.21.15.28:0/4008048286 wait complete. 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.582+0000 7f981a25d740 1 -- start start 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 1 Processor -- start 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e2ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e25f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e26490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98101b0470 con 0x55fb27e18b40 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98101afda0 con 0x55fb27e175d0 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98101ad000 con 0x55fb27e22370 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981a25d740 10 client.0 _ll_register_callbacks cb 0x7f980c09bc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e25f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e25f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50116/0 (socket says 172.21.15.28:50116) 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 -- 172.21.15.28:0/391273355 learned_addr learned my addr 172.21.15.28:0/391273355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f9816421640 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e26490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f9815c20640 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e2ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 -- 172.21.15.28:0/391273355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 msgr2=0x55fb27e26490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e26490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 -- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 msgr2=0x55fb27e2ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e2ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.583+0000 7f981541f640 1 -- 172.21.15.28:0/391273355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb27e26cc0 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f981541f640 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e25f50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9800005890 tx=0x7f98000133c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac7afe86cbf3d04 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f97f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f97f97fa640 1 -- 172.21.15.28:0/391273355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980002a070 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f97f97fa640 1 -- 172.21.15.28:0/391273355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f9800011ad0 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9800011920 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f97f97fa640 1 -- 172.21.15.28:0/391273355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9800036050 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f981a25d740 10 client.17487 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fb27ea1060 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f97f97fa640 1 -- 172.21.15.28:0/391273355 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9800011920 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f97f97fa640 1 -- 172.21.15.28:0/391273355 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f9800026640 con 0x55fb27e18b40 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f981a25d740 10 client.17487 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.584+0000 7f981a25d740 10 client.17487 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f981a25d740 20 client.17487 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f981a25d740 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27ea16d0 0x55fb27ea3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f981a25d740 4 client.17487 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fb27d84d70 con 0x55fb27ea16d0 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f9815c20640 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27ea16d0 0x55fb27ea3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f9815c20640 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27ea16d0 0x55fb27ea3ac0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebc66c9630efe957 server_cookie=962d4b6896b0ab4b in_seq=0 out_seq=0 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f97f97fa640 10 client.17487 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f97f97fa640 1 -- 172.21.15.28:0/391273355 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fb27d84d70 con 0x55fb27ea16d0 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f97f97fa640 10 client.17487 handle_command_reply: tid=0 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.585+0000 7f981a25d740 1 client.17487 shutdown 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 msgr2=0x55fb27e25f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e25f50 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9800005890 tx=0x7f98000133c0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 shutdown_connections 2024-09-15T22:06:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x55fb27ea16d0 0x55fb27ea3ac0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fb27e22370 0x55fb27e26490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb27e18b40 0x55fb27e25f50 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f97f97fa640 0 client.17487 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 --2- 172.21.15.28:0/391273355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fb27e175d0 0x55fb27e2ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 >> 172.21.15.28:0/391273355 conn(0x55fb27de0310 msgr2=0x55fb27e21060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 shutdown_connections 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 1 -- 172.21.15.28:0/391273355 wait complete. 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f981a25d740 20 client.17487 trim_cache size 0 max 16384 2024-09-15T22:06:42.593 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:06:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f97fc0a33b0 msgr2=0x7f97fc0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f97fc0a33b0 0x7f97fc0a5870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9808009a60 tx=0x7f980802f9e0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9810100480 msgr2=0x7f9810198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9810100480 0x7f9810198410 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f980c00ddc0 tx=0x7f980c00a620 comp rx=0 tx=0).stop 2024-09-15T22:06:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 shutdown_connections 2024-09-15T22:06:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f97fc0a33b0 0x7f97fc0a5870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98101055b0 0x7f981019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9810100e30 0x7f9810198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 --2- 172.21.15.28:0/497536314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9810100480 0x7f9810198410 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.586+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 >> 172.21.15.28:0/497536314 conn(0x7f9810077210 msgr2=0x7f9810103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.587+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 shutdown_connections 2024-09-15T22:06:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:42.587+0000 7f9817e84640 1 -- 172.21.15.28:0/497536314 wait complete. 2024-09-15T22:06:44.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:43 smithi203 ceph-mon[35546]: pgmap v288: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:06:44.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:43 smithi174 ceph-mon[35244]: pgmap v288: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:06:44.374 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T22:06:44.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T22:06:44.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:43 smithi028 ceph-mon[30969]: pgmap v288: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-09-15T22:06:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 -- 172.21.15.28:0/985005422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b41036a0 msgr2=0x7f65b410bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 --2- 172.21.15.28:0/985005422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b41036a0 0x7f65b410bb90 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009900 tx=0x7f65a802a480 comp rx=0 tx=0).stop 2024-09-15T22:06:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 -- 172.21.15.28:0/985005422 shutdown_connections 2024-09-15T22:06:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 --2- 172.21.15.28:0/985005422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b410c160 0x7f65b410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 --2- 172.21.15.28:0/985005422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b41036a0 0x7f65b410bb90 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 --2- 172.21.15.28:0/985005422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65b4102d80 0x7f65b4103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 -- 172.21.15.28:0/985005422 >> 172.21.15.28:0/985005422 conn(0x7f65b40fc780 msgr2=0x7f65b40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 -- 172.21.15.28:0/985005422 shutdown_connections 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 -- 172.21.15.28:0/985005422 wait complete. 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.103+0000 7f65b9f07640 1 Processor -- start 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b9f07640 1 -- start start 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b9f07640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b4102d80 0x7f65b4198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b9f07640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65b41036a0 0x7f65b41989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b9f07640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b410c160 0x7f65b419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b9f07640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65b4111f40 con 0x7f65b4102d80 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b9f07640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65b4111dc0 con 0x7f65b410c160 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b9f07640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65b41120c0 con 0x7f65b41036a0 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b4102d80 0x7f65b4198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b4102d80 0x7f65b4198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50136/0 (socket says 172.21.15.28:50136) 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 -- 172.21.15.28:0/926200000 learned_addr learned my addr 172.21.15.28:0/926200000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b3fff640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b410c160 0x7f65b419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65b41036a0 0x7f65b41989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 -- 172.21.15.28:0/926200000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65b41036a0 msgr2=0x7f65b41989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65b41036a0 0x7f65b41989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 -- 172.21.15.28:0/926200000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b410c160 msgr2=0x7f65b419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b410c160 0x7f65b419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b37fe640 1 -- 172.21.15.28:0/926200000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b419d430 con 0x7f65b4102d80 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b3fff640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b410c160 0x7f65b419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.104+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65b41036a0 0x7f65b41989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.105+0000 7f65b37fe640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b4102d80 0x7f65b4198480 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f65a400d770 tx=0x7f65a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.105+0000 7f65b0ff9640 1 -- 172.21.15.28:0/926200000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4017070 con 0x7f65b4102d80 2024-09-15T22:06:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.105+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b419d720 con 0x7f65b4102d80 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.105+0000 7f65b0ff9640 1 -- 172.21.15.28:0/926200000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a4005500 con 0x7f65b4102d80 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.105+0000 7f65b0ff9640 1 -- 172.21.15.28:0/926200000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4004e00 con 0x7f65b4102d80 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.105+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b41a5000 con 0x7f65b4102d80 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.107+0000 7f65b0ff9640 1 -- 172.21.15.28:0/926200000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f65a4004fa0 con 0x7f65b4102d80 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.107+0000 7f65bc2e0740 1 Processor -- start 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65bc2e0740 1 -- start start 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fce9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcfb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56418fc4f280 con 0x56418fcea380 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56418fbf6b10 con 0x56418fcf9150 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56418fc4b240 con 0x56418fce9a60 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b0ff9640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f659c0a33b0 0x7f659c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50138/0 (socket says 172.21.15.28:50138) 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 -- 172.21.15.28:0/2067885170 learned_addr learned my addr 172.21.15.28:0/2067885170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b3fff640 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fce9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcfb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b3fff640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f659c0a33b0 0x7f659c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b0ff9640 1 -- 172.21.15.28:0/926200000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f65a40cfc60 con 0x7f65b4102d80 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 -- 172.21.15.28:0/2067885170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 msgr2=0x56418fce9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fce9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 -- 172.21.15.28:0/2067885170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 msgr2=0x56418fcfb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcfb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b37fe640 1 -- 172.21.15.28:0/2067885170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56418fc031f0 con 0x56418fcea380 2024-09-15T22:06:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.108+0000 7f65b3fff640 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fce9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.109+0000 7f65b37fe640 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf8c10 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f65a40ccee0 tx=0x7f65a40048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35546ebe9f143891 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.109+0000 7f65b3fff640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f659c0a33b0 0x7f659c0a5870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f65ac005cc0 tx=0x7f65ac005c30 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.109+0000 7f6599ffb640 1 -- 172.21.15.28:0/2067885170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4020070 con 0x56418fcea380 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.109+0000 7f6599ffb640 1 -- 172.21.15.28:0/2067885170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a409bb30 con 0x56418fcea380 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.109+0000 7f6599ffb640 1 -- 172.21.15.28:0/2067885170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40981a0 con 0x56418fcea380 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2067885170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 msgr2=0x56418fcf8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf8c10 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f65a40ccee0 tx=0x7f65a40048a0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2067885170 shutdown_connections 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcfb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf8c10 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2067885170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fce9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2067885170 >> 172.21.15.28:0/2067885170 conn(0x56418fce60f0 msgr2=0x56418fce64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:45.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2067885170 shutdown_connections 2024-09-15T22:06:45.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2067885170 wait complete. 2024-09-15T22:06:45.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 -- start start 2024-09-15T22:06:45.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.110+0000 7f65bc2e0740 1 Processor -- start 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fcf6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcf6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56418fcfbd20 con 0x56418fcea380 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56418fc09f10 con 0x56418fcf9150 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56418fcfbea0 con 0x56418fce9a60 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65bc2e0740 10 client.0 _ll_register_callbacks cb 0x56418fbd2a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50144/0 (socket says 172.21.15.28:50144) 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fcf6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 -- 172.21.15.28:0/3878572857 learned_addr learned my addr 172.21.15.28:0/3878572857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:45.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b3fff640 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcf6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 -- 172.21.15.28:0/3878572857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 msgr2=0x56418fcf6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fcf6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 -- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 msgr2=0x56418fcf6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcf6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 -- 172.21.15.28:0/3878572857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56418f9c06a0 con 0x56418fcea380 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf65c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009900 tx=0x7f65a80317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265f9aac83c9a606 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.111+0000 7f6592ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f6592ffd640 1 -- 172.21.15.28:0/3878572857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8040070 con 0x56418fcea380 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65a409bb30 con 0x56418fcea380 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f6592ffd640 1 -- 172.21.15.28:0/3878572857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f65a8004250 con 0x56418fcea380 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f6592ffd640 1 -- 172.21.15.28:0/3878572857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f65a804d050 con 0x56418fcea380 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 10 client.17499 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56418fd079a0 con 0x56418fcea380 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f6592ffd640 1 -- 172.21.15.28:0/3878572857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a409bb30 con 0x56418fcea380 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f6592ffd640 1 -- 172.21.15.28:0/3878572857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f65a80049c0 con 0x56418fcea380 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 10 client.17499 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 10 client.17499 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 20 client.17499 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd08eb0 0x56418fd0b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 4 client.17499 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.112+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56418fc5c7f0 con 0x56418fd08eb0 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.113+0000 7f65b37fe640 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd08eb0 0x56418fd0b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.113+0000 7f65b37fe640 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd08eb0 0x56418fd0b2d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66febed6217316b server_cookie=16bef6aa4b6ce5c5 in_seq=0 out_seq=0 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.113+0000 7f6592ffd640 10 client.17499 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f6592ffd640 1 -- 172.21.15.28:0/3878572857 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56418fc5c7f0 con 0x56418fd08eb0 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f6592ffd640 10 client.17499 handle_command_reply: tid=0 2024-09-15T22:06:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 client.17499 shutdown 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 msgr2=0x56418fcf65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf65c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009900 tx=0x7f65a80317e0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 shutdown_connections 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd08eb0 0x56418fd0b2d0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fcf6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fcf65c0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f6592ffd640 0 client.17499 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/3878572857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fce9a60 0x56418fcf6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 >> 172.21.15.28:0/3878572857 conn(0x56418fce60f0 msgr2=0x56418fcebdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 shutdown_connections 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 1 -- 172.21.15.28:0/3878572857 wait complete. 2024-09-15T22:06:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.114+0000 7f65bc2e0740 20 client.17499 trim_cache size 0 max 16384 2024-09-15T22:06:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.131+0000 7f65bc2e0740 1 Processor -- start 2024-09-15T22:06:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.131+0000 7f65bc2e0740 1 -- start start 2024-09-15T22:06:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fce9a60 0x56418fcf6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fd10ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcf9150 0x56418fd110e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65b41acf10 con 0x56418fcea380 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65b41aea50 con 0x56418fce9a60 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65b41ad5e0 con 0x56418fcf9150 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fd10ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fd10ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50150/0 (socket says 172.21.15.28:50150) 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 -- 172.21.15.28:0/2058872017 learned_addr learned my addr 172.21.15.28:0/2058872017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b37fe640 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fce9a60 0x56418fcf6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b3fff640 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcf9150 0x56418fd110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 -- 172.21.15.28:0/2058872017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcf9150 msgr2=0x56418fd110e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcf9150 0x56418fd110e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 -- 172.21.15.28:0/2058872017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fce9a60 msgr2=0x56418fcf6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fce9a60 0x56418fcf6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 -- 172.21.15.28:0/2058872017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56418fce5b90 con 0x56418fcea380 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b3fff640 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcf9150 0x56418fd110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b37fe640 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fce9a60 0x56418fcf6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fd10ba0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f65a8006010 tx=0x7f65a802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e08e6f65ce90a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.132+0000 7f65997fa640 1 -- 172.21.15.28:0/2058872017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8040070 con 0x56418fcea380 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65997fa640 1 -- 172.21.15.28:0/2058872017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a8009520 con 0x56418fcea380 2024-09-15T22:06:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65997fa640 1 -- 172.21.15.28:0/2058872017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8035040 con 0x56418fcea380 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2058872017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 msgr2=0x56418fd10ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fd10ba0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f65a8006010 tx=0x7f65a802a480 comp rx=0 tx=0).stop 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2058872017 shutdown_connections 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcf9150 0x56418fd110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fcea380 0x56418fd10ba0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/2058872017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fce9a60 0x56418fcf6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2058872017 >> 172.21.15.28:0/2058872017 conn(0x56418fce60f0 msgr2=0x56418fcf7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2058872017 shutdown_connections 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 -- 172.21.15.28:0/2058872017 wait complete. 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.133+0000 7f65bc2e0740 1 -- start start 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 1 Processor -- start 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fce9a60 0x56418fcfd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcea380 0x56418fcfdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fd72b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65b41ae380 con 0x56418fce9a60 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65b41adcb0 con 0x56418fcf9150 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65b41aaf10 con 0x56418fcea380 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65bc2e0740 10 client.0 _ll_register_callbacks cb 0x7f65a8005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fce9a60 0x56418fcfd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fce9a60 0x56418fcfd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50154/0 (socket says 172.21.15.28:50154) 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b37fe640 1 -- 172.21.15.28:0/1018735483 learned_addr learned my addr 172.21.15.28:0/1018735483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b3fff640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fd72b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcea380 0x56418fcfdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b37fe640 1 -- 172.21.15.28:0/1018735483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcea380 msgr2=0x56418fcfdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b37fe640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcea380 0x56418fcfdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b37fe640 1 -- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 msgr2=0x56418fd72b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.134+0000 7f65b37fe640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fd72b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f65b37fe640 1 -- 172.21.15.28:0/1018735483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56418fd730e0 con 0x56418fce9a60 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcea380 0x56418fcfdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f65b37fe640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fce9a60 0x56418fcfd5e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f65a409bac0 tx=0x7f65a4009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e72f39d9fe4a68f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f6592ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f6592ffd640 1 -- 172.21.15.28:0/1018735483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a409ce60 con 0x56418fce9a60 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f6592ffd640 1 -- 172.21.15.28:0/1018735483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f65a40981a0 con 0x56418fce9a60 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65a8009520 con 0x56418fce9a60 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f6592ffd640 1 -- 172.21.15.28:0/1018735483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f65a4020040 con 0x56418fce9a60 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f65bc2e0740 10 client.17505 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56418fd74370 con 0x56418fce9a60 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.135+0000 7f6592ffd640 1 -- 172.21.15.28:0/1018735483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65a8009520 con 0x56418fce9a60 2024-09-15T22:06:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f6592ffd640 1 -- 172.21.15.28:0/1018735483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f65a40ba070 con 0x56418fce9a60 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65bc2e0740 10 client.17505 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65bc2e0740 10 client.17505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65bc2e0740 20 client.17505 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd74660 0x56418fd76a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65bc2e0740 4 client.17505 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56418fcf1ca0 con 0x56418fd74660 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd74660 0x56418fd76a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f65b2ffd640 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd74660 0x56418fd76a50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce2cf9799cacad98 server_cookie=5b06d7ac50590155 in_seq=0 out_seq=0 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.136+0000 7f6592ffd640 10 client.17505 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.137+0000 7f6592ffd640 1 -- 172.21.15.28:0/1018735483 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56418fcf1ca0 con 0x56418fd74660 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.137+0000 7f6592ffd640 10 client.17505 handle_command_reply: tid=0 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.137+0000 7f65bc2e0740 1 client.17505 shutdown 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.137+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fce9a60 msgr2=0x56418fcfd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.137+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fce9a60 0x56418fcfd5e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f65a409bac0 tx=0x7f65a4009740 comp rx=0 tx=0).stop 2024-09-15T22:06:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 shutdown_connections 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x56418fd74660 0x56418fd76a50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56418fcf9150 0x56418fd72b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56418fcea380 0x56418fcfdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 --2- 172.21.15.28:0/1018735483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56418fce9a60 0x56418fcfd5e0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 >> 172.21.15.28:0/1018735483 conn(0x56418fce60f0 msgr2=0x56418fcf7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f6592ffd640 0 client.17505 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 shutdown_connections 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 1 -- 172.21.15.28:0/1018735483 wait complete. 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65bc2e0740 20 client.17505 trim_cache size 0 max 16384 2024-09-15T22:06:45.144 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:45.145 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-15T22:06:45.145 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "a6573f87-e209-4583-85c8-44154680be94", 2024-09-15T22:06:45.145 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-15T22:06:45.145 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f659c0a33b0 msgr2=0x7f659c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f659c0a33b0 0x7f659c0a5870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f65ac005cc0 tx=0x7f65ac005c30 comp rx=0 tx=0).stop 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b4102d80 msgr2=0x7f65b4198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b4102d80 0x7f65b4198480 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f65a400d770 tx=0x7f65a400dc30 comp rx=0 tx=0).stop 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 shutdown_connections 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f659c0a33b0 0x7f659c0a5870 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65b410c160 0x7f65b419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65b41036a0 0x7f65b41989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 --2- 172.21.15.28:0/926200000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b4102d80 0x7f65b4198480 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.138+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 >> 172.21.15.28:0/926200000 conn(0x7f65b40fc780 msgr2=0x7f65b40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.139+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 shutdown_connections 2024-09-15T22:06:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:45.139+0000 7f65b9f07640 1 -- 172.21.15.28:0/926200000 wait complete. 2024-09-15T22:06:45.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:06:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:45 smithi028 ceph-mon[30969]: pgmap v289: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-09-15T22:06:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:45 smithi028 ceph-mon[30969]: scrub queued for path: / 2024-09-15T22:06:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:45 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:46.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:45 smithi028 ceph-mon[30969]: scrub summary: active paths [/] 2024-09-15T22:06:46.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:45 smithi028 ceph-mon[30969]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:06:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:45 smithi203 ceph-mon[35546]: pgmap v289: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-09-15T22:06:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:45 smithi203 ceph-mon[35546]: scrub queued for path: / 2024-09-15T22:06:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:45 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:45 smithi203 ceph-mon[35546]: scrub summary: active paths [/] 2024-09-15T22:06:46.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:45 smithi203 ceph-mon[35546]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:45 smithi174 ceph-mon[35244]: pgmap v289: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-09-15T22:06:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:45 smithi174 ceph-mon[35244]: scrub queued for path: / 2024-09-15T22:06:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:45 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:45 smithi174 ceph-mon[35244]: scrub summary: active paths [/] 2024-09-15T22:06:46.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:45 smithi174 ceph-mon[35244]: scrub summary: idle+waiting paths [/] 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 -- 172.21.15.28:0/3075669101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 msgr2=0x7f526410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3075669101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526410ec40 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f525c009cd0 tx=0x7f525c02a8e0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 -- 172.21.15.28:0/3075669101 shutdown_connections 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3075669101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526410ec40 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3075669101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52641036e0 0x7f5264107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3075669101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5264102d30 0x7f5264103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 -- 172.21.15.28:0/3075669101 >> 172.21.15.28:0/3075669101 conn(0x7f52640fc7d0 msgr2=0x7f52640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:46.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 -- 172.21.15.28:0/3075669101 shutdown_connections 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.628+0000 7f526c1d9640 1 -- 172.21.15.28:0/3075669101 wait complete. 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 Processor -- start 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 -- start start 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5264102d30 0x7f5264198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52641036e0 0x7f5264198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52641120f0 con 0x7f52641082a0 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5264111f70 con 0x7f52641036e0 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526c1d9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5264112270 con 0x7f5264102d30 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526a776640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526a776640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526419cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50176/0 (socket says 172.21.15.28:50176) 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f526a776640 1 -- 172.21.15.28:0/3456298226 learned_addr learned my addr 172.21.15.28:0/3456298226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.629+0000 7f5269f75640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5264102d30 0x7f5264198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526a776640 1 -- 172.21.15.28:0/3456298226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5264102d30 msgr2=0x7f5264198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526a776640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5264102d30 0x7f5264198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526a776640 1 -- 172.21.15.28:0/3456298226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52641036e0 msgr2=0x7f5264198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526a776640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52641036e0 0x7f5264198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526a776640 1 -- 172.21.15.28:0/3456298226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526419d5e0 con 0x7f52641082a0 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526a776640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526419cf00 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f525c033040 tx=0x7f525c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f525affd640 1 -- 172.21.15.28:0/3456298226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c004040 con 0x7f52641082a0 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f525affd640 1 -- 172.21.15.28:0/3456298226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f525c0046c0 con 0x7f52641082a0 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526419d870 con 0x7f52641082a0 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f525affd640 1 -- 172.21.15.28:0/3456298226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c046480 con 0x7f52641082a0 2024-09-15T22:06:46.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.630+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52641a51b0 con 0x7f52641082a0 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.632+0000 7f525affd640 1 -- 172.21.15.28:0/3456298226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f525c0305a0 con 0x7f52641082a0 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 Processor -- start 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 -- start start 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386507880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386512140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386512680 0x559386514a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55938646d2e0 con 0x5593865074a0 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559386414ce0 con 0x559386512680 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f526e5b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5593864692a0 con 0x559386507dc0 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f5269774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386507880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f5269774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386507880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50178/0 (socket says 172.21.15.28:50178) 2024-09-15T22:06:46.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.633+0000 7f5269774640 1 -- 172.21.15.28:0/1179674381 learned_addr learned my addr 172.21.15.28:0/1179674381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f525affd640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52500a33b0 0x7f52500a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269f75640 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386512140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269774640 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386512680 0x559386514a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f526a776640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52500a33b0 0x7f52500a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269774640 1 -- 172.21.15.28:0/1179674381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 msgr2=0x559386512140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269774640 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386512140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269774640 1 -- 172.21.15.28:0/1179674381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386512680 msgr2=0x559386514a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269774640 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386512680 0x559386514a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269774640 1 -- 172.21.15.28:0/1179674381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593863f9440 con 0x5593865074a0 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f525affd640 1 -- 172.21.15.28:0/3456298226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f525c02e030 con 0x7f52641082a0 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f5269774640 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386507880 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f5254014730 tx=0x7f525400d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4732382747dada server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f526a776640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52500a33b0 0x7f52500a5870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f525c0b9a70 tx=0x7f525c03d040 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f524ffff640 1 -- 172.21.15.28:0/1179674381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254005220 con 0x5593865074a0 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f524ffff640 1 -- 172.21.15.28:0/1179674381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5254004c80 con 0x5593865074a0 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.634+0000 7f524ffff640 1 -- 172.21.15.28:0/1179674381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540079e0 con 0x5593865074a0 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 -- 172.21.15.28:0/1179674381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 msgr2=0x559386507880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386507880 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f5254014730 tx=0x7f525400d2e0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 -- 172.21.15.28:0/1179674381 shutdown_connections 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386512680 0x559386514a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386512140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 --2- 172.21.15.28:0/1179674381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386507880 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 -- 172.21.15.28:0/1179674381 >> 172.21.15.28:0/1179674381 conn(0x559386503a30 msgr2=0x559386503e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 -- 172.21.15.28:0/1179674381 shutdown_connections 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 -- 172.21.15.28:0/1179674381 wait complete. 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.635+0000 7f526e5b2740 1 -- start start 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 1 Processor -- start 2024-09-15T22:06:46.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386511fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386512680 0x55938650f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559386515250 con 0x559386512680 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559386428050 con 0x559386507dc0 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5593865153d0 con 0x5593865074a0 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f526e5b2740 10 client.0 _ll_register_callbacks cb 0x559386468160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386512680 0x55938650f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386512680 0x55938650f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50190/0 (socket says 172.21.15.28:50190) 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 -- 172.21.15.28:0/2544076990 learned_addr learned my addr 172.21.15.28:0/2544076990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269f75640 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386511fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 -- 172.21.15.28:0/2544076990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 msgr2=0x559386511fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386511fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 -- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 msgr2=0x55938650f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.636+0000 7f5269774640 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f5269774640 1 -- 172.21.15.28:0/2544076990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593861de3c0 con 0x559386512680 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f5269774640 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386512680 0x55938650f7b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f525400dfd0 tx=0x7f52540075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673b5c77377b5188 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f524d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f524d7fa640 1 -- 172.21.15.28:0/2544076990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254005460 con 0x559386512680 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f524d7fa640 1 -- 172.21.15.28:0/2544076990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f5254007760 con 0x559386512680 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5254004c80 con 0x559386512680 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f524d7fa640 1 -- 172.21.15.28:0/2544076990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5254005600 con 0x559386512680 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f526e5b2740 10 client.17517 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559386520c00 con 0x559386512680 2024-09-15T22:06:46.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.637+0000 7f524d7fa640 1 -- 172.21.15.28:0/2544076990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5254004c80 con 0x559386512680 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f524d7fa640 1 -- 172.21.15.28:0/2544076990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f525400c030 con 0x559386512680 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f526e5b2740 10 client.17517 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f526e5b2740 10 client.17517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f526e5b2740 20 client.17517 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386521f50 0x559386524340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f526e5b2740 4 client.17517 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55938647a850 con 0x559386521f50 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f5269f75640 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386521f50 0x559386524340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f524d7fa640 10 client.17517 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.638+0000 7f5269f75640 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386521f50 0x559386524340 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e16c7983ab9d0a2 server_cookie=f08cc1c98fdde9a0 in_seq=0 out_seq=0 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f524d7fa640 1 -- 172.21.15.28:0/2544076990 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55938647a850 con 0x559386521f50 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f524d7fa640 10 client.17517 handle_command_reply: tid=0 2024-09-15T22:06:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 client.17517 shutdown 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386512680 msgr2=0x55938650f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386512680 0x55938650f7b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f525400dfd0 tx=0x7f52540075c0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 shutdown_connections 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386521f50 0x559386524340 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386512680 0x55938650f7b0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2544076990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386511fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f524d7fa640 0 client.17517 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 >> 172.21.15.28:0/2544076990 conn(0x559386503a30 msgr2=0x559386509760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 shutdown_connections 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.639+0000 7f526e5b2740 1 -- 172.21.15.28:0/2544076990 wait complete. 2024-09-15T22:06:46.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.640+0000 7f526e5b2740 20 client.17517 trim_cache size 0 max 16384 2024-09-15T22:06:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.657+0000 7f526e5b2740 1 Processor -- start 2024-09-15T22:06:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.657+0000 7f526e5b2740 1 -- start start 2024-09-15T22:06:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.657+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386503a30 0x55938650a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.657+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386502a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.657+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f526e5b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52641aefe0 con 0x559386503a30 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f526e5b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52641b0b20 con 0x559386507dc0 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f526e5b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52641af6b0 con 0x5593865074a0 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386503a30 0x55938650a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386503a30 0x55938650a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50200/0 (socket says 172.21.15.28:50200) 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 -- 172.21.15.28:0/4006618767 learned_addr learned my addr 172.21.15.28:0/4006618767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269f75640 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386502a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 -- 172.21.15.28:0/4006618767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 msgr2=0x559386502a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386502a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 -- 172.21.15.28:0/4006618767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 msgr2=0x55938650c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 -- 172.21.15.28:0/4006618767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559386512330 con 0x559386503a30 2024-09-15T22:06:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269f75640 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386502a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:06:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f5269774640 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386503a30 0x55938650a0e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5254005c40 tx=0x7f5254005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e93395feceb793c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f524f7fe640 1 -- 172.21.15.28:0/4006618767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254005600 con 0x559386503a30 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f524f7fe640 1 -- 172.21.15.28:0/4006618767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f525400d8e0 con 0x559386503a30 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.658+0000 7f524f7fe640 1 -- 172.21.15.28:0/4006618767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254007660 con 0x559386503a30 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 -- 172.21.15.28:0/4006618767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386503a30 msgr2=0x55938650a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386503a30 0x55938650a0e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f5254005c40 tx=0x7f5254005b90 comp rx=0 tx=0).stop 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 -- 172.21.15.28:0/4006618767 shutdown_connections 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386507dc0 0x55938650c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5593865074a0 0x559386502a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 --2- 172.21.15.28:0/4006618767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559386503a30 0x55938650a0e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 -- 172.21.15.28:0/4006618767 >> 172.21.15.28:0/4006618767 conn(0x55938650f910 msgr2=0x559386502370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 -- 172.21.15.28:0/4006618767 shutdown_connections 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.659+0000 7f526e5b2740 1 -- 172.21.15.28:0/4006618767 wait complete. 2024-09-15T22:06:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 -- start start 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 Processor -- start 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386503a30 0x5593865251d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386525710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386525c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52641b0450 con 0x5593865074a0 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52641afd80 con 0x559386503a30 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52641acfe0 con 0x559386507dc0 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f526e5b2740 10 client.0 _ll_register_callbacks cb 0x7f525400dc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f5269f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386525710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f5269f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386525710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50214/0 (socket says 172.21.15.28:50214) 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f5269f75640 1 -- 172.21.15.28:0/2233475810 learned_addr learned my addr 172.21.15.28:0/2233475810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:06:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.660+0000 7f5269774640 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386503a30 0x5593865251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f5269774640 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386525c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f5269f75640 1 -- 172.21.15.28:0/2233475810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 msgr2=0x559386525c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f5269f75640 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386525c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f5269f75640 1 -- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386503a30 msgr2=0x5593865251d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f5269f75640 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386503a30 0x5593865251d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f5269f75640 1 -- 172.21.15.28:0/2233475810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55938658b990 con 0x5593865074a0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f5269f75640 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386525710 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f526000bd50 tx=0x7f52600118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5637dc0f81c63f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f524d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f524d7fa640 1 -- 172.21.15.28:0/2233475810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526001d910 con 0x5593865074a0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525400d8e0 con 0x5593865074a0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f524d7fa640 1 -- 172.21.15.28:0/2233475810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9467+0+0 (secure 0 0 0) 0x7f526001dab0 con 0x5593865074a0 2024-09-15T22:06:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.661+0000 7f524d7fa640 1 -- 172.21.15.28:0/2233475810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f526002f050 con 0x5593865074a0 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 10 client.17523 fetch_fsmap learned FSMap version 35 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559386591100 con 0x5593865074a0 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f524d7fa640 1 -- 172.21.15.28:0/2233475810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f525400d8e0 con 0x5593865074a0 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f524d7fa640 1 -- 172.21.15.28:0/2233475810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8374+0+0 (secure 0 0 0) 0x7f5260024380 con 0x5593865074a0 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 10 client.17523 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 10 client.17523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 20 client.17523 populate_metadata read hostname 'smithi028' 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386512680 0x5593865938a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 4 client.17523 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559386474d60 con 0x559386512680 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.662+0000 7f5269774640 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386512680 0x5593865938a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f5269774640 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386512680 0x5593865938a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83dafc2c2c59bb83 server_cookie=da431bdeeeac9bd5 in_seq=0 out_seq=0 2024-09-15T22:06:46.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f524d7fa640 10 client.17523 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f524d7fa640 1 -- 172.21.15.28:0/2233475810 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559386474d60 con 0x559386512680 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f524d7fa640 10 client.17523 handle_command_reply: tid=0 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 client.17523 shutdown 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 msgr2=0x559386525710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386525710 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f526000bd50 tx=0x7f52600118d0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 shutdown_connections 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x559386512680 0x5593865938a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559386507dc0 0x559386525c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5593865074a0 0x559386525710 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f524d7fa640 0 client.17523 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 --2- 172.21.15.28:0/2233475810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559386503a30 0x5593865251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 >> 172.21.15.28:0/2233475810 conn(0x55938650f910 msgr2=0x559386502b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:46.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 shutdown_connections 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.663+0000 7f526e5b2740 1 -- 172.21.15.28:0/2233475810 wait complete. 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526e5b2740 20 client.17523 trim_cache size 0 max 16384 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "a6573f87-e209-4583-85c8-44154680be94": { 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "a6573f87-e209-4583-85c8-44154680be94", 2024-09-15T22:06:46.677 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-09-15T22:06:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:06:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:06:46.678 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:06:46.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52500a33b0 msgr2=0x7f52500a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52500a33b0 0x7f52500a5870 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f525c0b9a70 tx=0x7f525c03d040 comp rx=0 tx=0).stop 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 msgr2=0x7f526419cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526419cf00 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f525c033040 tx=0x7f525c003970 comp rx=0 tx=0).stop 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 shutdown_connections 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f52500a33b0 0x7f52500a5870 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52641082a0 0x7f526419cf00 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52641036e0 0x7f5264198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 --2- 172.21.15.28:0/3456298226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5264102d30 0x7f5264198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 >> 172.21.15.28:0/3456298226 conn(0x7f52640fc7d0 msgr2=0x7f52640fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 shutdown_connections 2024-09-15T22:06:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:06:46.664+0000 7f526c1d9640 1 -- 172.21.15.28:0/3456298226 wait complete. 2024-09-15T22:06:47.368 INFO:tasks.cephfs.filesystem:scrub status for tag:a6573f87-e209-4583-85c8-44154680be94 - {'path': '/', 'tag': 'a6573f87-e209-4583-85c8-44154680be94', 'options': 'recursive,force'} 2024-09-15T22:06:48.220 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:47 smithi203 ceph-mon[35546]: pgmap v290: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-09-15T22:06:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:47 smithi174 ceph-mon[35244]: pgmap v290: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-09-15T22:06:48.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:47 smithi028 ceph-mon[30969]: pgmap v290: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-09-15T22:06:49.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:48 smithi203 ceph-mon[35546]: scrub summary: idle 2024-09-15T22:06:49.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:48 smithi174 ceph-mon[35244]: scrub summary: idle 2024-09-15T22:06:49.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:48 smithi028 ceph-mon[30969]: scrub summary: idle 2024-09-15T22:06:50.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:49 smithi203 ceph-mon[35546]: pgmap v291: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 op/s 2024-09-15T22:06:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:49 smithi174 ceph-mon[35244]: pgmap v291: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 op/s 2024-09-15T22:06:50.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:49 smithi028 ceph-mon[30969]: pgmap v291: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 op/s 2024-09-15T22:06:52.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:51 smithi203 ceph-mon[35546]: pgmap v292: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:06:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:51 smithi174 ceph-mon[35244]: pgmap v292: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:06:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:51 smithi028 ceph-mon[30969]: pgmap v292: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-15T22:06:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:53 smithi203 ceph-mon[35546]: pgmap v293: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 op/s 2024-09-15T22:06:54.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:53 smithi174 ceph-mon[35244]: pgmap v293: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 op/s 2024-09-15T22:06:54.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:53 smithi028 ceph-mon[30969]: pgmap v293: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 op/s 2024-09-15T22:06:56.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:55 smithi203 ceph-mon[35546]: pgmap v294: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-15T22:06:56.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:55 smithi174 ceph-mon[35244]: pgmap v294: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-15T22:06:56.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:55 smithi028 ceph-mon[30969]: pgmap v294: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-15T22:06:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:06:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:06:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:06:57.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:56 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:06:57.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:06:57.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:06:57.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:06:57.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:56 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:06:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:06:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:06:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:06:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:56 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:06:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:57 smithi174 ceph-mon[35244]: pgmap v295: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-15T22:06:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:57 smithi174 ceph-mon[35244]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:06:58.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:57 smithi028 ceph-mon[30969]: pgmap v295: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-15T22:06:58.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:57 smithi028 ceph-mon[30969]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:06:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:57 smithi203 ceph-mon[35546]: pgmap v295: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-09-15T22:06:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:57 smithi203 ceph-mon[35546]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T22:07:00.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:06:59 smithi028 ceph-mon[30969]: pgmap v296: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 op/s 2024-09-15T22:07:00.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:06:59 smithi174 ceph-mon[35244]: pgmap v296: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 op/s 2024-09-15T22:07:00.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:06:59 smithi203 ceph-mon[35546]: pgmap v296: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 7.4 KiB/s rd, 12 op/s 2024-09-15T22:07:02.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:01 smithi174 ceph-mon[35244]: pgmap v297: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:02.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:01 smithi028 ceph-mon[30969]: pgmap v297: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:02.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:01 smithi203 ceph-mon[35546]: pgmap v297: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:04.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:03 smithi174 ceph-mon[35244]: pgmap v298: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:07:04.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:03 smithi028 ceph-mon[30969]: pgmap v298: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:07:04.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:03 smithi203 ceph-mon[35546]: pgmap v298: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:07:06.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:05 smithi174 ceph-mon[35244]: pgmap v299: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:07:06.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:05 smithi028 ceph-mon[30969]: pgmap v299: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:07:06.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:05 smithi203 ceph-mon[35546]: pgmap v299: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:07:08.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:07 smithi174 ceph-mon[35244]: pgmap v300: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:07:08.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:07 smithi028 ceph-mon[30969]: pgmap v300: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:07:08.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:07 smithi203 ceph-mon[35546]: pgmap v300: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-09-15T22:07:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:09 smithi174 ceph-mon[35244]: pgmap v301: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:07:10.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:09 smithi174 ceph-mon[35244]: osdmap e98: 12 total, 12 up, 12 in 2024-09-15T22:07:10.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:09 smithi028 ceph-mon[30969]: pgmap v301: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:07:10.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:09 smithi028 ceph-mon[30969]: osdmap e98: 12 total, 12 up, 12 in 2024-09-15T22:07:10.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:09 smithi203 ceph-mon[35546]: pgmap v301: 129 pgs: 129 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:07:10.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:09 smithi203 ceph-mon[35546]: osdmap e98: 12 total, 12 up, 12 in 2024-09-15T22:07:12.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:11 smithi174 ceph-mon[35244]: pgmap v303: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-15T22:07:12.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:11 smithi028 ceph-mon[30969]: pgmap v303: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-15T22:07:12.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:11 smithi203 ceph-mon[35546]: pgmap v303: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 19 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-09-15T22:07:14.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:13 smithi174 ceph-mon[35244]: pgmap v304: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:14.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:13 smithi028 ceph-mon[30969]: pgmap v304: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:14.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:13 smithi203 ceph-mon[35546]: pgmap v304: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 18 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:16.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:16 smithi174 ceph-mon[35244]: pgmap v305: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 17 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:16.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:16 smithi028 ceph-mon[30969]: pgmap v305: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 17 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:16.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:16 smithi203 ceph-mon[35546]: pgmap v305: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 17 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:17.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T22:07:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:18.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:18 smithi174 ceph-mon[35244]: pgmap v306: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 17 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:18.270 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:18 smithi203 ceph-mon[35546]: pgmap v306: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 17 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:18.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:18 smithi028 ceph-mon[30969]: pgmap v306: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 17 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:20 smithi174 ceph-mon[35244]: pgmap v307: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 16 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:20.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:20 smithi028 ceph-mon[30969]: pgmap v307: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 16 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:20.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:20 smithi203 ceph-mon[35546]: pgmap v307: 129 pgs: 4 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 119 active+clean; 16 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-09-15T22:07:22.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:22 smithi028 ceph-mon[30969]: pgmap v308: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 179 B/s wr, 9 op/s 2024-09-15T22:07:22.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:22 smithi203 ceph-mon[35546]: pgmap v308: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 179 B/s wr, 9 op/s 2024-09-15T22:07:22.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:22 smithi174 ceph-mon[35244]: pgmap v308: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 179 B/s wr, 9 op/s 2024-09-15T22:07:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 -- 172.21.15.28:0/2043799539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 msgr2=0x7ff8641063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 --2- 172.21.15.28:0/2043799539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff8641063d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009ca0 tx=0x7ff85c02a940 comp rx=0 tx=0).stop 2024-09-15T22:07:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 -- 172.21.15.28:0/2043799539 shutdown_connections 2024-09-15T22:07:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 --2- 172.21.15.28:0/2043799539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff8641063d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 --2- 172.21.15.28:0/2043799539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff864101450 0x7ff864105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 --2- 172.21.15.28:0/2043799539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff864100b30 0x7ff864100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 -- 172.21.15.28:0/2043799539 >> 172.21.15.28:0/2043799539 conn(0x7ff8640fc7a0 msgr2=0x7ff8640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 -- 172.21.15.28:0/2043799539 shutdown_connections 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.506+0000 7ff869f7f640 1 -- 172.21.15.28:0/2043799539 wait complete. 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 Processor -- start 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 -- start start 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff864100b30 0x7ff8641a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff864101450 0x7ff8641a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff86419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff864118560 con 0x7ff864105f40 2024-09-15T22:07:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641183e0 con 0x7ff864101450 2024-09-15T22:07:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff869f7f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641186e0 con 0x7ff864100b30 2024-09-15T22:07:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff86419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.507+0000 7ff863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff86419e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46534/0 (socket says 172.21.15.28:46534) 2024-09-15T22:07:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff863fff640 1 -- 172.21.15.28:0/1791163604 learned_addr learned my addr 172.21.15.28:0/1791163604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff8637fe640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff864100b30 0x7ff8641a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff85affd640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff864101450 0x7ff8641a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff863fff640 1 -- 172.21.15.28:0/1791163604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff864100b30 msgr2=0x7ff8641a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff863fff640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff864100b30 0x7ff8641a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff863fff640 1 -- 172.21.15.28:0/1791163604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff864101450 msgr2=0x7ff8641a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff863fff640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff864101450 0x7ff8641a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff863fff640 1 -- 172.21.15.28:0/1791163604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86419efd0 con 0x7ff864105f40 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff85affd640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff864101450 0x7ff8641a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff863fff640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff86419e830 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005e30 tx=0x7ff85c02f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff8617fa640 1 -- 172.21.15.28:0/1791163604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c034040 con 0x7ff864105f40 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff8617fa640 1 -- 172.21.15.28:0/1791163604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff85c003e50 con 0x7ff864105f40 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff86419f260 con 0x7ff864105f40 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff8617fa640 1 -- 172.21.15.28:0/1791163604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c004680 con 0x7ff864105f40 2024-09-15T22:07:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.508+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8641ab510 con 0x7ff864105f40 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff8617fa640 1 -- 172.21.15.28:0/1791163604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7ff85c037020 con 0x7ff864105f40 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 Processor -- start 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 -- start start 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x563843769880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843774140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843776a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638436cf2e0 con 0x5638437694a0 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563843676ce0 con 0x563843774680 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff86c358740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5638436cb2a0 con 0x563843769dc0 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff85affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x563843769880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff85affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x563843769880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46544/0 (socket says 172.21.15.28:46544) 2024-09-15T22:07:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff85affd640 1 -- 172.21.15.28:0/4017012597 learned_addr learned my addr 172.21.15.28:0/4017012597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff85affd640 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843776a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff8637fe640 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843774140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff85affd640 1 -- 172.21.15.28:0/4017012597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 msgr2=0x563843774140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff85affd640 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843774140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.512+0000 7ff85affd640 1 -- 172.21.15.28:0/4017012597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 msgr2=0x563843776a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff85affd640 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843776a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff85affd640 1 -- 172.21.15.28:0/4017012597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384365b440 con 0x5638437694a0 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff8637fe640 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843774140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff8617fa640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff8500a33b0 0x7ff8500a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff85affd640 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x563843769880 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff84c013520 tx=0x7ff84c0096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6dbfa4f6ce35da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff8637fe640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff8500a33b0 0x7ff8500a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff8617fa640 1 -- 172.21.15.28:0/1791163604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9555+0+0 (secure 0 0 0) 0x7ff85c0f1000 con 0x7ff864105f40 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff859ffb640 1 -- 172.21.15.28:0/4017012597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c011470 con 0x5638437694a0 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff859ffb640 1 -- 172.21.15.28:0/4017012597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84c011af0 con 0x5638437694a0 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.513+0000 7ff859ffb640 1 -- 172.21.15.28:0/4017012597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c00f9f0 con 0x5638437694a0 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 -- 172.21.15.28:0/4017012597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 msgr2=0x563843769880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x563843769880 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff84c013520 tx=0x7ff84c0096d0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff8637fe640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff8500a33b0 0x7ff8500a5870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x563843766890 tx=0x7ff85400b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 -- 172.21.15.28:0/4017012597 shutdown_connections 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843776a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843774140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 --2- 172.21.15.28:0/4017012597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x563843769880 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 -- 172.21.15.28:0/4017012597 >> 172.21.15.28:0/4017012597 conn(0x563843765a30 msgr2=0x563843765e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 -- 172.21.15.28:0/4017012597 shutdown_connections 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 -- 172.21.15.28:0/4017012597 wait complete. 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.514+0000 7ff86c358740 1 -- start start 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 1 Processor -- start 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x563843773f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x563843771240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843771780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563843777250 con 0x563843769dc0 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56384368a050 con 0x563843774680 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5638437773d0 con 0x5638437694a0 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff86c358740 10 client.0 _ll_register_callbacks cb 0x5638436ca160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x563843771240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x563843771240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46556/0 (socket says 172.21.15.28:46556) 2024-09-15T22:07:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 -- 172.21.15.28:0/3281370068 learned_addr learned my addr 172.21.15.28:0/3281370068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff863fff640 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843771780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff8637fe640 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x563843773f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 -- 172.21.15.28:0/3281370068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 msgr2=0x563843773f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x563843773f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 -- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 msgr2=0x563843771780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843771780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.515+0000 7ff85affd640 1 -- 172.21.15.28:0/3281370068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638434403c0 con 0x563843769dc0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff85affd640 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x563843771240 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff84c013e70 tx=0x7ff84c013300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257b6795fbae5d1b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff842ffd640 1 -- 172.21.15.28:0/3281370068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c011880 con 0x563843769dc0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff842ffd640 1 -- 172.21.15.28:0/3281370068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9555+0+0 (secure 0 0 0) 0x7ff84c00f790 con 0x563843769dc0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff84c011af0 con 0x563843769dc0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff842ffd640 1 -- 172.21.15.28:0/3281370068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff84c032050 con 0x563843769dc0 2024-09-15T22:07:22.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff86c358740 10 client.17535 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563843782bd0 con 0x563843769dc0 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff842ffd640 1 -- 172.21.15.28:0/3281370068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84c011af0 con 0x563843769dc0 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff842ffd640 1 -- 172.21.15.28:0/3281370068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7ff84c0279b0 con 0x563843769dc0 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.516+0000 7ff86c358740 10 client.17535 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff86c358740 10 client.17535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff86c358740 20 client.17535 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff86c358740 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563843784030 0x563843786420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff86c358740 4 client.17535 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638436dc850 con 0x563843784030 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff8637fe640 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563843784030 0x563843786420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff842ffd640 10 client.17535 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.517+0000 7ff8637fe640 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563843784030 0x563843786420 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58b98c1a18552f1f server_cookie=ae875bd9372357f0 in_seq=0 out_seq=0 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff842ffd640 1 -- 172.21.15.28:0/3281370068 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638436dc850 con 0x563843784030 2024-09-15T22:07:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff842ffd640 10 client.17535 handle_command_reply: tid=0 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 client.17535 shutdown 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 msgr2=0x563843771240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x563843771240 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff84c013e70 tx=0x7ff84c013300 comp rx=0 tx=0).stop 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 shutdown_connections 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x563843784030 0x563843786420 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x563843771780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x563843771240 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 --2- 172.21.15.28:0/3281370068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x563843773f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff842ffd640 0 client.17535 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.518+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 >> 172.21.15.28:0/3281370068 conn(0x563843765a30 msgr2=0x56384376b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.519+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 shutdown_connections 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.519+0000 7ff86c358740 1 -- 172.21.15.28:0/3281370068 wait complete. 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.519+0000 7ff86c358740 20 client.17535 trim_cache size 0 max 16384 2024-09-15T22:07:22.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.536+0000 7ff86c358740 1 Processor -- start 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.536+0000 7ff86c358740 1 -- start start 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.536+0000 7ff86c358740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x5638437804b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.536+0000 7ff86c358740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x56384378bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff86c358740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x56384378c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff86c358740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b56c0 con 0x563843769dc0 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff86c358740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b7200 con 0x563843774680 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff86c358740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b5d90 con 0x5638437694a0 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x56384378bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x56384378bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46560/0 (socket says 172.21.15.28:46560) 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 -- 172.21.15.28:0/2588710204 learned_addr learned my addr 172.21.15.28:0/2588710204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff8637fe640 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x5638437804b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 -- 172.21.15.28:0/2588710204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 msgr2=0x5638437804b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x5638437804b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff863fff640 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x56384378c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 -- 172.21.15.28:0/2588710204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 msgr2=0x56384378c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x56384378c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 -- 172.21.15.28:0/2588710204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563843774300 con 0x563843769dc0 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff85affd640 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x56384378bc00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00f680 tx=0x7ff84c028fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185e8479e91d13ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff8597fa640 1 -- 172.21.15.28:0/2588710204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c03a030 con 0x563843769dc0 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff8597fa640 1 -- 172.21.15.28:0/2588710204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84c00f420 con 0x563843769dc0 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.537+0000 7ff8597fa640 1 -- 172.21.15.28:0/2588710204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c005490 con 0x563843769dc0 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 -- 172.21.15.28:0/2588710204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 msgr2=0x56384378bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x56384378bc00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00f680 tx=0x7ff84c028fa0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 -- 172.21.15.28:0/2588710204 shutdown_connections 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x56384378c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563843769dc0 0x56384378bc00 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 --2- 172.21.15.28:0/2588710204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638437694a0 0x5638437804b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 -- 172.21.15.28:0/2588710204 >> 172.21.15.28:0/2588710204 conn(0x563843765a30 msgr2=0x5638437732c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 -- 172.21.15.28:0/2588710204 shutdown_connections 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 -- 172.21.15.28:0/2588710204 wait complete. 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.538+0000 7ff86c358740 1 -- start start 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 1 Processor -- start 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x5638437796b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843779bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x5638437ecbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b6b30 con 0x5638437694a0 2024-09-15T22:07:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b6460 con 0x563843774680 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b36c0 con 0x563843769dc0 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff86c358740 10 client.0 _ll_register_callbacks cb 0x7ff84c013400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x5638437796b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x5638437796b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46564/0 (socket says 172.21.15.28:46564) 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 -- 172.21.15.28:0/1017376689 learned_addr learned my addr 172.21.15.28:0/1017376689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff85affd640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843779bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff863fff640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x5638437ecbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 -- 172.21.15.28:0/1017376689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 msgr2=0x563843779bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843779bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 -- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 msgr2=0x5638437ecbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x5638437ecbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff8637fe640 1 -- 172.21.15.28:0/1017376689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384377a480 con 0x5638437694a0 2024-09-15T22:07:22.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.539+0000 7ff85affd640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843779bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff8637fe640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x5638437796b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x563843786b70 tx=0x7ff854014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74d6cc6bb0e61cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff842ffd640 1 -- 172.21.15.28:0/1017376689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854005250 con 0x5638437694a0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff842ffd640 1 -- 172.21.15.28:0/1017376689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9555+0+0 (secure 0 0 0) 0x7ff8540053f0 con 0x5638437694a0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff84c00f420 con 0x5638437694a0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff842ffd640 1 -- 172.21.15.28:0/1017376689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff854037050 con 0x5638437694a0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 10 client.17541 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5638437ee250 con 0x5638437694a0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff842ffd640 1 -- 172.21.15.28:0/1017376689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84c00f420 con 0x5638437694a0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff842ffd640 1 -- 172.21.15.28:0/1017376689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7ff854030060 con 0x5638437694a0 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 10 client.17541 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 10 client.17541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:07:22.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 20 client.17541 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5638437ee9d0 0x5638437f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 4 client.17541 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.540+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638436d6d60 con 0x5638437ee9d0 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff85affd640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5638437ee9d0 0x5638437f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff85affd640 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5638437ee9d0 0x5638437f0df0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f1bf53b4c92b743 server_cookie=f15050675a5ffe15 in_seq=0 out_seq=0 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff842ffd640 10 client.17541 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff842ffd640 1 -- 172.21.15.28:0/1017376689 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5638436d6d60 con 0x5638437ee9d0 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff842ffd640 10 client.17541 handle_command_reply: tid=0 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff86c358740 1 client.17541 shutdown 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 msgr2=0x5638437796b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.541+0000 7ff86c358740 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x5638437796b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x563843786b70 tx=0x7ff854014040 comp rx=0 tx=0).stop 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 shutdown_connections 2024-09-15T22:07:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x5638437ee9d0 0x5638437f0df0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563843774680 0x5638437ecbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563843769dc0 0x563843779bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 --2- 172.21.15.28:0/1017376689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638437694a0 0x5638437796b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff842ffd640 0 client.17541 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 >> 172.21.15.28:0/1017376689 conn(0x563843765a30 msgr2=0x56384376f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 shutdown_connections 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 1 -- 172.21.15.28:0/1017376689 wait complete. 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff86c358740 20 client.17541 trim_cache size 0 max 16384 2024-09-15T22:07:22.899 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff8500a33b0 msgr2=0x7ff8500a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff8500a33b0 0x7ff8500a5870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x563843766890 tx=0x7ff85400b390 comp rx=0 tx=0).stop 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 msgr2=0x7ff86419e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff86419e830 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005e30 tx=0x7ff85c02f630 comp rx=0 tx=0).stop 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 shutdown_connections 2024-09-15T22:07:22.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7ff8500a33b0 0x7ff8500a5870 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864105f40 0x7ff86419e830 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff864101450 0x7ff8641a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 --2- 172.21.15.28:0/1791163604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff864100b30 0x7ff8641a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 >> 172.21.15.28:0/1791163604 conn(0x7ff8640fc7a0 msgr2=0x7ff8641038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 shutdown_connections 2024-09-15T22:07:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:21.542+0000 7ff869f7f640 1 -- 172.21.15.28:0/1791163604 wait complete. 2024-09-15T22:07:24.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:24 smithi028 ceph-mon[30969]: pgmap v309: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-15T22:07:24.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:24 smithi203 ceph-mon[35546]: pgmap v309: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-15T22:07:24.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:24 smithi174 ceph-mon[35244]: pgmap v309: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-15T22:07:25.353 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T22:07:25.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:07:25.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 -- 172.21.15.28:0/558587194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63901027a0 msgr2=0x7f63901067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 --2- 172.21.15.28:0/558587194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63901027a0 0x7f63901067f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f6384009900 tx=0x7f638402a410 comp rx=0 tx=0).stop 2024-09-15T22:07:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 -- 172.21.15.28:0/558587194 shutdown_connections 2024-09-15T22:07:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 --2- 172.21.15.28:0/558587194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6390106f20 0x7f639010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 --2- 172.21.15.28:0/558587194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63901027a0 0x7f63901067f0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 --2- 172.21.15.28:0/558587194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6390101df0 0x7f63901021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 -- 172.21.15.28:0/558587194 >> 172.21.15.28:0/558587194 conn(0x7f63900fd8f0 msgr2=0x7f63900ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 -- 172.21.15.28:0/558587194 shutdown_connections 2024-09-15T22:07:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.117+0000 7f6394ead640 1 -- 172.21.15.28:0/558587194 wait complete. 2024-09-15T22:07:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 Processor -- start 2024-09-15T22:07:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 -- start start 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6390101df0 0x7f639019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63901027a0 0x7f639019abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6390106f20 0x7f639019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6390111f80 con 0x7f6390106f20 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6390111e00 con 0x7f63901027a0 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f6394ead640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6390112100 con 0x7f6390101df0 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f638ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6390106f20 0x7f639019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f638ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6390106f20 0x7f639019ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46574/0 (socket says 172.21.15.28:46574) 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f638ed9d640 1 -- 172.21.15.28:0/4119653004 learned_addr learned my addr 172.21.15.28:0/4119653004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f638e59c640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6390101df0 0x7f639019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f638dd9b640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63901027a0 0x7f639019abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f638ed9d640 1 -- 172.21.15.28:0/4119653004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6390101df0 msgr2=0x7f639019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.118+0000 7f638ed9d640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6390101df0 0x7f639019a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f638ed9d640 1 -- 172.21.15.28:0/4119653004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63901027a0 msgr2=0x7f639019abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f638ed9d640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63901027a0 0x7f639019abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f638ed9d640 1 -- 172.21.15.28:0/4119653004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639019f630 con 0x7f6390106f20 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f638dd9b640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63901027a0 0x7f639019abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f638ed9d640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6390106f20 0x7f639019ef50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f638800eb10 tx=0x7f638800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f637f7fe640 1 -- 172.21.15.28:0/4119653004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388004b20 con 0x7f6390106f20 2024-09-15T22:07:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f637f7fe640 1 -- 172.21.15.28:0/4119653004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6388004cc0 con 0x7f6390106f20 2024-09-15T22:07:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639019f920 con 0x7f6390106f20 2024-09-15T22:07:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f637f7fe640 1 -- 172.21.15.28:0/4119653004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388010760 con 0x7f6390106f20 2024-09-15T22:07:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.119+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63901a7200 con 0x7f6390106f20 2024-09-15T22:07:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.120+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63901021d0 con 0x7f6390106f20 2024-09-15T22:07:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.121+0000 7f637f7fe640 1 -- 172.21.15.28:0/4119653004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6388005020 con 0x7f6390106f20 2024-09-15T22:07:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.122+0000 7f637f7fe640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f63780a33b0 0x7f63780a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.122+0000 7f638e59c640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f63780a33b0 0x7f63780a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.122+0000 7f637f7fe640 1 -- 172.21.15.28:0/4119653004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f63880d07c0 con 0x7f6390106f20 2024-09-15T22:07:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.123+0000 7f638e59c640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f63780a33b0 0x7f63780a5870 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f638000bbe0 tx=0x7f638000b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.126+0000 7f637f7fe640 1 -- 172.21.15.28:0/4119653004 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f638809cdf0 con 0x7f6390106f20 2024-09-15T22:07:26.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.338+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63901a7460 con 0x7f6390106f20 2024-09-15T22:07:26.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:26 smithi028 ceph-mon[30969]: pgmap v310: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-15T22:07:26.341 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:26 smithi028 ceph-mon[30969]: osdmap e99: 12 total, 12 up, 12 in 2024-09-15T22:07:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.339+0000 7f637f7fe640 1 -- 172.21.15.28:0/4119653004 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11986 (secure 0 0 0) 0x7f638809c6f0 con 0x7f6390106f20 2024-09-15T22:07:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:07:26.342 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"btime":"2024-09-15T22:06:56:960477+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:06:56.716040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:07:26.342 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-09-15T22:07:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f63780a33b0 msgr2=0x7f63780a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f63780a33b0 0x7f63780a5870 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f638000bbe0 tx=0x7f638000b390 comp rx=0 tx=0).stop 2024-09-15T22:07:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6390106f20 msgr2=0x7f639019ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6390106f20 0x7f639019ef50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f638800eb10 tx=0x7f638800c2d0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 shutdown_connections 2024-09-15T22:07:26.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f63780a33b0 0x7f63780a5870 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6390106f20 0x7f639019ef50 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63901027a0 0x7f639019abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 --2- 172.21.15.28:0/4119653004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6390101df0 0x7f639019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:26.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 >> 172.21.15.28:0/4119653004 conn(0x7f63900fd8f0 msgr2=0x7f63900fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:26.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.342+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 shutdown_connections 2024-09-15T22:07:26.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:26.343+0000 7f6394ead640 1 -- 172.21.15.28:0/4119653004 wait complete. 2024-09-15T22:07:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:26 smithi203 ceph-mon[35546]: pgmap v310: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-15T22:07:26.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:26 smithi203 ceph-mon[35546]: osdmap e99: 12 total, 12 up, 12 in 2024-09-15T22:07:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:26 smithi174 ceph-mon[35244]: pgmap v310: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-09-15T22:07:26.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:26 smithi174 ceph-mon[35244]: osdmap e99: 12 total, 12 up, 12 in 2024-09-15T22:07:26.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T22:07:27.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:27 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4119653004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:27.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:27 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4119653004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:07:27.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:27 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4119653004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 -- 172.21.15.28:0/3792461780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 msgr2=0x7f570c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 --2- 172.21.15.28:0/3792461780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c1031d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5708009900 tx=0x7f570802a410 comp rx=0 tx=0).stop 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 -- 172.21.15.28:0/3792461780 shutdown_connections 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 --2- 172.21.15.28:0/3792461780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f570c10c2f0 0x7f570c10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 --2- 172.21.15.28:0/3792461780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f570c1037a0 0x7f570c10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 --2- 172.21.15.28:0/3792461780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c1031d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 -- 172.21.15.28:0/3792461780 >> 172.21.15.28:0/3792461780 conn(0x7f570c0fc7d0 msgr2=0x7f570c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 -- 172.21.15.28:0/3792461780 shutdown_connections 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.673+0000 7f57138a9640 1 -- 172.21.15.28:0/3792461780 wait complete. 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 Processor -- start 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 -- start start 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f570c1037a0 0x7f570c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f570c10c2f0 0x7f570c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1121c0 con 0x7f570c102df0 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f570c112040 con 0x7f570c1037a0 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f57138a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f570c112340 con 0x7f570c10c2f0 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711645640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711645640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46598/0 (socket says 172.21.15.28:46598) 2024-09-15T22:07:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711645640 1 -- 172.21.15.28:0/2457594560 learned_addr learned my addr 172.21.15.28:0/2457594560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5710e44640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f570c1037a0 0x7f570c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711e46640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f570c10c2f0 0x7f570c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711645640 1 -- 172.21.15.28:0/2457594560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f570c10c2f0 msgr2=0x7f570c19cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711645640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f570c10c2f0 0x7f570c19cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711645640 1 -- 172.21.15.28:0/2457594560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f570c1037a0 msgr2=0x7f570c198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.674+0000 7f5711645640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f570c1037a0 0x7f570c198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f5711645640 1 -- 172.21.15.28:0/2457594560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c19d610 con 0x7f570c102df0 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f5711e46640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f570c10c2f0 0x7f570c19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f5711645640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c198660 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f5708004770 tx=0x7f5708003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f57027fc640 1 -- 172.21.15.28:0/2457594560 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570803f070 con 0x7f570c102df0 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570c19d8a0 con 0x7f570c102df0 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f57027fc640 1 -- 172.21.15.28:0/2457594560 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5708031ca0 con 0x7f570c102df0 2024-09-15T22:07:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f57027fc640 1 -- 172.21.15.28:0/2457594560 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708035030 con 0x7f570c102df0 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.675+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570c1a51e0 con 0x7f570c102df0 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.677+0000 7f57027fc640 1 -- 172.21.15.28:0/2457594560 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f5708042070 con 0x7f570c102df0 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 Processor -- start 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 -- start start 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f57027fc640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56f80a33b0 0x7f56f80a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96db40 0x561b6d96c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d96ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 0x561b6d97db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561b6d8d22f0 con 0x561b6d96db40 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561b6d879cf0 con 0x561b6d96c5d0 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5715c82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561b6d8ce2b0 con 0x561b6d97b780 2024-09-15T22:07:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5711645640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96db40 0x561b6d96c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5711645640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96db40 0x561b6d96c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46612/0 (socket says 172.21.15.28:46612) 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5711645640 1 -- 172.21.15.28:0/984261523 learned_addr learned my addr 172.21.15.28:0/984261523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5711e46640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56f80a33b0 0x7f56f80a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f57027fc640 1 -- 172.21.15.28:0/2457594560 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f57080f0860 con 0x7f570c102df0 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.678+0000 7f5710e44640 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d96ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f5711645640 1 -- 172.21.15.28:0/984261523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 msgr2=0x561b6d97db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f5711645640 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 0x561b6d97db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f5711645640 1 -- 172.21.15.28:0/984261523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 msgr2=0x561b6d96ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f5711645640 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d96ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f5711645640 1 -- 172.21.15.28:0/984261523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b6d85e3a0 con 0x561b6d96db40 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f5711645640 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96db40 0x561b6d96c090 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f57080edb40 tx=0x7f570803b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9810b860838b5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f5711e46640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56f80a33b0 0x7f56f80a5870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5704007fa0 tx=0x7f5704007900 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f56f77fe640 1 -- 172.21.15.28:0/984261523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708049020 con 0x561b6d96db40 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f56f77fe640 1 -- 172.21.15.28:0/984261523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57080c6260 con 0x561b6d96db40 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.679+0000 7f56f77fe640 1 -- 172.21.15.28:0/984261523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57080bda60 con 0x561b6d96db40 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 -- 172.21.15.28:0/984261523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96db40 msgr2=0x561b6d96c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96db40 0x561b6d96c090 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f57080edb40 tx=0x7f570803b890 comp rx=0 tx=0).stop 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 -- 172.21.15.28:0/984261523 shutdown_connections 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 0x561b6d97db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d96ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 --2- 172.21.15.28:0/984261523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96db40 0x561b6d96c090 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 -- 172.21.15.28:0/984261523 >> 172.21.15.28:0/984261523 conn(0x561b6d935310 msgr2=0x561b6d968c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 -- 172.21.15.28:0/984261523 shutdown_connections 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 -- 172.21.15.28:0/984261523 wait complete. 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 -- start start 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.680+0000 7f5715c82740 1 Processor -- start 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5715c82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d97b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5715c82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d978420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5715c82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d978a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5715c82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561b6d97e240 con 0x561b6d97b780 2024-09-15T22:07:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5715c82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561b6d88d060 con 0x561b6d96c5d0 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5715c82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561b6d97e3c0 con 0x561b6d96db40 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5715c82740 10 client.0 _ll_register_callbacks cb 0x561b6d5e08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d978a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711645640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d97b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d978a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46616/0 (socket says 172.21.15.28:46616) 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 -- 172.21.15.28:0/1978588049 learned_addr learned my addr 172.21.15.28:0/1978588049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5710e44640 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d978420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 -- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 msgr2=0x561b6d978420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d978420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 -- 172.21.15.28:0/1978588049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 msgr2=0x561b6d97b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d97b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 -- 172.21.15.28:0/1978588049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6d6433c0 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711645640 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d97b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f5711e46640 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d978a10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f5704011a90 tx=0x7f5704004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d7485b52441e17 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.681+0000 7f56f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f56f4ff9640 1 -- 172.21.15.28:0/1978588049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704019050 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f56f4ff9640 1 -- 172.21.15.28:0/1978588049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f570401e440 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57080c6260 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f56f4ff9640 1 -- 172.21.15.28:0/1978588049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f570402c050 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 10 client.17559 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561b6d989c20 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f56f4ff9640 1 -- 172.21.15.28:0/1978588049 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57080c6260 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f56f4ff9640 1 -- 172.21.15.28:0/1978588049 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f5704027620 con 0x561b6d97b780 2024-09-15T22:07:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 10 client.17559 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 10 client.17559 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 20 client.17559 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d98b0e0 0x561b6d98d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 4 client.17559 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.682+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b6d8df860 con 0x561b6d98b0e0 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.683+0000 7f5711645640 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d98b0e0 0x561b6d98d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.683+0000 7f56f4ff9640 10 client.17559 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.683+0000 7f5711645640 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d98b0e0 0x561b6d98d560 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f9859a537e9b28b8 server_cookie=33ae287d8f10a516 in_seq=0 out_seq=0 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f56f4ff9640 1 -- 172.21.15.28:0/1978588049 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b6d8df860 con 0x561b6d98b0e0 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f56f4ff9640 10 client.17559 handle_command_reply: tid=0 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 client.17559 shutdown 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 msgr2=0x561b6d978a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d978a10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f5704011a90 tx=0x7f5704004b50 comp rx=0 tx=0).stop 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 shutdown_connections 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d98b0e0 0x561b6d98d560 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d978a10 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d978420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f56f4ff9640 0 client.17559 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 --2- 172.21.15.28:0/1978588049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d97b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 >> 172.21.15.28:0/1978588049 conn(0x561b6d935310 msgr2=0x561b6d96e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 shutdown_connections 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.684+0000 7f5715c82740 1 -- 172.21.15.28:0/1978588049 wait complete. 2024-09-15T22:07:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.685+0000 7f5715c82740 20 client.17559 trim_cache size 0 max 16384 2024-09-15T22:07:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 Processor -- start 2024-09-15T22:07:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 -- start start 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d9874a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d992e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d993370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1acec0 con 0x561b6d97b780 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1aea00 con 0x561b6d96c5d0 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5715c82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1ad590 con 0x561b6d96db40 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d993370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d993370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46632/0 (socket says 172.21.15.28:46632) 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 -- 172.21.15.28:0/1454732663 learned_addr learned my addr 172.21.15.28:0/1454732663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5710e44640 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d992e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 -- 172.21.15.28:0/1454732663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 msgr2=0x561b6d992e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d992e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 -- 172.21.15.28:0/1454732663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 msgr2=0x561b6d9874a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711645640 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d9874a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d9874a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 -- 172.21.15.28:0/1454732663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b6d96d300 con 0x561b6d97b780 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.698+0000 7f5711e46640 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d993370 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f57040114e0 tx=0x7f5704007b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fa57675d18ffc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f56f6ffd640 1 -- 172.21.15.28:0/1454732663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704019040 con 0x561b6d97b780 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f56f6ffd640 1 -- 172.21.15.28:0/1454732663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5704034030 con 0x561b6d97b780 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f56f6ffd640 1 -- 172.21.15.28:0/1454732663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570401e8a0 con 0x561b6d97b780 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 -- 172.21.15.28:0/1454732663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 msgr2=0x561b6d993370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d993370 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f57040114e0 tx=0x7f5704007b50 comp rx=0 tx=0).stop 2024-09-15T22:07:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 -- 172.21.15.28:0/1454732663 shutdown_connections 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d97b780 0x561b6d993370 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d96db40 0x561b6d992e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 --2- 172.21.15.28:0/1454732663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96c5d0 0x561b6d9874a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 -- 172.21.15.28:0/1454732663 >> 172.21.15.28:0/1454732663 conn(0x561b6d935310 msgr2=0x561b6d9783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 -- 172.21.15.28:0/1454732663 shutdown_connections 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.699+0000 7f5715c82740 1 -- 172.21.15.28:0/1454732663 wait complete. 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 -- start start 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 Processor -- start 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96c5d0 0x561b6d97f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96db40 0x561b6d97fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 0x561b6d980070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1ae330 con 0x561b6d96c5d0 2024-09-15T22:07:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1adc60 con 0x561b6d96db40 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f570c1aaec0 con 0x561b6d97b780 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5715c82740 10 client.0 _ll_register_callbacks cb 0x7f5704007c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5711645640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96c5d0 0x561b6d97f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5711645640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96c5d0 0x561b6d97f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46638/0 (socket says 172.21.15.28:46638) 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.700+0000 7f5711645640 1 -- 172.21.15.28:0/476945705 learned_addr learned my addr 172.21.15.28:0/476945705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5711645640 1 -- 172.21.15.28:0/476945705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 msgr2=0x561b6d980070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5711645640 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 0x561b6d980070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5711645640 1 -- 172.21.15.28:0/476945705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96db40 msgr2=0x561b6d97fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5711645640 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96db40 0x561b6d97fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5711645640 1 -- 172.21.15.28:0/476945705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6d9f50f0 con 0x561b6d96c5d0 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5711645640 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96c5d0 0x561b6d97f5f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f57080bc6c0 tx=0x7f57080bceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c688f605f97f8bdf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f56f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f56f4ff9640 1 -- 172.21.15.28:0/476945705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708049040 con 0x561b6d96c5d0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f56f4ff9640 1 -- 172.21.15.28:0/476945705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f57080c55f0 con 0x561b6d96c5d0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5704034030 con 0x561b6d96c5d0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f56f4ff9640 1 -- 172.21.15.28:0/476945705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f57080cd050 con 0x561b6d96c5d0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5715c82740 10 client.17565 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561b6d9f6370 con 0x561b6d96c5d0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.701+0000 7f56f4ff9640 1 -- 172.21.15.28:0/476945705 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5704034030 con 0x561b6d96c5d0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f56f4ff9640 1 -- 172.21.15.28:0/476945705 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f57080ef070 con 0x561b6d96c5d0 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5715c82740 10 client.17565 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5715c82740 10 client.17565 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5715c82740 20 client.17565 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5715c82740 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d9f6900 0x561b6d9f8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5715c82740 4 client.17565 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 --> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b6d8d9d70 con 0x561b6d9f6900 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5710e44640 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d9f6900 0x561b6d9f8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f5710e44640 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d9f6900 0x561b6d9f8da0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e6c99246dc221abe server_cookie=9b4f985c1ffbcb18 in_seq=0 out_seq=0 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.702+0000 7f56f4ff9640 10 client.17565 ms_handle_connect on v2:172.21.15.203:6834/1524895518 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f56f4ff9640 1 -- 172.21.15.28:0/476945705 <== mds.4 v2:172.21.15.203:6834/1524895518 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b6d8d9d70 con 0x561b6d9f6900 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f56f4ff9640 10 client.17565 handle_command_reply: tid=0 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 client.17565 shutdown 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96c5d0 msgr2=0x561b6d97f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96c5d0 0x561b6d97f5f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f57080bc6c0 tx=0x7f57080bceb0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 shutdown_connections 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.203:6834/1524895518,v1:172.21.15.203:6835/1524895518] conn(0x561b6d9f6900 0x561b6d9f8da0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b6d97b780 0x561b6d980070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b6d96db40 0x561b6d97fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f56f4ff9640 0 client.17565 ms_handle_reset on v2:172.21.15.203:6834/1524895518 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 --2- 172.21.15.28:0/476945705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561b6d96c5d0 0x561b6d97f5f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 >> 172.21.15.28:0/476945705 conn(0x561b6d935310 msgr2=0x561b6d979c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 shutdown_connections 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 1 -- 172.21.15.28:0/476945705 wait complete. 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.703+0000 7f5715c82740 20 client.17565 trim_cache size 0 max 16384 2024-09-15T22:07:27.715 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56f80a33b0 msgr2=0x7f56f80a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56f80a33b0 0x7f56f80a5870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5704007fa0 tx=0x7f5704007900 comp rx=0 tx=0).stop 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 msgr2=0x7f570c198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c198660 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f5708004770 tx=0x7f5708003c10 comp rx=0 tx=0).stop 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 shutdown_connections 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f56f80a33b0 0x7f56f80a5870 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f570c10c2f0 0x7f570c19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f570c1037a0 0x7f570c198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 --2- 172.21.15.28:0/2457594560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f570c102df0 0x7f570c198660 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 >> 172.21.15.28:0/2457594560 conn(0x7f570c0fc7d0 msgr2=0x7f570c0fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 shutdown_connections 2024-09-15T22:07:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:27.704+0000 7f57138a9640 1 -- 172.21.15.28:0/2457594560 wait complete. 2024-09-15T22:07:28.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:28 smithi028 ceph-mon[30969]: pgmap v312: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-15T22:07:28.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:28 smithi203 ceph-mon[35546]: pgmap v312: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-15T22:07:28.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:28 smithi174 ceph-mon[35244]: pgmap v312: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-09-15T22:07:28.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T22:07:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 -- 172.21.15.28:0/2723107214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558810c170 msgr2=0x7f558810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 --2- 172.21.15.28:0/2723107214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558810c170 0x7f558810e610 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f5580009ca0 tx=0x7f558002a950 comp rx=0 tx=0).stop 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 -- 172.21.15.28:0/2723107214 shutdown_connections 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 --2- 172.21.15.28:0/2723107214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f558810c170 0x7f558810e610 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 --2- 172.21.15.28:0/2723107214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5588103660 0x7f558810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 --2- 172.21.15.28:0/2723107214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5588102d40 0x7f5588103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 -- 172.21.15.28:0/2723107214 >> 172.21.15.28:0/2723107214 conn(0x7f55880fc7a0 msgr2=0x7f55880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 -- 172.21.15.28:0/2723107214 shutdown_connections 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.358+0000 7f558d51a640 1 -- 172.21.15.28:0/2723107214 wait complete. 2024-09-15T22:07:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 Processor -- start 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 -- start start 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5588102d40 0x7f5588198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5588103660 0x7f55881989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f558810c170 0x7f558819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5588111fd0 con 0x7f5588103660 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5588111e50 con 0x7f5588102d40 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f558d51a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5588112150 con 0x7f558810c170 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.359+0000 7f55867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5588103660 0x7f55881989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5588103660 0x7f55881989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46658/0 (socket says 172.21.15.28:46658) 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 -- 172.21.15.28:0/541716328 learned_addr learned my addr 172.21.15.28:0/541716328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55877fe640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f558810c170 0x7f558819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f5586ffd640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5588102d40 0x7f5588198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 -- 172.21.15.28:0/541716328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f558810c170 msgr2=0x7f558819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f558810c170 0x7f558819cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 -- 172.21.15.28:0/541716328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5588102d40 msgr2=0x7f5588198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5588102d40 0x7f5588198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 -- 172.21.15.28:0/541716328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558819d440 con 0x7f5588103660 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f5586ffd640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5588102d40 0x7f5588198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f55867fc640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5588103660 0x7f55881989d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f557c00b3f0 tx=0x7f557c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f556ffff640 1 -- 172.21.15.28:0/541716328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c015020 con 0x7f5588103660 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558819d730 con 0x7f5588103660 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f556ffff640 1 -- 172.21.15.28:0/541716328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f557c011d10 con 0x7f5588103660 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f556ffff640 1 -- 172.21.15.28:0/541716328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c00f2e0 con 0x7f5588103660 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.360+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55881a5010 con 0x7f5588103660 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f556ffff640 1 -- 172.21.15.28:0/541716328 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f557c018020 con 0x7f5588103660 2024-09-15T22:07:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 Processor -- start 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 -- start start 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5001090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5001a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b500e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b4b4f672f0 con 0x55b4b500c370 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b4b4f0ecf0 con 0x55b4b5002b40 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f558f8f3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b4b4f632b0 con 0x55b4b50015d0 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f55867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b500e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f55867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b500e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46666/0 (socket says 172.21.15.28:46666) 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.363+0000 7f55867fc640 1 -- 172.21.15.28:0/3347377146 learned_addr learned my addr 172.21.15.28:0/3347377146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f55867fc640 1 -- 172.21.15.28:0/3347377146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 msgr2=0x55b4b5001a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f55867fc640 1 --2- 172.21.15.28:0/3347377146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5001a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f55867fc640 1 -- 172.21.15.28:0/3347377146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 msgr2=0x55b4b5001090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f55867fc640 1 --2- 172.21.15.28:0/3347377146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5001090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f55867fc640 1 -- 172.21.15.28:0/3347377146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4b4ef33a0 con 0x55b4b500c370 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f55867fc640 1 --2- 172.21.15.28:0/3347377146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b500e760 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f557c0ccbe0 tx=0x7f557c0d7690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f404c7f6fbbbd10 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f556ffff640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f55700a33b0 0x7f55700a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f556d7fa640 1 -- 172.21.15.28:0/3347377146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c0de070 con 0x55b4b500c370 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f55877fe640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f55700a33b0 0x7f55700a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f556d7fa640 1 -- 172.21.15.28:0/3347377146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f557c010eb0 con 0x55b4b500c370 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f556d7fa640 1 -- 172.21.15.28:0/3347377146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c097aa0 con 0x55b4b500c370 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.364+0000 7f556ffff640 1 -- 172.21.15.28:0/541716328 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f557c0cf7c0 con 0x7f5588103660 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 -- 172.21.15.28:0/3347377146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 msgr2=0x55b4b500e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3347377146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b500e760 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f557c0ccbe0 tx=0x7f557c0d7690 comp rx=0 tx=0).stop 2024-09-15T22:07:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f55877fe640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f55700a33b0 0x7f55700a5870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5580005fe0 tx=0x7f5580005f70 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 -- 172.21.15.28:0/3347377146 shutdown_connections 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3347377146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b500e760 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3347377146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5001a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3347377146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5001090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 -- 172.21.15.28:0/3347377146 >> 172.21.15.28:0/3347377146 conn(0x55b4b4fca310 msgr2=0x55b4b4ffdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 -- 172.21.15.28:0/3347377146 shutdown_connections 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 -- 172.21.15.28:0/3347377146 wait complete. 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.365+0000 7f558f8f3740 1 -- start start 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 1 Processor -- start 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b50015d0 0x55b4b500bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5008fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b500c370 0x55b4b5009570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b4b500f070 con 0x55b4b50015d0 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b4b4f22060 con 0x55b4b5002b40 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b4b500f1f0 con 0x55b4b500c370 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f558f8f3740 10 client.0 _ll_register_callbacks cb 0x55b4b4c758c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b50015d0 0x55b4b500bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b50015d0 0x55b4b500bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46668/0 (socket says 172.21.15.28:46668) 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 -- 172.21.15.28:0/4008875668 learned_addr learned my addr 172.21.15.28:0/4008875668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f55867fc640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5008fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f55877fe640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b500c370 0x55b4b5009570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 -- 172.21.15.28:0/4008875668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b500c370 msgr2=0x55b4b5009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b500c370 0x55b4b5009570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 -- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 msgr2=0x55b4b5008fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5008fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f5586ffd640 1 -- 172.21.15.28:0/4008875668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4b4cd83c0 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.366+0000 7f55867fc640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5008fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f5586ffd640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b50015d0 0x55b4b500bd20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f557800e370 tx=0x7f557800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a470b7b736ae766 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f556aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f556aff5640 1 -- 172.21.15.28:0/4008875668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578016070 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f556aff5640 1 -- 172.21.15.28:0/4008875668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f55780108a0 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b4b4f74860 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f556aff5640 1 -- 172.21.15.28:0/4008875668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5578022050 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f558f8f3740 10 client.17577 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b4b501ab20 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.367+0000 7f556aff5640 1 -- 172.21.15.28:0/4008875668 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b4b4f74860 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f556aff5640 1 -- 172.21.15.28:0/4008875668 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f557800f040 con 0x55b4b50015d0 2024-09-15T22:07:29.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f558f8f3740 10 client.17577 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:29.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f558f8f3740 10 client.17577 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:07:29.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f558f8f3740 20 client.17577 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:29.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f558f8f3740 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b501bf30 0x55b4b501e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f558f8f3740 4 client.17577 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:07:29.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4b4f6ed70 con 0x55b4b501bf30 2024-09-15T22:07:29.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.368+0000 7f55867fc640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b501bf30 0x55b4b501e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.369+0000 7f55867fc640 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b501bf30 0x55b4b501e380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a7ca50ed07e6e19 server_cookie=1b041783673773e0 in_seq=0 out_seq=0 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.369+0000 7f556aff5640 10 client.17577 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f556aff5640 1 -- 172.21.15.28:0/4008875668 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b4b4f6ed70 con 0x55b4b501bf30 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f556aff5640 10 client.17577 handle_command_reply: tid=0 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 client.17577 shutdown 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b50015d0 msgr2=0x55b4b500bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b50015d0 0x55b4b500bd20 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f557800e370 tx=0x7f557800e830 comp rx=0 tx=0).stop 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 shutdown_connections 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b500c370 0x55b4b5009570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5008fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b501bf30 0x55b4b501e380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 --2- 172.21.15.28:0/4008875668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b50015d0 0x55b4b500bd20 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 >> 172.21.15.28:0/4008875668 conn(0x55b4b4fca310 msgr2=0x55b4b5003960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f556aff5640 0 client.17577 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:07:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 shutdown_connections 2024-09-15T22:07:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 1 -- 172.21.15.28:0/4008875668 wait complete. 2024-09-15T22:07:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.370+0000 7f558f8f3740 20 client.17577 trim_cache size 0 max 16384 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 Processor -- start 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 -- start start 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5018310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5004120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5004660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55881af1c0 con 0x55b4b500c370 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f55881b0d00 con 0x55b4b5002b40 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.396+0000 7f558f8f3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55881af890 con 0x55b4b50015d0 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5004660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5004660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46684/0 (socket says 172.21.15.28:46684) 2024-09-15T22:07:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 -- 172.21.15.28:0/1008741373 learned_addr learned my addr 172.21.15.28:0/1008741373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55867fc640 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5004120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f5586ffd640 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5018310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 -- 172.21.15.28:0/1008741373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 msgr2=0x55b4b5018310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5018310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 -- 172.21.15.28:0/1008741373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 msgr2=0x55b4b5004120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5004120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 -- 172.21.15.28:0/1008741373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4b5004ba0 con 0x55b4b500c370 2024-09-15T22:07:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f55877fe640 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5004660 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f5580035000 tx=0x7f558003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a51fe80912b2dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f556cff9640 1 -- 172.21.15.28:0/1008741373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5580034040 con 0x55b4b500c370 2024-09-15T22:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f556cff9640 1 -- 172.21.15.28:0/1008741373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5580004040 con 0x55b4b500c370 2024-09-15T22:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.397+0000 7f556cff9640 1 -- 172.21.15.28:0/1008741373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55800306a0 con 0x55b4b500c370 2024-09-15T22:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 -- 172.21.15.28:0/1008741373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 msgr2=0x55b4b5004660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5004660 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f5580035000 tx=0x7f558003f000 comp rx=0 tx=0).stop 2024-09-15T22:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 -- 172.21.15.28:0/1008741373 shutdown_connections 2024-09-15T22:07:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5004660 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b5004120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 --2- 172.21.15.28:0/1008741373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5018310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 -- 172.21.15.28:0/1008741373 >> 172.21.15.28:0/1008741373 conn(0x55b4b4fca310 msgr2=0x55b4b50080c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 -- 172.21.15.28:0/1008741373 shutdown_connections 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.398+0000 7f558f8f3740 1 -- 172.21.15.28:0/1008741373 wait complete. 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 -- start start 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 Processor -- start 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5014890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b500fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5010100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55881b0630 con 0x55b4b500c370 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f55881aff60 con 0x55b4b5002b40 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55881ad1c0 con 0x55b4b50015d0 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f558f8f3740 10 client.0 _ll_register_callbacks cb 0x7f5578009300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f55877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5010100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f55877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5010100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46700/0 (socket says 172.21.15.28:46700) 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f55877fe640 1 -- 172.21.15.28:0/3914960105 learned_addr learned my addr 172.21.15.28:0/3914960105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f5586ffd640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5014890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.399+0000 7f55867fc640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b500fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f55877fe640 1 -- 172.21.15.28:0/3914960105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 msgr2=0x55b4b5014890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f55877fe640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5014890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f55877fe640 1 -- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 msgr2=0x55b4b500fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f55877fe640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b500fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f55877fe640 1 -- 172.21.15.28:0/3914960105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4b5010640 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f55867fc640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b500fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f55877fe640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5010100 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f558002a990 tx=0x7f558002ffe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ead29ba1a8d5b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f556aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f556aff5640 1 -- 172.21.15.28:0/3914960105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5580034040 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f556aff5640 1 -- 172.21.15.28:0/3914960105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f5580003a30 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5580004040 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f556aff5640 1 -- 172.21.15.28:0/3914960105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5580063050 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f558f8f3740 10 client.17583 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.400+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b4b5086940 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f556aff5640 1 -- 172.21.15.28:0/3914960105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5580004040 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f556aff5640 1 -- 172.21.15.28:0/3914960105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f5580052430 con 0x55b4b500c370 2024-09-15T22:07:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f558f8f3740 10 client.17583 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f558f8f3740 10 client.17583 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f558f8f3740 20 client.17583 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b5087050 0x55b4b5089440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f558f8f3740 4 client.17583 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 --> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4b4eeb170 con 0x55b4b5087050 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f5586ffd640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b5087050 0x55b4b5089440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.401+0000 7f556aff5640 10 client.17583 ms_handle_connect on v2:172.21.15.174:6836/3327498469 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f5586ffd640 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b5087050 0x55b4b5089440 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2de978bebe4714f5 server_cookie=a0a74dbc0d265e6 in_seq=0 out_seq=0 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f556aff5640 1 -- 172.21.15.28:0/3914960105 <== mds.3 v2:172.21.15.174:6836/3327498469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4b4eeb170 con 0x55b4b5087050 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f556aff5640 10 client.17583 handle_command_reply: tid=0 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 client.17583 shutdown 2024-09-15T22:07:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 msgr2=0x55b4b5010100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5010100 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f558002a990 tx=0x7f558002ffe0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 shutdown_connections 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:6836/3327498469,v1:172.21.15.174:6837/3327498469] conn(0x55b4b5087050 0x55b4b5089440 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b4b500c370 0x55b4b5010100 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b4b5002b40 0x55b4b500fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 --2- 172.21.15.28:0/3914960105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b4b50015d0 0x55b4b5014890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f556aff5640 0 client.17583 ms_handle_reset on v2:172.21.15.174:6836/3327498469 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 >> 172.21.15.28:0/3914960105 conn(0x55b4b4fca310 msgr2=0x55b4b4ffc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 shutdown_connections 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 1 -- 172.21.15.28:0/3914960105 wait complete. 2024-09-15T22:07:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.402+0000 7f558f8f3740 20 client.17583 trim_cache size 0 max 16384 2024-09-15T22:07:29.410 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:07:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f55700a33b0 msgr2=0x7f55700a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f55700a33b0 0x7f55700a5870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5580005fe0 tx=0x7f5580005f70 comp rx=0 tx=0).stop 2024-09-15T22:07:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5588103660 msgr2=0x7f55881989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5588103660 0x7f55881989d0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f557c00b3f0 tx=0x7f557c00b8b0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 shutdown_connections 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f55700a33b0 0x7f55700a5870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f558810c170 0x7f558819cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5588103660 0x7f55881989d0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 --2- 172.21.15.28:0/541716328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5588102d40 0x7f5588198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 >> 172.21.15.28:0/541716328 conn(0x7f55880fc7a0 msgr2=0x7f55880fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 shutdown_connections 2024-09-15T22:07:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:29.403+0000 7f558d51a640 1 -- 172.21.15.28:0/541716328 wait complete. 2024-09-15T22:07:30.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T22:07:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:30.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:30 smithi028 ceph-mon[30969]: pgmap v313: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-15T22:07:30.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:30 smithi203 ceph-mon[35546]: pgmap v313: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-15T22:07:30.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:30 smithi174 ceph-mon[35244]: pgmap v313: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 -- 172.21.15.28:0/3205243665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 msgr2=0x7fd25c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 --2- 172.21.15.28:0/3205243665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c106e60 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009cd0 tx=0x7fd24c02a8e0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 -- 172.21.15.28:0/3205243665 shutdown_connections 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 --2- 172.21.15.28:0/3205243665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd25c101420 0x7fd25c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 --2- 172.21.15.28:0/3205243665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd25c100a80 0x7fd25c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 --2- 172.21.15.28:0/3205243665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c106e60 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 -- 172.21.15.28:0/3205243665 >> 172.21.15.28:0/3205243665 conn(0x7fd25c0fc7a0 msgr2=0x7fd25c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 -- 172.21.15.28:0/3205243665 shutdown_connections 2024-09-15T22:07:30.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.895+0000 7fd261f6a640 1 -- 172.21.15.28:0/3205243665 wait complete. 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 Processor -- start 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 -- start start 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd25c100a80 0x7fd25c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd25c101420 0x7fd25c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c110e40 con 0x7fd25c106a80 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c110cc0 con 0x7fd25c101420 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd261f6a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c110fc0 con 0x7fd25c100a80 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd25bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd25bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c19cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60156/0 (socket says 172.21.15.28:60156) 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd25bfff640 1 -- 172.21.15.28:0/345898565 learned_addr learned my addr 172.21.15.28:0/345898565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.896+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd25c100a80 0x7fd25c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd25affd640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd25c101420 0x7fd25c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd25bfff640 1 -- 172.21.15.28:0/345898565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd25c100a80 msgr2=0x7fd25c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd25bfff640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd25c100a80 0x7fd25c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd25bfff640 1 -- 172.21.15.28:0/345898565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd25c101420 msgr2=0x7fd25c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd25bfff640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd25c101420 0x7fd25c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd25bfff640 1 -- 172.21.15.28:0/345898565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c19d3c0 con 0x7fd25c106a80 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd25bfff640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c19cce0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd25400e970 tx=0x7fd25400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd258ff9640 1 -- 172.21.15.28:0/345898565 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25400ccc0 con 0x7fd25c106a80 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c19d6b0 con 0x7fd25c106a80 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd258ff9640 1 -- 172.21.15.28:0/345898565 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25400ce60 con 0x7fd25c106a80 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd258ff9640 1 -- 172.21.15.28:0/345898565 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254010490 con 0x7fd25c106a80 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.897+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25c1a4f90 con 0x7fd25c106a80 2024-09-15T22:07:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd258ff9640 1 -- 172.21.15.28:0/345898565 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd254005020 con 0x7fd25c106a80 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 Processor -- start 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 -- start start 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e81517c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e81517ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e815189760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e8150e22f0 con 0x55e815187370 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e815089cf0 con 0x55e81517db40 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.900+0000 7fd264343740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e8150de2b0 con 0x55e81517c5d0 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e815189760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e815189760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60160/0 (socket says 172.21.15.28:60160) 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 -- 172.21.15.28:0/1434410055 learned_addr learned my addr 172.21.15.28:0/1434410055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e81517c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e81517ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd258ff9640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2440a33b0 0x7fd2440a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 -- 172.21.15.28:0/1434410055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 msgr2=0x55e81517ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e81517ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 -- 172.21.15.28:0/1434410055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 msgr2=0x55e81517c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e81517c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 -- 172.21.15.28:0/1434410055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e81506e3a0 con 0x55e815187370 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25bfff640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2440a33b0 0x7fd2440a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd258ff9640 1 -- 172.21.15.28:0/345898565 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fd2540d05a0 con 0x7fd25c106a80 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e815189760 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fd2500149b0 tx=0x7fd25000c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab0594474a83854 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd241ffb640 1 -- 172.21.15.28:0/1434410055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2500142d0 con 0x55e815187370 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd241ffb640 1 -- 172.21.15.28:0/1434410055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd250005500 con 0x55e815187370 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.901+0000 7fd241ffb640 1 -- 172.21.15.28:0/1434410055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25000f760 con 0x55e815187370 2024-09-15T22:07:30.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 -- 172.21.15.28:0/1434410055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 msgr2=0x55e815189760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e815189760 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fd2500149b0 tx=0x7fd25000c4b0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd25bfff640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2440a33b0 0x7fd2440a5870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd2540cd9f0 tx=0x7fd254091960 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 -- 172.21.15.28:0/1434410055 shutdown_connections 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e815189760 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e81517ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 --2- 172.21.15.28:0/1434410055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e81517c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 -- 172.21.15.28:0/1434410055 >> 172.21.15.28:0/1434410055 conn(0x55e815145310 msgr2=0x55e815178c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 -- 172.21.15.28:0/1434410055 shutdown_connections 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.902+0000 7fd264343740 1 -- 172.21.15.28:0/1434410055 wait complete. 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 -- start start 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 Processor -- start 2024-09-15T22:07:30.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e815186d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e815183ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e8151845e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e81518a070 con 0x55e815187370 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e81509d060 con 0x55e81517db40 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e81518a1f0 con 0x55e81517c5d0 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd264343740 10 client.0 _ll_register_callbacks cb 0x55e814df08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e8151845e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e8151845e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60176/0 (socket says 172.21.15.28:60176) 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd25affd640 1 -- 172.21.15.28:0/3079471159 learned_addr learned my addr 172.21.15.28:0/3079471159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd25affd640 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e815186d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.903+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e815183ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd25affd640 1 -- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 msgr2=0x55e815186d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd25affd640 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e815186d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd25affd640 1 -- 172.21.15.28:0/3079471159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 msgr2=0x55e815183ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd25affd640 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e815183ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd25affd640 1 -- 172.21.15.28:0/3079471159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e814e533c0 con 0x55e815187370 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd25affd640 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e8151845e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd250001eb0 tx=0x7fd250004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192b38ace47f447a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd23affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd23affd640 1 -- 172.21.15.28:0/3079471159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250004030 con 0x55e815187370 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd23affd640 1 -- 172.21.15.28:0/3079471159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fd2500056b0 con 0x55e815187370 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd250005500 con 0x55e815187370 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd23affd640 1 -- 172.21.15.28:0/3079471159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd2500041d0 con 0x55e815187370 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd264343740 10 client.17595 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e815195ae0 con 0x55e815187370 2024-09-15T22:07:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.904+0000 7fd23affd640 1 -- 172.21.15.28:0/3079471159 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd250005500 con 0x55e815187370 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd23affd640 1 -- 172.21.15.28:0/3079471159 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fd250003680 con 0x55e815187370 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd264343740 10 client.17595 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd264343740 10 client.17595 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd264343740 20 client.17595 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd264343740 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815196ee0 0x55e8151992d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd264343740 4 client.17595 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8150ef860 con 0x55e815196ee0 2024-09-15T22:07:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.905+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815196ee0 0x55e8151992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.906+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815196ee0 0x55e8151992d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=57c893f983b4d343 server_cookie=5603f7e765027e3f in_seq=0 out_seq=0 2024-09-15T22:07:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.906+0000 7fd23affd640 10 client.17595 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:07:30.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.906+0000 7fd23affd640 1 -- 172.21.15.28:0/3079471159 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e8150ef860 con 0x55e815196ee0 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.906+0000 7fd23affd640 10 client.17595 handle_command_reply: tid=0 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.906+0000 7fd264343740 1 client.17595 shutdown 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.906+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 msgr2=0x55e8151845e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e8151845e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd250001eb0 tx=0x7fd250004c40 comp rx=0 tx=0).stop 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 shutdown_connections 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815196ee0 0x55e8151992d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd23affd640 0 client.17595 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e8151845e0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517db40 0x55e815183ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 --2- 172.21.15.28:0/3079471159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517c5d0 0x55e815186d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 >> 172.21.15.28:0/3079471159 conn(0x55e815145310 msgr2=0x55e81517e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 shutdown_connections 2024-09-15T22:07:30.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 1 -- 172.21.15.28:0/3079471159 wait complete. 2024-09-15T22:07:30.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.907+0000 7fd264343740 20 client.17595 trim_cache size 0 max 16384 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 Processor -- start 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 -- start start 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517c5d0 0x55e815193310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815182fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e81519e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1af0f0 con 0x55e815187370 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1b0c30 con 0x55e81517c5d0 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd264343740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1af7c0 con 0x55e81517db40 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.920+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e81519e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e81519e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60184/0 (socket says 172.21.15.28:60184) 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 -- 172.21.15.28:0/2618305550 learned_addr learned my addr 172.21.15.28:0/2618305550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517c5d0 0x55e815193310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815182fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 -- 172.21.15.28:0/2618305550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 msgr2=0x55e815182fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815182fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 -- 172.21.15.28:0/2618305550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517c5d0 msgr2=0x55e815193310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517c5d0 0x55e815193310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 -- 172.21.15.28:0/2618305550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e815185460 con 0x55e815187370 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd25affd640 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e81519e9c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd250000c00 tx=0x7fd25000c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b29e8ffac5a72a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd2417fa640 1 -- 172.21.15.28:0/2618305550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250005b90 con 0x55e815187370 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd2417fa640 1 -- 172.21.15.28:0/2618305550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25000c950 con 0x55e815187370 2024-09-15T22:07:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.921+0000 7fd2417fa640 1 -- 172.21.15.28:0/2618305550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250003850 con 0x55e815187370 2024-09-15T22:07:30.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 -- 172.21.15.28:0/2618305550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 msgr2=0x55e81519e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e81519e9c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd250000c00 tx=0x7fd25000c4b0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 -- 172.21.15.28:0/2618305550 shutdown_connections 2024-09-15T22:07:30.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e815187370 0x55e81519e9c0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815182fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 --2- 172.21.15.28:0/2618305550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e81517c5d0 0x55e815193310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 -- 172.21.15.28:0/2618305550 >> 172.21.15.28:0/2618305550 conn(0x55e815145310 msgr2=0x55e8151828c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 -- 172.21.15.28:0/2618305550 shutdown_connections 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 -- 172.21.15.28:0/2618305550 wait complete. 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 -- start start 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.922+0000 7fd264343740 1 Processor -- start 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd264343740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81517c5d0 0x55e81518fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd264343740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815180e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd264343740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e815196ee0 0x55e815181620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd264343740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1b0560 con 0x55e81517c5d0 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd264343740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1afe90 con 0x55e815196ee0 2024-09-15T22:07:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd264343740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1ad0f0 con 0x55e81517db40 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd264343740 10 client.0 _ll_register_callbacks cb 0x7fd25000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81517c5d0 0x55e81518fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81517c5d0 0x55e81518fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60196/0 (socket says 172.21.15.28:60196) 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 -- 172.21.15.28:0/1434414075 learned_addr learned my addr 172.21.15.28:0/1434414075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815180e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e815196ee0 0x55e815181620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 -- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 msgr2=0x55e815180e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815180e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 -- 172.21.15.28:0/1434414075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e815196ee0 msgr2=0x55e815181620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e815196ee0 0x55e815181620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 -- 172.21.15.28:0/1434414075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e815200990 con 0x55e81517c5d0 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd25affd640 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81517c5d0 0x55e81518fd60 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fd250003bc0 tx=0x7fd250014410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b53721d999b03d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.923+0000 7fd23affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd23affd640 1 -- 172.21.15.28:0/1434414075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25000cbd0 con 0x55e81517c5d0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd23affd640 1 -- 172.21.15.28:0/1434414075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fd250003c40 con 0x55e81517c5d0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd25000c950 con 0x55e81517c5d0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd23affd640 1 -- 172.21.15.28:0/1434414075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd250043050 con 0x55e81517c5d0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 10 client.17601 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e815201b70 con 0x55e81517c5d0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd23affd640 1 -- 172.21.15.28:0/1434414075 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25000c950 con 0x55e81517c5d0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd23affd640 1 -- 172.21.15.28:0/1434414075 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7fd25000f850 con 0x55e81517c5d0 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 10 client.17601 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 10 client.17601 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T22:07:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 20 client.17601 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815201ff0 0x55e8152043e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 4 client.17601 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.924+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 --> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8150e9d70 con 0x55e815201ff0 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.925+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815201ff0 0x55e8152043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.925+0000 7fd23affd640 10 client.17601 ms_handle_connect on v2:172.21.15.203:6836/803844131 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.925+0000 7fd25b7fe640 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815201ff0 0x55e8152043e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fd55b4b8164bab0 server_cookie=1cd6eeafd2456e1c in_seq=0 out_seq=0 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.925+0000 7fd23affd640 1 -- 172.21.15.28:0/1434414075 <== mds.2 v2:172.21.15.203:6836/803844131 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8150e9d70 con 0x55e815201ff0 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.925+0000 7fd23affd640 10 client.17601 handle_command_reply: tid=0 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.925+0000 7fd264343740 1 client.17601 shutdown 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81517c5d0 msgr2=0x55e81518fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81517c5d0 0x55e81518fd60 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fd250003bc0 tx=0x7fd250014410 comp rx=0 tx=0).stop 2024-09-15T22:07:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 shutdown_connections 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:6836/803844131,v1:172.21.15.203:6837/803844131] conn(0x55e815201ff0 0x55e8152043e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e815196ee0 0x55e815181620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e81517db40 0x55e815180e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 --2- 172.21.15.28:0/1434414075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e81517c5d0 0x55e81518fd60 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd23affd640 0 client.17601 ms_handle_reset on v2:172.21.15.203:6836/803844131 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 >> 172.21.15.28:0/1434414075 conn(0x55e815145310 msgr2=0x55e8151830e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 shutdown_connections 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 1 -- 172.21.15.28:0/1434414075 wait complete. 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd264343740 20 client.17601 trim_cache size 0 max 16384 2024-09-15T22:07:30.933 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:07:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2440a33b0 msgr2=0x7fd2440a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2440a33b0 0x7fd2440a5870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd2540cd9f0 tx=0x7fd254091960 comp rx=0 tx=0).stop 2024-09-15T22:07:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 msgr2=0x7fd25c19cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c19cce0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fd25400e970 tx=0x7fd25400ed70 comp rx=0 tx=0).stop 2024-09-15T22:07:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 shutdown_connections 2024-09-15T22:07:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd2440a33b0 0x7fd2440a5870 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd25c106a80 0x7fd25c19cce0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd25c101420 0x7fd25c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 --2- 172.21.15.28:0/345898565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd25c100a80 0x7fd25c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.926+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 >> 172.21.15.28:0/345898565 conn(0x7fd25c0fc7a0 msgr2=0x7fd25c06a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.927+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 shutdown_connections 2024-09-15T22:07:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:30.927+0000 7fd261f6a640 1 -- 172.21.15.28:0/345898565 wait complete. 2024-09-15T22:07:31.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-09-15T22:07:31.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T22:07:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.754+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/2467528377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa81069c0 msgr2=0x7fbfa8106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.754+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/2467528377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa81069c0 0x7fbfa8106da0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009930 tx=0x7fbfa402a420 comp rx=0 tx=0).stop 2024-09-15T22:07:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/2467528377 shutdown_connections 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/2467528377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfa8101350 0x7fbfa810d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/2467528377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa81009b0 0x7fbfa8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/2467528377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa81069c0 0x7fbfa8106da0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/2467528377 >> 172.21.15.28:0/2467528377 conn(0x7fbfa8077990 msgr2=0x7fbfa80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/2467528377 shutdown_connections 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/2467528377 wait complete. 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 Processor -- start 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 -- start start 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.755+0000 7fbfaf4e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa81009b0 0x7fbfa8199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaf4e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8101350 0x7fbfa81996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaf4e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfa81069c0 0x7fbfa81a4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaf4e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8110e00 con 0x7fbfa8101350 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaf4e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8110c80 con 0x7fbfa81069c0 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaf4e6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa8110f80 con 0x7fbfa81009b0 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8101350 0x7fbfa81996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8101350 0x7fbfa81996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60214/0 (socket says 172.21.15.28:60214) 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 -- 172.21.15.28:0/3854390603 learned_addr learned my addr 172.21.15.28:0/3854390603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfad282640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa81009b0 0x7fbfa8199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfada83640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfa81069c0 0x7fbfa81a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 -- 172.21.15.28:0/3854390603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa81009b0 msgr2=0x7fbfa8199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa81009b0 0x7fbfa8199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 -- 172.21.15.28:0/3854390603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfa81069c0 msgr2=0x7fbfa81a4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfa81069c0 0x7fbfa81a4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 -- 172.21.15.28:0/3854390603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa81a4750 con 0x7fbfa8101350 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfada83640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfa81069c0 0x7fbfa81a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaca81640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8101350 0x7fbfa81996d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00d770 tx=0x7fbf9c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbf9a7fc640 1 -- 172.21.15.28:0/3854390603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c017070 con 0x7fbfa8101350 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbf9a7fc640 1 -- 172.21.15.28:0/3854390603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf9c005500 con 0x7fbfa8101350 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa81a4920 con 0x7fbfa8101350 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbf9a7fc640 1 -- 172.21.15.28:0/3854390603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c004e00 con 0x7fbfa8101350 2024-09-15T22:07:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.756+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa81a4e20 con 0x7fbfa8101350 2024-09-15T22:07:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.757+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa8100e10 con 0x7fbfa8101350 2024-09-15T22:07:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.759+0000 7fbf9a7fc640 1 -- 172.21.15.28:0/3854390603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fbf9c004fa0 con 0x7fbfa8101350 2024-09-15T22:07:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.759+0000 7fbf9a7fc640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbf940a3460 0x7fbf940a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.760+0000 7fbfad282640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbf940a3460 0x7fbf940a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.760+0000 7fbf9a7fc640 1 -- 172.21.15.28:0/3854390603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fbf9c0cff80 con 0x7fbfa8101350 2024-09-15T22:07:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.760+0000 7fbfad282640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbf940a3460 0x7fbf940a5920 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009900 tx=0x7fbfa40058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.763+0000 7fbf9a7fc640 1 -- 172.21.15.28:0/3854390603 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbf9c09c5b0 con 0x7fbfa8101350 2024-09-15T22:07:31.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.964+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fbfa8103e60 con 0x7fbf940a3460 2024-09-15T22:07:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.973+0000 7fbf9a7fc640 1 -- 172.21.15.28:0/3854390603 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fbfa8103e60 con 0x7fbf940a3460 2024-09-15T22:07:31.974 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-09-15T00:00:00", "created": "2024-09-15T21:57:30", "first": "2024-09-15T21:58:00", "last": "2024-09-15T22:07:00", "last_pruned": "2024-09-15T22:07:00", "created_count": 10, "pruned_count": 3, "active": true} 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbf940a3460 msgr2=0x7fbf940a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbf940a3460 0x7fbf940a5920 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009900 tx=0x7fbfa40058d0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8101350 msgr2=0x7fbfa81996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8101350 0x7fbfa81996d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00d770 tx=0x7fbf9c00dc30 comp rx=0 tx=0).stop 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 shutdown_connections 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fbf940a3460 0x7fbf940a5920 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfa81069c0 0x7fbfa81a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8101350 0x7fbfa81996d0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 --2- 172.21.15.28:0/3854390603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa81009b0 0x7fbfa8199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 >> 172.21.15.28:0/3854390603 conn(0x7fbfa8077990 msgr2=0x7fbfa8103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 shutdown_connections 2024-09-15T22:07:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:31.975+0000 7fbfaf4e6640 1 -- 172.21.15.28:0/3854390603 wait complete. 2024-09-15T22:07:31.986 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:31.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-09-15T22:07:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 -- 172.21.15.28:0/2842468580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7941069c0 msgr2=0x7fa794106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 --2- 172.21.15.28:0/2842468580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7941069c0 0x7fa794106da0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009930 tx=0x7fa77c02a420 comp rx=0 tx=0).stop 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 -- 172.21.15.28:0/2842468580 shutdown_connections 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 --2- 172.21.15.28:0/2842468580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa794101350 0x7fa79410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 --2- 172.21.15.28:0/2842468580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7941009b0 0x7fa794100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 --2- 172.21.15.28:0/2842468580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7941069c0 0x7fa794106da0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 -- 172.21.15.28:0/2842468580 >> 172.21.15.28:0/2842468580 conn(0x7fa794077990 msgr2=0x7fa7940fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 -- 172.21.15.28:0/2842468580 shutdown_connections 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.132+0000 7fa79999f640 1 -- 172.21.15.28:0/2842468580 wait complete. 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 Processor -- start 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 -- start start 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7941009b0 0x7fa7940fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa794101350 0x7fa7940fd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7941069c0 0x7fa7940fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa794110e00 con 0x7fa794101350 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa794110c80 con 0x7fa7941069c0 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa79999f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa794110f80 con 0x7fa7941009b0 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa7927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa794101350 0x7fa7940fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa7927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa794101350 0x7fa7940fd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60242/0 (socket says 172.21.15.28:60242) 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.133+0000 7fa7927fc640 1 -- 172.21.15.28:0/2047287688 learned_addr learned my addr 172.21.15.28:0/2047287688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa7937fe640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7941069c0 0x7fa7940fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa7927fc640 1 -- 172.21.15.28:0/2047287688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7941009b0 msgr2=0x7fa7940fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa7927fc640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7941009b0 0x7fa7940fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa7927fc640 1 -- 172.21.15.28:0/2047287688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7941069c0 msgr2=0x7fa7940fd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa7927fc640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7941069c0 0x7fa7940fd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa7927fc640 1 -- 172.21.15.28:0/2047287688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7941a68b0 con 0x7fa794101350 2024-09-15T22:07:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa7927fc640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa794101350 0x7fa7940fd320 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa78400c970 tx=0x7fa78400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa79899d640 1 -- 172.21.15.28:0/2047287688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784007bb0 con 0x7fa794101350 2024-09-15T22:07:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa79899d640 1 -- 172.21.15.28:0/2047287688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa784007d50 con 0x7fa794101350 2024-09-15T22:07:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7941a6ba0 con 0x7fa794101350 2024-09-15T22:07:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa79899d640 1 -- 172.21.15.28:0/2047287688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78400f450 con 0x7fa794101350 2024-09-15T22:07:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.134+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7941a70f0 con 0x7fa794101350 2024-09-15T22:07:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.135+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa794100e10 con 0x7fa794101350 2024-09-15T22:07:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.136+0000 7fa79899d640 1 -- 172.21.15.28:0/2047287688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fa784005020 con 0x7fa794101350 2024-09-15T22:07:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.137+0000 7fa79899d640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7800a34e0 0x7fa7800a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.137+0000 7fa79899d640 1 -- 172.21.15.28:0/2047287688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fa7840cfee0 con 0x7fa794101350 2024-09-15T22:07:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.138+0000 7fa792ffd640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7800a34e0 0x7fa7800a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.138+0000 7fa792ffd640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7800a34e0 0x7fa7800a59a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa77c02a940 tx=0x7fa77c000bc0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.143+0000 7fa79899d640 1 -- 172.21.15.28:0/2047287688 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa784014030 con 0x7fa794101350 2024-09-15T22:07:32.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:32 smithi028 ceph-mon[30969]: pgmap v314: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-15T22:07:32.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.348+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 --> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa79406a0f0 con 0x7fa7800a34e0 2024-09-15T22:07:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.356+0000 7fa79899d640 1 -- 172.21.15.28:0/2047287688 <== mgr.15330 v2:172.21.15.28:6800/1380427157 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa79406a0f0 con 0x7fa7800a34e0 2024-09-15T22:07:32.358 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-09-15T22:07:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7800a34e0 msgr2=0x7fa7800a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7800a34e0 0x7fa7800a59a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa77c02a940 tx=0x7fa77c000bc0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa794101350 msgr2=0x7fa7940fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa794101350 0x7fa7940fd320 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa78400c970 tx=0x7fa78400ce30 comp rx=0 tx=0).stop 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 shutdown_connections 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa7800a34e0 0x7fa7800a59a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7941069c0 0x7fa7940fd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa794101350 0x7fa7940fd320 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 --2- 172.21.15.28:0/2047287688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7941009b0 0x7fa7940fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.358+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 >> 172.21.15.28:0/2047287688 conn(0x7fa794077990 msgr2=0x7fa7940699e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.359+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 shutdown_connections 2024-09-15T22:07:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.359+0000 7fa79999f640 1 -- 172.21.15.28:0/2047287688 wait complete. 2024-09-15T22:07:32.371 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-09-15T22:07:32.382 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-09-15T22:07:32.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:32 smithi203 ceph-mon[35546]: pgmap v314: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-15T22:07:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 -- 172.21.15.28:0/1614506043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40101de0 msgr2=0x7f9d401021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 --2- 172.21.15.28:0/1614506043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40101de0 0x7f9d401021c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f9d30009900 tx=0x7f9d3002a410 comp rx=0 tx=0).stop 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 -- 172.21.15.28:0/1614506043 shutdown_connections 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 --2- 172.21.15.28:0/1614506043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d40106f10 0x7f9d4010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 --2- 172.21.15.28:0/1614506043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d40102790 0x7f9d401067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 --2- 172.21.15.28:0/1614506043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40101de0 0x7f9d401021c0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 -- 172.21.15.28:0/1614506043 >> 172.21.15.28:0/1614506043 conn(0x7f9d400fd8c0 msgr2=0x7f9d400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 -- 172.21.15.28:0/1614506043 shutdown_connections 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.498+0000 7f9d450e0640 1 -- 172.21.15.28:0/1614506043 wait complete. 2024-09-15T22:07:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 Processor -- start 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 -- start start 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d40101de0 0x7f9d40198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d40102790 0x7f9d40198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40106f10 0x7f9d4019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d40111f30 con 0x7f9d40106f10 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d40111db0 con 0x7f9d40102790 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d450e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401120b0 con 0x7f9d40101de0 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d3f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40106f10 0x7f9d4019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d3f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40106f10 0x7f9d4019cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60254/0 (socket says 172.21.15.28:60254) 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.499+0000 7f9d3f59e640 1 -- 172.21.15.28:0/490417975 learned_addr learned my addr 172.21.15.28:0/490417975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d37fff640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d40102790 0x7f9d40198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d40101de0 0x7f9d40198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3f59e640 1 -- 172.21.15.28:0/490417975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d40101de0 msgr2=0x7f9d40198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3f59e640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d40101de0 0x7f9d40198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3f59e640 1 -- 172.21.15.28:0/490417975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d40102790 msgr2=0x7f9d40198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3f59e640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d40102790 0x7f9d40198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3f59e640 1 -- 172.21.15.28:0/490417975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4019d4c0 con 0x7f9d40106f10 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3f59e640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40106f10 0x7f9d4019cde0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800eaa0 tx=0x7f9d3800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3cd99640 1 -- 172.21.15.28:0/490417975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3800cdb0 con 0x7f9d40106f10 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4019d7b0 con 0x7f9d40106f10 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3cd99640 1 -- 172.21.15.28:0/490417975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d38005020 con 0x7f9d40106f10 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d3cd99640 1 -- 172.21.15.28:0/490417975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d380116b0 con 0x7f9d40106f10 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.500+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d401a5090 con 0x7f9d40106f10 2024-09-15T22:07:32.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 Processor -- start 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d3cd99640 1 -- 172.21.15.28:0/490417975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f9d38005560 con 0x7f9d40106f10 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 -- start start 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aaeb40 0x55e384aad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aad5d0 0x55e384aada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384abc780 0x55e384abeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e384a132f0 con 0x55e384aaeb40 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e3849bacf0 con 0x55e384abc780 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d474b9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e384a0f2b0 con 0x55e384aad5d0 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.503+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aaeb40 0x55e384aad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aaeb40 0x55e384aad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60270/0 (socket says 172.21.15.28:60270) 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 -- 172.21.15.28:0/3392250475 learned_addr learned my addr 172.21.15.28:0/3392250475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:32.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384abc780 0x55e384abeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aad5d0 0x55e384aada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d3cd99640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d2c0a33b0 0x7f9d2c0a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 -- 172.21.15.28:0/3392250475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aad5d0 msgr2=0x55e384aada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aad5d0 0x55e384aada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 -- 172.21.15.28:0/3392250475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384abc780 msgr2=0x55e384abeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384abc780 0x55e384abeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 -- 172.21.15.28:0/3392250475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e38499f3a0 con 0x55e384aaeb40 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d3f59e640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d2c0a33b0 0x7f9d2c0a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d37fff640 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aaeb40 0x55e384aad090 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800b6c0 tx=0x7f9d2800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0a2247aa7be0f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d3cd99640 1 -- 172.21.15.28:0/490417975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f9d380d0210 con 0x7f9d40106f10 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d357fa640 1 -- 172.21.15.28:0/3392250475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28012da0 con 0x55e384aaeb40 2024-09-15T22:07:32.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d357fa640 1 -- 172.21.15.28:0/3392250475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d28010770 con 0x55e384aaeb40 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.504+0000 7f9d357fa640 1 -- 172.21.15.28:0/3392250475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28003850 con 0x55e384aaeb40 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 -- 172.21.15.28:0/3392250475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aaeb40 msgr2=0x55e384aad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d3f59e640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d2c0a33b0 0x7f9d2c0a5870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d3809c180 tx=0x7f9d38091680 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aaeb40 0x55e384aad090 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800b6c0 tx=0x7f9d2800b650 comp rx=0 tx=0).stop 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 -- 172.21.15.28:0/3392250475 shutdown_connections 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384abc780 0x55e384abeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aad5d0 0x55e384aada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 --2- 172.21.15.28:0/3392250475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aaeb40 0x55e384aad090 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 -- 172.21.15.28:0/3392250475 >> 172.21.15.28:0/3392250475 conn(0x55e384a76310 msgr2=0x55e384aa9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 -- 172.21.15.28:0/3392250475 shutdown_connections 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.505+0000 7f9d474b9740 1 -- 172.21.15.28:0/3392250475 wait complete. 2024-09-15T22:07:32.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 -- start start 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 Processor -- start 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ab9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384ab9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e384abf420 con 0x55e384aad5d0 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e3849ce060 con 0x55e384aaeb40 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e384abf5a0 con 0x55e384abc780 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d474b9740 10 client.0 _ll_register_callbacks cb 0x55e3847218c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60282/0 (socket says 172.21.15.28:60282) 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.506+0000 7f9d37fff640 1 -- 172.21.15.28:0/4109705838 learned_addr learned my addr 172.21.15.28:0/4109705838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ab9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d37fff640 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384ab9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d37fff640 1 -- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 msgr2=0x55e384ab9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d37fff640 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384ab9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d37fff640 1 -- 172.21.15.28:0/4109705838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 msgr2=0x55e384ab9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d37fff640 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ab9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d37fff640 1 -- 172.21.15.28:0/4109705838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3847843c0 con 0x55e384aad5d0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d37fff640 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abc190 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f9d28015000 tx=0x7f9d28012da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e198157788f61495 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d267fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d267fc640 1 -- 172.21.15.28:0/4109705838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28010d30 con 0x55e384aad5d0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d267fc640 1 -- 172.21.15.28:0/4109705838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f9d28005040 con 0x55e384aad5d0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d28010770 con 0x55e384aad5d0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d267fc640 1 -- 172.21.15.28:0/4109705838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9d28033050 con 0x55e384aad5d0 2024-09-15T22:07:32.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d474b9740 10 client.17625 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.507+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e384acaef0 con 0x55e384aad5d0 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d267fc640 1 -- 172.21.15.28:0/4109705838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d28010770 con 0x55e384aad5d0 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d267fc640 1 -- 172.21.15.28:0/4109705838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f9d28004a50 con 0x55e384aad5d0 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d474b9740 10 client.17625 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d474b9740 10 client.17625 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d474b9740 20 client.17625 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d474b9740 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384acc240 0x55e384ace630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d474b9740 4 client.17625 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:07:32.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e384a20860 con 0x55e384acc240 2024-09-15T22:07:32.519 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:32 smithi174 ceph-mon[35244]: pgmap v314: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-09-15T22:07:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.508+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384acc240 0x55e384ace630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.509+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384acc240 0x55e384ace630 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78e88a8072198ee9 server_cookie=4095bf15cc12858f in_seq=0 out_seq=0 2024-09-15T22:07:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.509+0000 7f9d267fc640 10 client.17625 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:07:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.509+0000 7f9d267fc640 1 -- 172.21.15.28:0/4109705838 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e384a20860 con 0x55e384acc240 2024-09-15T22:07:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.509+0000 7f9d267fc640 10 client.17625 handle_command_reply: tid=0 2024-09-15T22:07:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 client.17625 shutdown 2024-09-15T22:07:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 msgr2=0x55e384abc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abc190 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f9d28015000 tx=0x7f9d28012da0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 shutdown_connections 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384acc240 0x55e384ace630 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d267fc640 0 client.17625 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384ab9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ab9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 --2- 172.21.15.28:0/4109705838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abc190 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 >> 172.21.15.28:0/4109705838 conn(0x55e384a76310 msgr2=0x55e384aaf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 shutdown_connections 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 1 -- 172.21.15.28:0/4109705838 wait complete. 2024-09-15T22:07:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.510+0000 7f9d474b9740 20 client.17625 trim_cache size 0 max 16384 2024-09-15T22:07:32.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 Processor -- start 2024-09-15T22:07:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 -- start start 2024-09-15T22:07:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aad5d0 0x55e384ac86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aaeb40 0x55e384ab8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384abc780 0x55e384aa9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401acdb0 con 0x55e384abc780 2024-09-15T22:07:32.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401ae8f0 con 0x55e384aad5d0 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d474b9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401ad480 con 0x55e384aaeb40 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.523+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384abc780 0x55e384aa9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384abc780 0x55e384aa9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60288/0 (socket says 172.21.15.28:60288) 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 -- 172.21.15.28:0/1774632685 learned_addr learned my addr 172.21.15.28:0/1774632685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aaeb40 0x55e384ab8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aad5d0 0x55e384ac86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 -- 172.21.15.28:0/1774632685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aaeb40 msgr2=0x55e384ab8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aaeb40 0x55e384ab8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 -- 172.21.15.28:0/1774632685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aad5d0 msgr2=0x55e384ac86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aad5d0 0x55e384ac86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 -- 172.21.15.28:0/1774632685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e384ab0120 con 0x55e384abc780 2024-09-15T22:07:32.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aaeb40 0x55e384ab8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d37fff640 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384abc780 0x55e384aa9510 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f9d28003670 tx=0x7f9d28022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9d912670d471b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d34ff9640 1 -- 172.21.15.28:0/1774632685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800bab0 con 0x55e384abc780 2024-09-15T22:07:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d34ff9640 1 -- 172.21.15.28:0/1774632685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d28010740 con 0x55e384abc780 2024-09-15T22:07:32.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.524+0000 7f9d34ff9640 1 -- 172.21.15.28:0/1774632685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d280049d0 con 0x55e384abc780 2024-09-15T22:07:32.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 -- 172.21.15.28:0/1774632685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384abc780 msgr2=0x55e384aa9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384abc780 0x55e384aa9510 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f9d28003670 tx=0x7f9d28022040 comp rx=0 tx=0).stop 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 -- 172.21.15.28:0/1774632685 shutdown_connections 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384abc780 0x55e384aa9510 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384aaeb40 0x55e384ab8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 --2- 172.21.15.28:0/1774632685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aad5d0 0x55e384ac86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 -- 172.21.15.28:0/1774632685 >> 172.21.15.28:0/1774632685 conn(0x55e384a76310 msgr2=0x55e384ab8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 -- 172.21.15.28:0/1774632685 shutdown_connections 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 -- 172.21.15.28:0/1774632685 wait complete. 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.525+0000 7f9d474b9740 1 -- start start 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 1 Processor -- start 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ac04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384b2ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401ae220 con 0x55e384aad5d0 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401adb50 con 0x55e384aaeb40 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401aadb0 con 0x55e384abc780 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d474b9740 10 client.0 _ll_register_callbacks cb 0x7f9d2800be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60304/0 (socket says 172.21.15.28:60304) 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 -- 172.21.15.28:0/988800143 learned_addr learned my addr 172.21.15.28:0/988800143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ac04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384b2ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 -- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 msgr2=0x55e384b2ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384b2ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 -- 172.21.15.28:0/988800143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 msgr2=0x55e384ac04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ac04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 -- 172.21.15.28:0/988800143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e384b2f2f0 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d37fff640 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abff80 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f9d280108f0 tx=0x7f9d2800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2d83dc2da41000 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.526+0000 7f9d267fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d267fc640 1 -- 172.21.15.28:0/988800143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28044060 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d267fc640 1 -- 172.21.15.28:0/988800143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f9d28003c00 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d28010740 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d267fc640 1 -- 172.21.15.28:0/988800143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9d28003e00 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 10 client.17631 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e384b36660 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d267fc640 1 -- 172.21.15.28:0/988800143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d28010740 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d267fc640 1 -- 172.21.15.28:0/988800143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f9d28005270 con 0x55e384aad5d0 2024-09-15T22:07:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 10 client.17631 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 10 client.17631 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 20 client.17631 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384b36cd0 0x55e384b390c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 4 client.17631 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.527+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 --> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e384a1ad70 con 0x55e384b36cd0 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384b36cd0 0x55e384b390c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d3ed9d640 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384b36cd0 0x55e384b390c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e92b1c2cf5e4ed51 server_cookie=cb8cf9d7fa9d21b8 in_seq=0 out_seq=0 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d267fc640 10 client.17631 ms_handle_connect on v2:172.21.15.203:6838/3242294678 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d267fc640 1 -- 172.21.15.28:0/988800143 <== mds.1 v2:172.21.15.203:6838/3242294678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e384a1ad70 con 0x55e384b36cd0 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d267fc640 10 client.17631 handle_command_reply: tid=0 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d474b9740 1 client.17631 shutdown 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 msgr2=0x55e384abff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d474b9740 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abff80 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f9d280108f0 tx=0x7f9d2800bdf0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 shutdown_connections 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.528+0000 7f9d474b9740 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:6838/3242294678,v1:172.21.15.203:6839/3242294678] conn(0x55e384b36cd0 0x55e384b390c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d474b9740 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e384abc780 0x55e384b2ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d267fc640 0 client.17631 ms_handle_reset on v2:172.21.15.203:6838/3242294678 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d474b9740 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e384aaeb40 0x55e384ac04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d474b9740 1 --2- 172.21.15.28:0/988800143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e384aad5d0 0x55e384abff80 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 >> 172.21.15.28:0/988800143 conn(0x55e384a76310 msgr2=0x55e384abaca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 shutdown_connections 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d474b9740 1 -- 172.21.15.28:0/988800143 wait complete. 2024-09-15T22:07:32.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d474b9740 20 client.17631 trim_cache size 0 max 16384 2024-09-15T22:07:32.537 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:07:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d2c0a33b0 msgr2=0x7f9d2c0a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d2c0a33b0 0x7f9d2c0a5870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d3809c180 tx=0x7f9d38091680 comp rx=0 tx=0).stop 2024-09-15T22:07:32.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40106f10 msgr2=0x7f9d4019cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40106f10 0x7f9d4019cde0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800eaa0 tx=0x7f9d3800ef60 comp rx=0 tx=0).stop 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 shutdown_connections 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9d2c0a33b0 0x7f9d2c0a5870 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d40106f10 0x7f9d4019cde0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d40102790 0x7f9d40198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 --2- 172.21.15.28:0/490417975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d40101de0 0x7f9d40198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.529+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 >> 172.21.15.28:0/490417975 conn(0x7f9d400fd8c0 msgr2=0x7f9d400fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.530+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 shutdown_connections 2024-09-15T22:07:32.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:32.530+0000 7f9d450e0640 1 -- 172.21.15.28:0/490417975 wait complete. 2024-09-15T22:07:33.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:33 smithi028 ceph-mon[30969]: from='client.17607 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T22:07:33.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T22:07:33.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:33 smithi203 ceph-mon[35546]: from='client.17607 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T22:07:33.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:33 smithi174 ceph-mon[35244]: from='client.17607 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T22:07:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 -- 172.21.15.28:0/3512129459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 msgr2=0x7f6b0810d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/3512129459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0810d930 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009cd0 tx=0x7f6b0002a8e0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 -- 172.21.15.28:0/3512129459 shutdown_connections 2024-09-15T22:07:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/3512129459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0810d930 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/3512129459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b08100a60 0x7f6b08100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/3512129459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b08106a60 0x7f6b08106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 -- 172.21.15.28:0/3512129459 >> 172.21.15.28:0/3512129459 conn(0x7f6b080fc780 msgr2=0x7f6b080feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 -- 172.21.15.28:0/3512129459 shutdown_connections 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.074+0000 7f6b0d22f640 1 -- 172.21.15.28:0/3512129459 wait complete. 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 Processor -- start 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 -- start start 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b08100a60 0x7f6b0819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b08106a60 0x7f6b0819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b08110e60 con 0x7f6b08101400 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b08110ce0 con 0x7f6b08106a60 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0d22f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b08110fe0 con 0x7f6b08100a60 2024-09-15T22:07:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0819aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60322/0 (socket says 172.21.15.28:60322) 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 -- 172.21.15.28:0/1937694951 learned_addr learned my addr 172.21.15.28:0/1937694951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b08100a60 0x7f6b0819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0759e640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b08106a60 0x7f6b0819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 -- 172.21.15.28:0/1937694951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b08100a60 msgr2=0x7f6b0819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b08100a60 0x7f6b0819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 -- 172.21.15.28:0/1937694951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b08106a60 msgr2=0x7f6b0819ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b08106a60 0x7f6b0819ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6afffff640 1 -- 172.21.15.28:0/1937694951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0819f560 con 0x7f6b08101400 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b08100a60 0x7f6b0819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.075+0000 7f6b0759e640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b08106a60 0x7f6b0819ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.076+0000 7f6afffff640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0819aaf0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6af000e9f0 tx=0x7f6af000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.076+0000 7f6b04d99640 1 -- 172.21.15.28:0/1937694951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af000cd80 con 0x7f6b08101400 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.076+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0819f850 con 0x7f6b08101400 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.076+0000 7f6b04d99640 1 -- 172.21.15.28:0/1937694951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6af0010910 con 0x7f6b08101400 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.076+0000 7f6b04d99640 1 -- 172.21.15.28:0/1937694951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af00188f0 con 0x7f6b08101400 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.076+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b081a7130 con 0x7f6b08101400 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.078+0000 7f6b04d99640 1 -- 172.21.15.28:0/1937694951 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f6af0010ac0 con 0x7f6b08101400 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.078+0000 7f6b0f608740 1 Processor -- start 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.078+0000 7f6b0f608740 1 -- start start 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3d3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d35d0 0x562c0d3d3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3e0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0f608740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562c0d3392f0 con 0x562c0d3d35d0 2024-09-15T22:07:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0f608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562c0d2e0cf0 con 0x562c0d3de370 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0f608740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c0d3352b0 con 0x562c0d3d4b40 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d35d0 0x562c0d3d3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d35d0 0x562c0d3d3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60338/0 (socket says 172.21.15.28:60338) 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 -- 172.21.15.28:0/3339535934 learned_addr learned my addr 172.21.15.28:0/3339535934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6afffff640 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3e0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0759e640 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3d3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 -- 172.21.15.28:0/3339535934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 msgr2=0x562c0d3d3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b04d99640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6af40a33b0 0x7f6af40a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3d3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 -- 172.21.15.28:0/3339535934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 msgr2=0x562c0d3e0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3e0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b06d9d640 1 -- 172.21.15.28:0/3339535934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c0d2c53a0 con 0x562c0d3d35d0 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.079+0000 7f6b0759e640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6af40a33b0 0x7f6af40a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.080+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d35d0 0x562c0d3d3a10 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009bd0 tx=0x7f6af800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c832f025efb43b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.080+0000 7f6b04d99640 1 -- 172.21.15.28:0/1937694951 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f6af00d0630 con 0x7f6b08101400 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.080+0000 7f6afd7fa640 1 -- 172.21.15.28:0/3339535934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af8016070 con 0x562c0d3d35d0 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.080+0000 7f6afd7fa640 1 -- 172.21.15.28:0/3339535934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6af8005500 con 0x562c0d3d35d0 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.080+0000 7f6afd7fa640 1 -- 172.21.15.28:0/3339535934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af80116d0 con 0x562c0d3d35d0 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 -- 172.21.15.28:0/3339535934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d35d0 msgr2=0x562c0d3d3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0759e640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6af40a33b0 0x7f6af40a5870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b00004af0 tx=0x7f6b000095c0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d35d0 0x562c0d3d3a10 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009bd0 tx=0x7f6af800e6c0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 -- 172.21.15.28:0/3339535934 shutdown_connections 2024-09-15T22:07:34.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3e0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d35d0 0x562c0d3d3a10 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 --2- 172.21.15.28:0/3339535934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3d3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 -- 172.21.15.28:0/3339535934 >> 172.21.15.28:0/3339535934 conn(0x562c0d39c310 msgr2=0x562c0d3cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 -- 172.21.15.28:0/3339535934 shutdown_connections 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 -- 172.21.15.28:0/3339535934 wait complete. 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 -- start start 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 Processor -- start 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3ddce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.081+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3daf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3db4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0f608740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562c0d3e1070 con 0x562c0d3de370 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0f608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562c0d2f4060 con 0x562c0d3d35d0 2024-09-15T22:07:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0f608740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562c0d3e11f0 con 0x562c0d3d4b40 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0f608740 10 client.0 _ll_register_callbacks cb 0x562c0d0478c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3db4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3db4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60352/0 (socket says 172.21.15.28:60352) 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 -- 172.21.15.28:0/649132912 learned_addr learned my addr 172.21.15.28:0/649132912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3ddce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6afffff640 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3daf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 -- 172.21.15.28:0/649132912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 msgr2=0x562c0d3daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3daf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 -- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 msgr2=0x562c0d3ddce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3ddce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 -- 172.21.15.28:0/649132912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c0d0aa3c0 con 0x562c0d3de370 2024-09-15T22:07:34.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0759e640 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3db4f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031e50 tx=0x7f6b0002a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788f263899f1b9bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6aee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6aee7fc640 1 -- 172.21.15.28:0/649132912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00004510 con 0x562c0d3de370 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6af8005500 con 0x562c0d3de370 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.082+0000 7f6aee7fc640 1 -- 172.21.15.28:0/649132912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f6b000046b0 con 0x562c0d3de370 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6aee7fc640 1 -- 172.21.15.28:0/649132912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6b000305a0 con 0x562c0d3de370 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 10 client.17643 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562c0d3eca80 con 0x562c0d3de370 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6aee7fc640 1 -- 172.21.15.28:0/649132912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6af8005500 con 0x562c0d3de370 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6aee7fc640 1 -- 172.21.15.28:0/649132912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f6b00030070 con 0x562c0d3de370 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 10 client.17643 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 10 client.17643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 20 client.17643 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:34.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d3edf40 0x562c0d3f0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 4 client.17643 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c0d346860 con 0x562c0d3edf40 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.083+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d3edf40 0x562c0d3f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.084+0000 7f6aee7fc640 10 client.17643 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.084+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d3edf40 0x562c0d3f0390 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8e3370e16dcce79 server_cookie=175eeb6753e65c0b in_seq=0 out_seq=0 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6aee7fc640 1 -- 172.21.15.28:0/649132912 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562c0d346860 con 0x562c0d3edf40 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6aee7fc640 10 client.17643 handle_command_reply: tid=0 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 client.17643 shutdown 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 msgr2=0x562c0d3db4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3db4f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031e50 tx=0x7f6b0002a8e0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 shutdown_connections 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d3edf40 0x562c0d3f0390 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3db4f0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3daf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6aee7fc640 0 client.17643 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 --2- 172.21.15.28:0/649132912 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3ddce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 >> 172.21.15.28:0/649132912 conn(0x562c0d39c310 msgr2=0x562c0d3d5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 shutdown_connections 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 1 -- 172.21.15.28:0/649132912 wait complete. 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.085+0000 7f6b0f608740 20 client.17643 trim_cache size 0 max 16384 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 Processor -- start 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 -- start start 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d35d0 0x562c0d3dcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d4b40 0x562c0d3d6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3d6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b081b1310 con 0x562c0d3d4b40 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b081b2e50 con 0x562c0d3de370 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6b0f608740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b081b19e0 con 0x562c0d3d35d0 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6afffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d4b40 0x562c0d3d6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6afffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d4b40 0x562c0d3d6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60356/0 (socket says 172.21.15.28:60356) 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.098+0000 7f6afffff640 1 -- 172.21.15.28:0/2831912997 learned_addr learned my addr 172.21.15.28:0/2831912997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6b0759e640 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3d6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d35d0 0x562c0d3dcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afffff640 1 -- 172.21.15.28:0/2831912997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d35d0 msgr2=0x562c0d3dcf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afffff640 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d35d0 0x562c0d3dcf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afffff640 1 -- 172.21.15.28:0/2831912997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 msgr2=0x562c0d3d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afffff640 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3d6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afffff640 1 -- 172.21.15.28:0/2831912997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c0d3d4760 con 0x562c0d3d4b40 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afffff640 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d4b40 0x562c0d3d6120 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6af00cda30 tx=0x7f6af009c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7caf2e61af615678 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afcff9640 1 -- 172.21.15.28:0/2831912997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af009cc60 con 0x562c0d3d4b40 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afcff9640 1 -- 172.21.15.28:0/2831912997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6af00d0070 con 0x562c0d3d4b40 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.099+0000 7f6afcff9640 1 -- 172.21.15.28:0/2831912997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af00d0220 con 0x562c0d3d4b40 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 -- 172.21.15.28:0/2831912997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d4b40 msgr2=0x562c0d3d6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d4b40 0x562c0d3d6120 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6af00cda30 tx=0x7f6af009c450 comp rx=0 tx=0).stop 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 -- 172.21.15.28:0/2831912997 shutdown_connections 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3de370 0x562c0d3d6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3d4b40 0x562c0d3d6120 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 --2- 172.21.15.28:0/2831912997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d35d0 0x562c0d3dcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 -- 172.21.15.28:0/2831912997 >> 172.21.15.28:0/2831912997 conn(0x562c0d39c310 msgr2=0x562c0d3dc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 -- 172.21.15.28:0/2831912997 shutdown_connections 2024-09-15T22:07:34.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 -- 172.21.15.28:0/2831912997 wait complete. 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 -- start start 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.100+0000 7f6b0f608740 1 Processor -- start 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3e6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3e69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3e3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b081b2780 con 0x562c0d3de370 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b081b20b0 con 0x562c0d3d35d0 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b081af310 con 0x562c0d3d4b40 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 10 client.0 _ll_register_callbacks cb 0x7f6b00009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3e3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3e3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60358/0 (socket says 172.21.15.28:60358) 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 -- 172.21.15.28:0/869741515 learned_addr learned my addr 172.21.15.28:0/869741515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:34.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6afffff640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3e69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3e6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 -- 172.21.15.28:0/869741515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 msgr2=0x562c0d3e69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3e69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 -- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 msgr2=0x562c0d3e6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3e6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 -- 172.21.15.28:0/869741515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c0d3e7030 con 0x562c0d3de370 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3e6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0759e640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3e3450 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031da0 tx=0x7f6b00003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2be13a64f0567f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6aee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6aee7fc640 1 -- 172.21.15.28:0/869741515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00063060 con 0x562c0d3de370 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6aee7fc640 1 -- 172.21.15.28:0/869741515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f6b00004310 con 0x562c0d3de370 2024-09-15T22:07:34.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.101+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6af00d0070 con 0x562c0d3de370 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6aee7fc640 1 -- 172.21.15.28:0/869741515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6b00052e70 con 0x562c0d3de370 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 10 client.17649 fetch_fsmap learned FSMap version 36 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562c0d4591b0 con 0x562c0d3de370 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6aee7fc640 1 -- 172.21.15.28:0/869741515 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6af00d0070 con 0x562c0d3de370 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6aee7fc640 1 -- 172.21.15.28:0/869741515 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8370+0+0 (secure 0 0 0) 0x7f6b0005e480 con 0x562c0d3de370 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 10 client.17649 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 20 client.17649 populate_metadata read hostname 'smithi028' 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d4598a0 0x562c0d45bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 4 client.17649 mds_command: new command op to 24485 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 --> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562c0d340d70 con 0x562c0d4598a0 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.102+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d4598a0 0x562c0d45bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:34.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6aee7fc640 10 client.17649 ms_handle_connect on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b06d9d640 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d4598a0 0x562c0d45bcc0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc2c237abaeda4e3 server_cookie=6ea024f66d368c79 in_seq=0 out_seq=0 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6aee7fc640 1 -- 172.21.15.28:0/869741515 <== mds.0 v2:172.21.15.174:6838/3735159621 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562c0d340d70 con 0x562c0d4598a0 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6aee7fc640 10 client.17649 handle_command_reply: tid=0 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 client.17649 shutdown 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 msgr2=0x562c0d3e3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3e3450 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f6b00031da0 tx=0x7f6b00003e80 comp rx=0 tx=0).stop 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 shutdown_connections 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:6838/3735159621,v1:172.21.15.174:6839/3735159621] conn(0x562c0d4598a0 0x562c0d45bcc0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562c0d3de370 0x562c0d3e3450 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562c0d3d4b40 0x562c0d3e69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6aee7fc640 0 client.17649 ms_handle_reset on v2:172.21.15.174:6838/3735159621 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 --2- 172.21.15.28:0/869741515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562c0d3d35d0 0x562c0d3e6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 >> 172.21.15.28:0/869741515 conn(0x562c0d39c310 msgr2=0x562c0d3ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 shutdown_connections 2024-09-15T22:07:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 1 -- 172.21.15.28:0/869741515 wait complete. 2024-09-15T22:07:34.139 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-15T22:07:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.103+0000 7f6b0f608740 20 client.17649 trim_cache size 0 max 16384 2024-09-15T22:07:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6af40a33b0 msgr2=0x7f6af40a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6af40a33b0 0x7f6af40a5870 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b00004af0 tx=0x7f6b000095c0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 msgr2=0x7f6b0819aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0819aaf0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6af000e9f0 tx=0x7f6af000eeb0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 shutdown_connections 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f6af40a33b0 0x7f6af40a5870 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b08106a60 0x7f6b0819ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b08101400 0x7f6b0819aaf0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 --2- 172.21.15.28:0/1937694951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b08100a60 0x7f6b0819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 >> 172.21.15.28:0/1937694951 conn(0x7f6b080fc780 msgr2=0x7f6b0806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 shutdown_connections 2024-09-15T22:07:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:34.104+0000 7f6b0d22f640 1 -- 172.21.15.28:0/1937694951 wait complete. 2024-09-15T22:07:34.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:34 smithi028 ceph-mon[30969]: from='client.17613 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T22:07:34.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:34 smithi028 ceph-mon[30969]: pgmap v315: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:34 smithi203 ceph-mon[35546]: from='client.17613 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T22:07:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:34 smithi203 ceph-mon[35546]: pgmap v315: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:34.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:34 smithi174 ceph-mon[35244]: from='client.17613 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T22:07:34.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:34 smithi174 ceph-mon[35244]: pgmap v315: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:34.889 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-15T22:07:34.890 INFO:tasks.fwd_scrub:done joining 2024-09-15T22:07:34.891 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-15T22:07:34.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 -- 172.21.15.28:0/3140703303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd440101360 msgr2=0x7fd44010d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 --2- 172.21.15.28:0/3140703303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd440101360 0x7fd44010d890 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd438009cd0 tx=0x7fd43802a8e0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 -- 172.21.15.28:0/3140703303 shutdown_connections 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 --2- 172.21.15.28:0/3140703303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd440101360 0x7fd44010d890 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 --2- 172.21.15.28:0/3140703303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4401009c0 0x7fd440100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 --2- 172.21.15.28:0/3140703303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4401069d0 0x7fd440106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 -- 172.21.15.28:0/3140703303 >> 172.21.15.28:0/3140703303 conn(0x7fd4400fc6a0 msgr2=0x7fd4400feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 -- 172.21.15.28:0/3140703303 shutdown_connections 2024-09-15T22:07:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 -- 172.21.15.28:0/3140703303 wait complete. 2024-09-15T22:07:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 Processor -- start 2024-09-15T22:07:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.008+0000 7fd4468b7640 1 -- start start 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd4468b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4401009c0 0x7fd440197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd4468b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd440101360 0x7fd440197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd4468b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4401069d0 0x7fd44019bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd4468b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd440110e10 con 0x7fd4401069d0 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd4468b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd440110c90 con 0x7fd4401009c0 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd4468b7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd440110f90 con 0x7fd440101360 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4401069d0 0x7fd44019bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4401069d0 0x7fd44019bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60370/0 (socket says 172.21.15.28:60370) 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 -- 172.21.15.28:0/4017215037 learned_addr learned my addr 172.21.15.28:0/4017215037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd437fff640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd440101360 0x7fd440197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd43ffff640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4401009c0 0x7fd440197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 -- 172.21.15.28:0/4017215037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd440101360 msgr2=0x7fd440197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd440101360 0x7fd440197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 -- 172.21.15.28:0/4017215037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4401009c0 msgr2=0x7fd440197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4401009c0 0x7fd440197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 -- 172.21.15.28:0/4017215037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44019c2f0 con 0x7fd4401069d0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.009+0000 7fd444e54640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4401069d0 0x7fd44019bbc0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd438005cc0 tx=0x7fd438031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.010+0000 7fd43dffb640 1 -- 172.21.15.28:0/4017215037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43804a070 con 0x7fd4401069d0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.010+0000 7fd43dffb640 1 -- 172.21.15.28:0/4017215037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd438004040 con 0x7fd4401069d0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.010+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44019c580 con 0x7fd4401069d0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.010+0000 7fd43dffb640 1 -- 172.21.15.28:0/4017215037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4380308d0 con 0x7fd4401069d0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.010+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44006a3c0 con 0x7fd4401069d0 2024-09-15T22:07:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.011+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd440100e20 con 0x7fd4401069d0 2024-09-15T22:07:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.012+0000 7fd43dffb640 1 -- 172.21.15.28:0/4017215037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd4380041f0 con 0x7fd4401069d0 2024-09-15T22:07:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.013+0000 7fd43dffb640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd42c0a33c0 0x7fd42c0a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.013+0000 7fd43ffff640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd42c0a33c0 0x7fd42c0a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.013+0000 7fd43dffb640 1 -- 172.21.15.28:0/4017215037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fd43802e020 con 0x7fd4401069d0 2024-09-15T22:07:35.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.013+0000 7fd43ffff640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd42c0a33c0 0x7fd42c0a5880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd430005fd0 tx=0x7fd43000b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.017+0000 7fd43dffb640 1 -- 172.21.15.28:0/4017215037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd4380811b0 con 0x7fd4401069d0 2024-09-15T22:07:35.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.194+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd44006ade0 con 0x7fd4401069d0 2024-09-15T22:07:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.196+0000 7fd43dffb640 1 -- 172.21.15.28:0/4017215037 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+25319 (secure 0 0 0) 0x7fd4380be460 con 0x7fd4401069d0 2024-09-15T22:07:35.198 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:07:35.198 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T22:07:25.905648+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":14,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":14,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/972903625":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/3676950981":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/4057362541":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/955882315":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/1670367809":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6801/1670367809":"2024-09-16T21:57:20.180486+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":14,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":14,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd42c0a33c0 msgr2=0x7fd42c0a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd42c0a33c0 0x7fd42c0a5880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd430005fd0 tx=0x7fd43000b390 comp rx=0 tx=0).stop 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4401069d0 msgr2=0x7fd44019bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4401069d0 0x7fd44019bbc0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd438005cc0 tx=0x7fd438031e60 comp rx=0 tx=0).stop 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 shutdown_connections 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd42c0a33c0 0x7fd42c0a5880 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4401069d0 0x7fd44019bbc0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd440101360 0x7fd440197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 --2- 172.21.15.28:0/4017215037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4401009c0 0x7fd440197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 >> 172.21.15.28:0/4017215037 conn(0x7fd4400fc6a0 msgr2=0x7fd4400fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:35.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 shutdown_connections 2024-09-15T22:07:35.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.198+0000 7fd4468b7640 1 -- 172.21.15.28:0/4017215037 wait complete. 2024-09-15T22:07:35.212 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T21:50:13.957276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-15T21:54:54.805516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-15T21:54:57.410111+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-15T21:55:03.884115+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-15T22:07:35.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.316+0000 7fdd65d52640 1 -- 172.21.15.28:0/2986645203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60106990 msgr2=0x7fdd60106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.316+0000 7fdd65d52640 1 --2- 172.21.15.28:0/2986645203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60106990 0x7fdd60106d70 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009cd0 tx=0x7fdd5002a8e0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 -- 172.21.15.28:0/2986645203 shutdown_connections 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 --2- 172.21.15.28:0/2986645203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd60101320 0x7fdd6010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 --2- 172.21.15.28:0/2986645203 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd60100980 0x7fdd60100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 --2- 172.21.15.28:0/2986645203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60106990 0x7fdd60106d70 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 -- 172.21.15.28:0/2986645203 >> 172.21.15.28:0/2986645203 conn(0x7fdd600779d0 msgr2=0x7fdd600fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 -- 172.21.15.28:0/2986645203 shutdown_connections 2024-09-15T22:07:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 -- 172.21.15.28:0/2986645203 wait complete. 2024-09-15T22:07:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 Processor -- start 2024-09-15T22:07:35.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.317+0000 7fdd65d52640 1 -- start start 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd65d52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60100980 0x7fdd601972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd65d52640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd60101320 0x7fdd60197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd65d52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd60106990 0x7fdd6019bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd65d52640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd60110da0 con 0x7fdd60100980 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd65d52640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdd60110c20 con 0x7fdd60106990 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd65d52640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd60110f20 con 0x7fdd60101320 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60100980 0x7fdd601972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60100980 0x7fdd601972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60394/0 (socket says 172.21.15.28:60394) 2024-09-15T22:07:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 -- 172.21.15.28:0/679873804 learned_addr learned my addr 172.21.15.28:0/679873804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5effd640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd60101320 0x7fdd60197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5ffff640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd60106990 0x7fdd6019bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 -- 172.21.15.28:0/679873804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd60101320 msgr2=0x7fdd60197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd60101320 0x7fdd60197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 -- 172.21.15.28:0/679873804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd60106990 msgr2=0x7fdd6019bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd60106990 0x7fdd6019bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 -- 172.21.15.28:0/679873804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6019c2c0 con 0x7fdd60100980 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5ffff640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd60106990 0x7fdd6019bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5f7fe640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60100980 0x7fdd601972d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fdd5002ae00 tx=0x7fdd50003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5cff9640 1 -- 172.21.15.28:0/679873804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50004240 con 0x7fdd60100980 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5cff9640 1 -- 172.21.15.28:0/679873804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd500305a0 con 0x7fdd60100980 2024-09-15T22:07:35.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6019c550 con 0x7fdd60100980 2024-09-15T22:07:35.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.318+0000 7fdd5cff9640 1 -- 172.21.15.28:0/679873804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5003a660 con 0x7fdd60100980 2024-09-15T22:07:35.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.319+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd601a3e40 con 0x7fdd60100980 2024-09-15T22:07:35.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.320+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd60100de0 con 0x7fdd60100980 2024-09-15T22:07:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.321+0000 7fdd5cff9640 1 -- 172.21.15.28:0/679873804 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fdd50030ac0 con 0x7fdd60100980 2024-09-15T22:07:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.322+0000 7fdd5cff9640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fdd4c0a3440 0x7fdd4c0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.322+0000 7fdd5effd640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fdd4c0a3440 0x7fdd4c0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.322+0000 7fdd5cff9640 1 -- 172.21.15.28:0/679873804 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fdd5002e030 con 0x7fdd60100980 2024-09-15T22:07:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.322+0000 7fdd5effd640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fdd4c0a3440 0x7fdd4c0a5900 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdd600fffb0 tx=0x7fdd4800a2d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.325+0000 7fdd5cff9640 1 -- 172.21.15.28:0/679873804 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdd500098b0 con 0x7fdd60100980 2024-09-15T22:07:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.501+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd60108f50 con 0x7fdd60100980 2024-09-15T22:07:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.501+0000 7fdd5cff9640 1 -- 172.21.15.28:0/679873804 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7fdd500bd960 con 0x7fdd60100980 2024-09-15T22:07:35.503 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-15T22:07:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fdd4c0a3440 msgr2=0x7fdd4c0a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fdd4c0a3440 0x7fdd4c0a5900 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdd600fffb0 tx=0x7fdd4800a2d0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60100980 msgr2=0x7fdd601972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60100980 0x7fdd601972d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fdd5002ae00 tx=0x7fdd50003ba0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 shutdown_connections 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fdd4c0a3440 0x7fdd4c0a5900 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd60106990 0x7fdd6019bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd60101320 0x7fdd60197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 --2- 172.21.15.28:0/679873804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd60100980 0x7fdd601972d0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.503+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 >> 172.21.15.28:0/679873804 conn(0x7fdd600779d0 msgr2=0x7fdd60103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.504+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 shutdown_connections 2024-09-15T22:07:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.504+0000 7fdd65d52640 1 -- 172.21.15.28:0/679873804 wait complete. 2024-09-15T22:07:35.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 -- 172.21.15.28:0/410969729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04106990 msgr2=0x7f0f04106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/410969729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04106990 0x7f0f04106d70 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009930 tx=0x7f0ef402a420 comp rx=0 tx=0).stop 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 -- 172.21.15.28:0/410969729 shutdown_connections 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/410969729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f04101320 0x7f0f0410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/410969729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f04100980 0x7f0f04100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/410969729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04106990 0x7f0f04106d70 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 -- 172.21.15.28:0/410969729 >> 172.21.15.28:0/410969729 conn(0x7f0f040779d0 msgr2=0x7f0f040fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 -- 172.21.15.28:0/410969729 shutdown_connections 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 -- 172.21.15.28:0/410969729 wait complete. 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.621+0000 7f0f09d4b640 1 Processor -- start 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f09d4b640 1 -- start start 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f09d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04100980 0x7f0f04198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f09d4b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f04101320 0x7f0f04198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f09d4b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f04106990 0x7f0f0419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f09d4b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f04110dd0 con 0x7f0f04100980 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f09d4b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f04110c50 con 0x7f0f04106990 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f09d4b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f04110f50 con 0x7f0f04101320 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04100980 0x7f0f04198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04100980 0x7f0f04198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60404/0 (socket says 172.21.15.28:60404) 2024-09-15T22:07:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 -- 172.21.15.28:0/544137482 learned_addr learned my addr 172.21.15.28:0/544137482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f02ffd640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f04101320 0x7f0f04198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f03fff640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f04106990 0x7f0f0419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 -- 172.21.15.28:0/544137482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f04101320 msgr2=0x7f0f04198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f04101320 0x7f0f04198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 -- 172.21.15.28:0/544137482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f04106990 msgr2=0x7f0f0419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f04106990 0x7f0f0419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 -- 172.21.15.28:0/544137482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0419d440 con 0x7f0f04100980 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f03fff640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f04106990 0x7f0f0419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:07:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.622+0000 7f0f037fe640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04100980 0x7f0f04198450 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009230 tx=0x7f0ef4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.623+0000 7f0f00ff9640 1 -- 172.21.15.28:0/544137482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef403f030 con 0x7f0f04100980 2024-09-15T22:07:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.623+0000 7f0f00ff9640 1 -- 172.21.15.28:0/544137482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef4003ad0 con 0x7f0f04100980 2024-09-15T22:07:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.623+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0419d6d0 con 0x7f0f04100980 2024-09-15T22:07:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.623+0000 7f0f00ff9640 1 -- 172.21.15.28:0/544137482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4035070 con 0x7f0f04100980 2024-09-15T22:07:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.623+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f041a4fc0 con 0x7f0f04100980 2024-09-15T22:07:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.624+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f04100de0 con 0x7f0f04100980 2024-09-15T22:07:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.625+0000 7f0f00ff9640 1 -- 172.21.15.28:0/544137482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f0ef4042030 con 0x7f0f04100980 2024-09-15T22:07:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.625+0000 7f0f00ff9640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0eec0a3440 0x7f0eec0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.626+0000 7f0f00ff9640 1 -- 172.21.15.28:0/544137482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f0ef40f05d0 con 0x7f0f04100980 2024-09-15T22:07:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.626+0000 7f0f02ffd640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0eec0a3440 0x7f0eec0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.626+0000 7f0f02ffd640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0eec0a3440 0x7f0eec0a5900 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005fd0 tx=0x7f0ef8005850 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.630+0000 7f0f00ff9640 1 -- 172.21.15.28:0/544137482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0ef40bcc00 con 0x7f0f04100980 2024-09-15T22:07:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.805+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0f041a5210 con 0x7f0f04100980 2024-09-15T22:07:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.805+0000 7f0f00ff9640 1 -- 172.21.15.28:0/544137482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) ==== 104+0+11 (secure 0 0 0) 0x7f0ef40bc500 con 0x7f0f04100980 2024-09-15T22:07:35.807 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-15T22:07:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0eec0a3440 msgr2=0x7f0eec0a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0eec0a3440 0x7f0eec0a5900 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8005fd0 tx=0x7f0ef8005850 comp rx=0 tx=0).stop 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04100980 msgr2=0x7f0f04198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04100980 0x7f0f04198450 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009230 tx=0x7f0ef4004270 comp rx=0 tx=0).stop 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 shutdown_connections 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f0eec0a3440 0x7f0eec0a5900 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f04106990 0x7f0f0419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f04101320 0x7f0f04198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 --2- 172.21.15.28:0/544137482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f04100980 0x7f0f04198450 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 >> 172.21.15.28:0/544137482 conn(0x7f0f040779d0 msgr2=0x7f0f04069490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 shutdown_connections 2024-09-15T22:07:35.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.807+0000 7f0f09d4b640 1 -- 172.21.15.28:0/544137482 wait complete. 2024-09-15T22:07:35.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-15T22:07:35.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 -- 172.21.15.28:0/2381131485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280102ce0 msgr2=0x7fa2801030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 --2- 172.21.15.28:0/2381131485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280102ce0 0x7fa2801030c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009900 tx=0x7fa27c02a480 comp rx=0 tx=0).stop 2024-09-15T22:07:35.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 -- 172.21.15.28:0/2381131485 shutdown_connections 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 --2- 172.21.15.28:0/2381131485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa28010c0b0 0x7fa28010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 --2- 172.21.15.28:0/2381131485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280103600 0x7fa28010bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 --2- 172.21.15.28:0/2381131485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280102ce0 0x7fa2801030c0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 -- 172.21.15.28:0/2381131485 >> 172.21.15.28:0/2381131485 conn(0x7fa2800fc690 msgr2=0x7fa2800feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 -- 172.21.15.28:0/2381131485 shutdown_connections 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.925+0000 7fa287d8d640 1 -- 172.21.15.28:0/2381131485 wait complete. 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 Processor -- start 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 -- start start 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280102ce0 0x7fa280198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280103600 0x7fa2801989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa28010c0b0 0x7fa28019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa280111eb0 con 0x7fa280103600 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa280111d30 con 0x7fa28010c0b0 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa287d8d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa280112030 con 0x7fa280102ce0 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa285328640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280103600 0x7fa2801989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa285328640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280103600 0x7fa2801989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60434/0 (socket says 172.21.15.28:60434) 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa285328640 1 -- 172.21.15.28:0/861038482 learned_addr learned my addr 172.21.15.28:0/861038482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa285b29640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280102ce0 0x7fa280198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa28632a640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa28010c0b0 0x7fa28019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa285328640 1 -- 172.21.15.28:0/861038482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280102ce0 msgr2=0x7fa280198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.926+0000 7fa285328640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280102ce0 0x7fa280198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa285328640 1 -- 172.21.15.28:0/861038482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa28010c0b0 msgr2=0x7fa28019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa285328640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa28010c0b0 0x7fa28019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa285328640 1 -- 172.21.15.28:0/861038482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28019d470 con 0x7fa280103600 2024-09-15T22:07:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa285b29640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280102ce0 0x7fa280198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:07:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa285328640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280103600 0x7fa2801989c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa27400c970 tx=0x7fa27400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa272ffd640 1 -- 172.21.15.28:0/861038482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa274007bb0 con 0x7fa280103600 2024-09-15T22:07:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa272ffd640 1 -- 172.21.15.28:0/861038482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa274007d50 con 0x7fa280103600 2024-09-15T22:07:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa28019d760 con 0x7fa280103600 2024-09-15T22:07:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa272ffd640 1 -- 172.21.15.28:0/861038482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27400f450 con 0x7fa280103600 2024-09-15T22:07:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.927+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2801a4ff0 con 0x7fa280103600 2024-09-15T22:07:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.928+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2801030c0 con 0x7fa280103600 2024-09-15T22:07:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.929+0000 7fa272ffd640 1 -- 172.21.15.28:0/861038482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fa274005020 con 0x7fa280103600 2024-09-15T22:07:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.930+0000 7fa272ffd640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa26c0a3440 0x7fa26c0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.930+0000 7fa285b29640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa26c0a3440 0x7fa26c0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.930+0000 7fa272ffd640 1 -- 172.21.15.28:0/861038482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fa2740cfef0 con 0x7fa280103600 2024-09-15T22:07:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.931+0000 7fa285b29640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa26c0a3440 0x7fa26c0a5900 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa27c005ab0 tx=0x7fa27c005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:35.935+0000 7fa272ffd640 1 -- 172.21.15.28:0/861038482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa274014030 con 0x7fa280103600 2024-09-15T22:07:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.112+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa28002d7c0 con 0x7fa280103600 2024-09-15T22:07:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.113+0000 7fa272ffd640 1 -- 172.21.15.28:0/861038482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) ==== 100+0+11 (secure 0 0 0) 0x7fa27409c520 con 0x7fa280103600 2024-09-15T22:07:36.115 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa26c0a3440 msgr2=0x7fa26c0a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa26c0a3440 0x7fa26c0a5900 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa27c005ab0 tx=0x7fa27c005a20 comp rx=0 tx=0).stop 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280103600 msgr2=0x7fa2801989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280103600 0x7fa2801989c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa27400c970 tx=0x7fa27400ce30 comp rx=0 tx=0).stop 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 shutdown_connections 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa26c0a3440 0x7fa26c0a5900 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa28010c0b0 0x7fa28019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa280103600 0x7fa2801989c0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 --2- 172.21.15.28:0/861038482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa280102ce0 0x7fa280198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 >> 172.21.15.28:0/861038482 conn(0x7fa2800fc690 msgr2=0x7fa2800fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 shutdown_connections 2024-09-15T22:07:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.115+0000 7fa287d8d640 1 -- 172.21.15.28:0/861038482 wait complete. 2024-09-15T22:07:36.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:36 smithi028 ceph-mon[30969]: pgmap v316: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:36.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:36 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/4017215037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T22:07:36.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:36 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/679873804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T22:07:36.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:36 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/544137482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T22:07:36.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-15T22:07:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2049643913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941009b0 msgr2=0x7f1b94100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2049643913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941009b0 0x7f1b94100e10 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f1b88009900 tx=0x7f1b8802a410 comp rx=0 tx=0).stop 2024-09-15T22:07:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2049643913 shutdown_connections 2024-09-15T22:07:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2049643913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b94101350 0x7f1b9410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2049643913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941009b0 0x7f1b94100e10 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2049643913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b941069c0 0x7f1b94106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2049643913 >> 172.21.15.28:0/2049643913 conn(0x7f1b940fc690 msgr2=0x7f1b940feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2049643913 shutdown_connections 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.233+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2049643913 wait complete. 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 Processor -- start 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 -- start start 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b941009b0 0x7f1b9410e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b94101350 0x7f1b9410e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941069c0 0x7f1b9410ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94111ed0 con 0x7f1b941069c0 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94111d50 con 0x7f1b941009b0 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9ab9f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94112050 con 0x7f1b94101350 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941069c0 0x7f1b9410ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941069c0 0x7f1b9410ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60446/0 (socket says 172.21.15.28:60446) 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 -- 172.21.15.28:0/2452735664 learned_addr learned my addr 172.21.15.28:0/2452735664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9893b640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b941009b0 0x7f1b9410e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b93fff640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b94101350 0x7f1b9410e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 -- 172.21.15.28:0/2452735664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b94101350 msgr2=0x7f1b9410e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b94101350 0x7f1b9410e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 -- 172.21.15.28:0/2452735664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b941009b0 msgr2=0x7f1b9410e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b941009b0 0x7f1b9410e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9913c640 1 -- 172.21.15.28:0/2452735664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b941a6960 con 0x7f1b941069c0 2024-09-15T22:07:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b9893b640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b941009b0 0x7f1b9410e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:07:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.234+0000 7f1b93fff640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b94101350 0x7f1b9410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:07:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.235+0000 7f1b9913c640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941069c0 0x7f1b9410ea90 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00eb10 tx=0x7f1b8c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.235+0000 7f1b91ffb640 1 -- 172.21.15.28:0/2452735664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c004b20 con 0x7f1b941069c0 2024-09-15T22:07:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.235+0000 7f1b91ffb640 1 -- 172.21.15.28:0/2452735664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b8c004cc0 con 0x7f1b941069c0 2024-09-15T22:07:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.235+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b941a6c50 con 0x7f1b941069c0 2024-09-15T22:07:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.235+0000 7f1b91ffb640 1 -- 172.21.15.28:0/2452735664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c010760 con 0x7f1b941069c0 2024-09-15T22:07:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.235+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b941a71a0 con 0x7f1b941069c0 2024-09-15T22:07:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.236+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b94100e10 con 0x7f1b941069c0 2024-09-15T22:07:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.237+0000 7f1b91ffb640 1 -- 172.21.15.28:0/2452735664 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f1b8c005020 con 0x7f1b941069c0 2024-09-15T22:07:36.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.238+0000 7f1b91ffb640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1b7c0a3440 0x7f1b7c0a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.238+0000 7f1b91ffb640 1 -- 172.21.15.28:0/2452735664 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f1b8c0d0790 con 0x7f1b941069c0 2024-09-15T22:07:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.238+0000 7f1b9893b640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1b7c0a3440 0x7f1b7c0a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.238+0000 7f1b9893b640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1b7c0a3440 0x7f1b7c0a5900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005fd0 tx=0x7f1b840094d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:36.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.242+0000 7f1b91ffb640 1 -- 172.21.15.28:0/2452735664 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1b8c09cd60 con 0x7f1b941069c0 2024-09-15T22:07:36.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.416+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1b94108f40 con 0x7f1b941069c0 2024-09-15T22:07:36.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.417+0000 7f1b91ffb640 1 -- 172.21.15.28:0/2452735664 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v99) ==== 103+0+11 (secure 0 0 0) 0x7f1b8c09c660 con 0x7f1b941069c0 2024-09-15T22:07:36.419 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1b7c0a3440 msgr2=0x7f1b7c0a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1b7c0a3440 0x7f1b7c0a5900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005fd0 tx=0x7f1b840094d0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941069c0 msgr2=0x7f1b9410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941069c0 0x7f1b9410ea90 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c00eb10 tx=0x7f1b8c00c2d0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 shutdown_connections 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f1b7c0a3440 0x7f1b7c0a5900 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b941069c0 0x7f1b9410ea90 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b94101350 0x7f1b9410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 --2- 172.21.15.28:0/2452735664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b941009b0 0x7f1b9410e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 >> 172.21.15.28:0/2452735664 conn(0x7f1b940fc690 msgr2=0x7f1b940fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:36.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 shutdown_connections 2024-09-15T22:07:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.419+0000 7f1b9ab9f640 1 -- 172.21.15.28:0/2452735664 wait complete. 2024-09-15T22:07:36.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-15T22:07:36.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:36 smithi203 ceph-mon[35546]: pgmap v316: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:36.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:36 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/4017215037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T22:07:36.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:36 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/679873804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T22:07:36.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:36 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/544137482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T22:07:36.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:36 smithi174 ceph-mon[35244]: pgmap v316: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:36.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:36 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/4017215037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T22:07:36.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:36 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/679873804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T22:07:36.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:36 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/544137482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T22:07:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.533+0000 7fa309352640 1 -- 172.21.15.28:0/881548297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 msgr2=0x7fa304106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.533+0000 7fa309352640 1 --2- 172.21.15.28:0/881548297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa304106d70 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009cd0 tx=0x7fa2f402a8e0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 -- 172.21.15.28:0/881548297 shutdown_connections 2024-09-15T22:07:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 --2- 172.21.15.28:0/881548297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa304101320 0x7fa30410d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 --2- 172.21.15.28:0/881548297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa304100980 0x7fa304100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 --2- 172.21.15.28:0/881548297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa304106d70 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 -- 172.21.15.28:0/881548297 >> 172.21.15.28:0/881548297 conn(0x7fa304077a10 msgr2=0x7fa3040fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 -- 172.21.15.28:0/881548297 shutdown_connections 2024-09-15T22:07:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 -- 172.21.15.28:0/881548297 wait complete. 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 Processor -- start 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.534+0000 7fa309352640 1 -- start start 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa304100980 0x7fa304198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa304101320 0x7fa304198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa30419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa304110dd0 con 0x7fa304106990 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa304110c50 con 0x7fa304101320 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa304110f50 con 0x7fa304100980 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa30419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa30419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60460/0 (socket says 172.21.15.28:60460) 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 -- 172.21.15.28:0/2592316064 learned_addr learned my addr 172.21.15.28:0/2592316064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa302ffd640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa304100980 0x7fa304198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3027fc640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa304101320 0x7fa304198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 -- 172.21.15.28:0/2592316064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa304100980 msgr2=0x7fa304198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa304100980 0x7fa304198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 -- 172.21.15.28:0/2592316064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa304101320 msgr2=0x7fa304198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa304101320 0x7fa304198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 -- 172.21.15.28:0/2592316064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30419d410 con 0x7fa304106990 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa3037fe640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa30419cce0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00d750 tx=0x7fa2fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa2fbfff640 1 -- 172.21.15.28:0/2592316064 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc017070 con 0x7fa304106990 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa2fbfff640 1 -- 172.21.15.28:0/2592316064 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2fc005500 con 0x7fa304106990 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa2fbfff640 1 -- 172.21.15.28:0/2592316064 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc004d90 con 0x7fa304106990 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30419d700 con 0x7fa304106990 2024-09-15T22:07:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.535+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3041a4f90 con 0x7fa304106990 2024-09-15T22:07:36.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.536+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa304100de0 con 0x7fa304106990 2024-09-15T22:07:36.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.538+0000 7fa2fbfff640 1 -- 172.21.15.28:0/2592316064 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fa2fc004f30 con 0x7fa304106990 2024-09-15T22:07:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.538+0000 7fa2fbfff640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa2f00a33c0 0x7fa2f00a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:07:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.538+0000 7fa2fbfff640 1 -- 172.21.15.28:0/2592316064 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fa2fc0cfe10 con 0x7fa304106990 2024-09-15T22:07:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.538+0000 7fa302ffd640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa2f00a33c0 0x7fa2f00a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:07:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.539+0000 7fa302ffd640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa2f00a33c0 0x7fa2f00a5880 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4006010 tx=0x7fa2f4009360 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:07:36.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.541+0000 7fa2fbfff640 1 -- 172.21.15.28:0/2592316064 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa2fc09c440 con 0x7fa304106990 2024-09-15T22:07:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.766+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fa3041a5230 con 0x7fa304106990 2024-09-15T22:07:36.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.774+0000 7fa2fbfff640 1 -- 172.21.15.28:0/2592316064 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+742869 (secure 0 0 0) 0x7fa2fc09bd40 con 0x7fa304106990 2024-09-15T22:07:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:07:36.787 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:07:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 25, 2024-09-15T22:07:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 15330, 2024-09-15T22:07:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-09-15T22:07:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1380427157 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1380427157 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/1380427157", 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-09-15T21:57:20.180772+0000", 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 24964, 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-15T22:07:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-15T22:07:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T22:07:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-15T22:07:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-15T22:07:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-15T22:07:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-15T22:07:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-15T22:07:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-15T22:07:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T22:07:36.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-15T22:07:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-15T22:07:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-15T22:07:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-15T22:07:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-15T22:07:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-15T22:07:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-15T22:07:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T22:07:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T22:07:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T22:07:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-15T22:07:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-15T22:07:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-15T22:07:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T22:07:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T22:07:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-15T22:07:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T22:07:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T22:07:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T22:07:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T22:07:36.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T22:07:36.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:36.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T22:07:36.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.955 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:36.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.962 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:36.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-15T22:07:36.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:36.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-15T22:07:36.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.980 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T22:07:36.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:36.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:36.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:36.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:36.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-15T22:07:37.002 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-15T22:07:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T22:07:37.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-15T22:07:37.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.020 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.023 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.028 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.029 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.031 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.034 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.037 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-15T22:07:37.039 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T22:07:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-15T22:07:37.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T22:07:37.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.052 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.060 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.063 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-15T22:07:37.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.070 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.076 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.080 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.081 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.086 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-15T22:07:37.088 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.092 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-15T22:07:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.099 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.102 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.102 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.102 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.104 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.104 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.104 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.104 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.129 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-15T22:07:37.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.132 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.138 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T22:07:37.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-15T22:07:37.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-15T22:07:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-15T22:07:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.187 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.189 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.194 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.195 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.203 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-15T22:07:37.205 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-15T22:07:37.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.222 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-15T22:07:37.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-15T22:07:37.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.233 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.234 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-15T22:07:37.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-15T22:07:37.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.240 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.242 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 24986, 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-09-15T22:07:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.258 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-15T22:07:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-15T22:07:37.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-15T22:07:37.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T22:07:37.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.287 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T22:07:37.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T22:07:37.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-15T22:07:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-15T22:07:37.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.310 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T22:07:37.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-15T22:07:37.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-15T22:07:37.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-15T22:07:37.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-15T22:07:37.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.334 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-15T22:07:37.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.336 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.338 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T22:07:37.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.347 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T22:07:37.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-15T22:07:37.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-15T22:07:37.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-15T22:07:37.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.355 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T22:07:37.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.359 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-15T22:07:37.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-15T22:07:37.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.360 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T22:07:37.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-15T22:07:37.364 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.365 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.368 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.370 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T22:07:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T22:07:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.383 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T22:07:37.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T22:07:37.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T22:07:37.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T22:07:37.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T22:07:37.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T22:07:37.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-15T22:07:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.426 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.427 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.428 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.429 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.432 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-15T22:07:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-15T22:07:37.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.441 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.450 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-15T22:07:37.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.462 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.463 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-15T22:07:37.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.486 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.489 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-15T22:07:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-15T22:07:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-15T22:07:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T22:07:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-15T22:07:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-15T22:07:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-15T22:07:37.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.574 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.575 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.582 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-15T22:07:37.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.589 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.591 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-15T22:07:37.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-15T22:07:37.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-15T22:07:37.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.601 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.602 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.605 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-15T22:07:37.622 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T22:07:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.631 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.632 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.634 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.635 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-15T22:07:37.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-15T22:07:37.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-15T22:07:37.657 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.675 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.676 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.683 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.684 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.686 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T22:07:37.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-15T22:07:37.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-15T22:07:37.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-15T22:07:37.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.702 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.719 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.720 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.723 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.735 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.737 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-15T22:07:37.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-15T22:07:37.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.741 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.743 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-09-15T22:07:37.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-09-15T22:07:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-09-15T22:07:37.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-09-15T22:07:37.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.757 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-09-15T22:07:37.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-09-15T22:07:37.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.762 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T22:07:37.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T22:07:37.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T22:07:37.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-09-15T22:07:37.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.772 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-09-15T22:07:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-15T22:07:37.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-09-15T22:07:37.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-09-15T22:07:37.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T22:07:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T22:07:37.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T22:07:37.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-09-15T22:07:37.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-09-15T22:07:37.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-09-15T22:07:37.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.803 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T22:07:37.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-09-15T22:07:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:37.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-09-15T22:07:37.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.824 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-09-15T22:07:37.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.828 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T22:07:37.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.831 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-09-15T22:07:37.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T22:07:37.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-09-15T22:07:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-09-15T22:07:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-09-15T22:07:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-09-15T22:07:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T22:07:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-09-15T22:07:37.860 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T22:07:37.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T22:07:37.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.874 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T22:07:37.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.881 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T22:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T22:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.890 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T22:07:37.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.897 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T22:07:37.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T22:07:37.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.906 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-15T22:07:37.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.915 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.920 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-15T22:07:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.928 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-15T22:07:37.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-15T22:07:37.933 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-09-15T22:07:37.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-09-15T22:07:37.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.946 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.950 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.953 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-09-15T22:07:37.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:37.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-09-15T22:07:37.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.961 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.964 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.967 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-09-15T22:07:37.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-09-15T22:07:37.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.971 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.977 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.979 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.980 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.987 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-09-15T22:07:37.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.992 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:37.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:37.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:37.995 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:37.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:37.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.002 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T22:07:38.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-09-15T22:07:38.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-09-15T22:07:38.010 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.011 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.013 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.014 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.017 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.019 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.020 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.026 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.027 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.033 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-09-15T22:07:38.036 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.037 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.038 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.041 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.043 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.044 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.049 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.050 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.053 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-09-15T22:07:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-09-15T22:07:38.057 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T22:07:38.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T22:07:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T22:07:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-15T22:07:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-09-15T22:07:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T22:07:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-09-15T22:07:38.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.097 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.100 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-09-15T22:07:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-09-15T22:07:38.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.113 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.125 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-09-15T22:07:38.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.128 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-09-15T22:07:38.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-09-15T22:07:38.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.142 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.143 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.144 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.149 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.150 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.153 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-09-15T22:07:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.156 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-09-15T22:07:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-09-15T22:07:38.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.188 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-09-15T22:07:38.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.214 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.215 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-09-15T22:07:38.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-09-15T22:07:38.218 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-09-15T22:07:38.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.221 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-09-15T22:07:38.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-15T22:07:38.224 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-15T22:07:38.225 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-15T22:07:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-15T22:07:38.227 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-09-15T22:07:38.228 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 93, 2024-09-15T22:07:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2467107481 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-09-15T22:07:38.230 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2386469283 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-15T22:07:38.231 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2401473811 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 507294006 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 7538259 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-09-15T22:07:38.233 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3891765874 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-15T22:07:38.234 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:07:38.235 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:37 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/861038482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T22:07:38.236 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:37 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2452735664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T22:07:38.236 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:37 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2592316064' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T22:07:38.236 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:37 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/861038482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T22:07:38.236 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:37 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2452735664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T22:07:38.236 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:37 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2592316064' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa2f00a33c0 msgr2=0x7fa2f00a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa2f00a33c0 0x7fa2f00a5880 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4006010 tx=0x7fa2f4009360 comp rx=0 tx=0).stop 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 msgr2=0x7fa30419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa30419cce0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00d750 tx=0x7fa2fc00dc10 comp rx=0 tx=0).stop 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 shutdown_connections 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fa2f00a33c0 0x7fa2f00a5880 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa304106990 0x7fa30419cce0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa304101320 0x7fa304198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 --2- 172.21.15.28:0/2592316064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa304100980 0x7fa304198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:07:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 >> 172.21.15.28:0/2592316064 conn(0x7fa304077a10 msgr2=0x7fa304069540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:07:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 shutdown_connections 2024-09-15T22:07:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:07:36.780+0000 7fa309352640 1 -- 172.21.15.28:0/2592316064 wait complete. 2024-09-15T22:07:38.244 DEBUG:tasks.check_counter:Getting stats from a 2024-09-15T22:07:38.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.a perf dump 2024-09-15T22:07:38.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:37 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/861038482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T22:07:38.244 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:37 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2452735664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T22:07:38.245 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:37 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2592316064' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T22:07:38.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:38 smithi028 ceph-mon[30969]: pgmap v317: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-15T22:07:38.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:38 smithi203 ceph-mon[35546]: pgmap v317: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-15T22:07:38.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:38 smithi174 ceph-mon[35244]: pgmap v317: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-15T22:07:38.513 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:39.043 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:07:39.043 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:39.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1411, 2024-09-15T22:07:39.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1411, 2024-09-15T22:07:39.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 510688, 2024-09-15T22:07:39.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 447875, 2024-09-15T22:07:39.043 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.149758051, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.049396195, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.037771161, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.050699603, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1410, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.066858011, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000047417 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:39.044 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 623, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 780, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 153738, 2024-09-15T22:07:39.045 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 632442, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.072946143, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.024373501, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.016409068, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.024494053, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 777, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.043444910, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000055913 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 434896 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 243, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 225, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1526012, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 130990, 2024-09-15T22:07:39.047 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 3, 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.039783445, 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.015141032, 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.036807931, 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000022461, 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 224, 2024-09-15T22:07:39.048 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.013803417, 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000061622 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1525744, 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 130672 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-15T22:07:39.049 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3298, 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.094245310, 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000028576 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-15T22:07:39.050 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.051 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 5, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 5, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 5, 2024-09-15T22:07:39.052 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-15T22:07:39.053 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-09-15T22:07:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:39.056 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-09-15T22:07:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 43, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 2, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-15T22:07:39.058 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 4281929, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 4279042, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 43 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 4, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 26, 2024-09-15T22:07:39.059 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 22, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 2, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 17, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 15, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 5, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 33, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 28, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 49544, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215328 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.061 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.062 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.064 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-15T22:07:39.065 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.066 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.066 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.066 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-15T22:07:39.066 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.066 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.066 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.067 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:39.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.071 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-15T22:07:39.072 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-09-15T22:07:39.073 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 3, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 2, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 2740, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:39.074 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:39.076 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 189410, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 52, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-15T22:07:39.077 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 192, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 17248, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 52, 2024-09-15T22:07:39.078 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 2029, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 2029, 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-15T22:07:39.079 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2029, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 23.804337871, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.011732054 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2029, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2030 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 2029, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 2029, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-15T22:07:39.080 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 1315, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 712, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:39.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-15T22:07:39.082 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-15T22:07:39.083 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 99, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 11, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 5, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 5, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-15T22:07:39.084 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-09-15T22:07:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 2273, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1970456, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 2273, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:39.086 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2273, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1970456, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:39.087 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 2029, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 87630, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2029, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 87630, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 2029, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 2029, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2029, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2029, 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:39.089 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:39.090 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x555a8d1d6b40": { 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-15T22:07:39.091 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:39.093 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:07:39.822 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-09-15T22:07:39.823 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-09-15T22:07:39.823 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-09-15T22:07:39.823 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:39.823 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:39.823 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-09-15T22:07:39.823 DEBUG:tasks.check_counter:Getting stats from d 2024-09-15T22:07:39.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.d perf dump 2024-09-15T22:07:40.082 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:40.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:40 smithi028 ceph-mon[30969]: pgmap v318: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-15T22:07:40.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:40 smithi203 ceph-mon[35546]: pgmap v318: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-15T22:07:40.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:40 smithi174 ceph-mon[35244]: pgmap v318: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-15T22:07:40.595 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:07:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:40.595 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 223, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 213, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1448736, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 131104, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.037561676, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.013921572, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.033752545, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:40.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 212, 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.013443846, 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000063414 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1448736, 2024-09-15T22:07:40.597 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 131104 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 165, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3344, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 186400, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.015608238, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.009385089, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.003861289, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T22:07:40.598 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 163, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.016000954, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000098165 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 186400 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:40.599 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.000326504, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000242861, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:40.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:40.601 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:40.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149017, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:40.604 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:40.605 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 225, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1417286, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 225, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 225, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1417286, 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:40.607 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:07:41.398 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-15T22:07:41.398 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-15T22:07:41.398 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-09-15T22:07:41.399 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-09-15T22:07:41.399 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-09-15T22:07:41.399 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-09-15T22:07:41.399 DEBUG:tasks.check_counter:Getting stats from g 2024-09-15T22:07:41.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.g perf dump 2024-09-15T22:07:41.658 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 742, 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 760, 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1693680, 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 292608, 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-09-15T22:07:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.101359575, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.040080526, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.052626205, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.014514658, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 759, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.038909976, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000051264 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1526352, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 131472 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 904, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1061, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 233406, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 722933, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-09-15T22:07:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.074037646, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.024494327, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.019838341, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.023533614, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1059, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.058548870, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000055286 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 437312 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 6, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1629, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1284, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-09-15T22:07:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001672643, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000072446, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001381097, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000077220, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002710416, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000677604 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2382, 2024-09-15T22:07:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.058849096, 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000024705 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-15T22:07:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-09-15T22:07:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-15T22:07:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T22:07:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-15T22:07:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-09-15T22:07:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 4, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 2, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-15T22:07:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 4195426, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 4195285, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 4 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 3, 2024-09-15T22:07:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 3, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 1, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 1, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 48552, 2024-09-15T22:07:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215328 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-15T22:07:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 0, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 1, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 1, 2024-09-15T22:07:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 1150, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 198504, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 56, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 192, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 8512, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 13, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-15T22:07:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 1400, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 1400, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1400, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 226.322703051, 2024-09-15T22:07:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.161659073 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1400, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1401 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 1400, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 1400, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 894, 2024-09-15T22:07:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 504, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-09-15T22:07:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-15T22:07:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-15T22:07:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 99, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 11, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 6, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 6, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-09-15T22:07:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 1646, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1765059, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 1646, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1646, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1765059, 2024-09-15T22:07:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 1400, 2024-09-15T22:07:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1127, 2024-09-15T22:07:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1400, 2024-09-15T22:07:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1127, 2024-09-15T22:07:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 1400, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1400, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1400, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1400, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55c622066b40": { 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:42.259 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:07:42.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:42 smithi028 ceph-mon[30969]: pgmap v319: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:42.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:42 smithi203 ceph-mon[35546]: pgmap v319: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:42.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:42 smithi174 ceph-mon[35244]: pgmap v319: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-15T22:07:43.037 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-09-15T22:07:43.037 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-09-15T22:07:43.037 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-09-15T22:07:43.037 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:43.037 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:43.037 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-09-15T22:07:43.037 DEBUG:tasks.check_counter:Getting stats from j 2024-09-15T22:07:43.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.j perf dump 2024-09-15T22:07:43.294 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3024, 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-09-15T22:07:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.003260679, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000033518, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002272717, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.004042535, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001347511 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:43.939 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 226, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 216, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1449616, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 132544, 2024-09-15T22:07:43.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.042256598, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.014900931, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.039537124, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 215, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.016470885, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000076608 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:43.941 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1449616, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 132544 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 165, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 186304, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-09-15T22:07:43.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.013346434, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.008919921, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002205321, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 162, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.017165766, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000105961 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:43.943 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 186304 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:43.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:43.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:43.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149011, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 32, 2024-09-15T22:07:43.947 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-09-15T22:07:43.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:43.949 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 228, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1417740, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 228, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 228, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1417740, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-15T22:07:43.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:43.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-15T22:07:43.951 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-15T22:07:44.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:44 smithi028 ceph-mon[30969]: pgmap v320: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-15T22:07:44.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:44 smithi203 ceph-mon[35546]: pgmap v320: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-15T22:07:44.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:44 smithi174 ceph-mon[35244]: pgmap v320: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-15T22:07:44.650 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-15T22:07:44.650 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-15T22:07:44.650 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-09-15T22:07:44.650 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-09-15T22:07:44.650 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-09-15T22:07:44.651 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-09-15T22:07:44.651 DEBUG:tasks.check_counter:Getting stats from b 2024-09-15T22:07:44.651 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.b perf dump 2024-09-15T22:07:44.950 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T22:07:45.763 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 5041, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 5333, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 3133883, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 18444346, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 131, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.488130616, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.155925517, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.215900402, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.111517540, 2024-09-15T22:07:45.764 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5332, 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.416109750, 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000265586 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2367, 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000473924, 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000200 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1669760, 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 500096 2024-09-15T22:07:45.765 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 23593, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 28516, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 8136988, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 24482882, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 19, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 7, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 1.493977234, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.560952192, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 1.391229217, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.164447625, 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 28514, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 8.573034580, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000300660 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 36549, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.007454118, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000203 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 43120, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 140048 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:45.767 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 5861, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 5628, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 2124455, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 25319820, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 28, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.561135203, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.219778526, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.196766981, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.143374457, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5626, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 3.012788402, 2024-09-15T22:07:45.768 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000535511 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3946, 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000755496, 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000191 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 349776, 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 88000 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T22:07:45.769 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5513, 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 7.087284595, 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001285558 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:45.770 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4835, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.235485779, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000048704 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 2965, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 2957, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2957, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 43.093605161, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.014573420 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T22:07:45.771 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 8, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 4, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 2, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 481, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 1, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 1297, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 831, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 455, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 11, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 95, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 15, 2024-09-15T22:07:45.772 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 3, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 4, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 3, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 3887, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 3750, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:45.773 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 51, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 4, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 10, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 4, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 15, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:45.774 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 1308, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 4, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 234, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 7, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 1308, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 320, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 10, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 310, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 988, 2024-09-15T22:07:45.775 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 10, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 1058, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 3773, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 6, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 14841, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 13438, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 147, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 1480 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 2, 2024-09-15T22:07:45.776 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 16, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 83, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 929, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 785, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 61, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:45.777 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 3, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 832, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 10, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 4, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 1, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T22:07:45.778 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 17389, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 17366, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 17366, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 23, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 2, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 45, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 44, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 44, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 1, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 1, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 1, 2024-09-15T22:07:45.779 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 61361271, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 61384236, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2362, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 141.406031404, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.059867075 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 801, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 1605, 2024-09-15T22:07:45.780 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 804, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 108, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 127, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 19, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 1297, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 2384, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 1087, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 4, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 1373, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 1369, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 84160, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 231712 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.781 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 3684, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 9, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 2965, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 43, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 26, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 765, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.761346060, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000995223 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 176, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.032026315, 2024-09-15T22:07:45.782 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000181967 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 61, 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.045653612, 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000748419 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T22:07:45.783 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 360, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.128434013, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000356761 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000232340, 2024-09-15T22:07:45.784 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000116170 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.785 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.786 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.786 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.786 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T22:07:45.786 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 10, 2024-09-15T22:07:45.786 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.001438972, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000143897 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 105, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.089299535, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000850471 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.787 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 10, 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 28.699494851, 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 2.869949485 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 176, 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.610949787, 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.003471305 2024-09-15T22:07:45.788 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 246, 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.859847786, 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.003495316 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 113, 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 9.657312884, 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.085462945 2024-09-15T22:07:45.789 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 99, 2024-09-15T22:07:45.790 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.035917161, 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.010463809 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3, 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.013465624, 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.004488541 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.791 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 8, 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.047146409, 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.005893301 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.792 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.001989423, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000497355 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 9, 2024-09-15T22:07:45.793 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.021857638, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002428626 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 13, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.007334837, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000564218 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 797, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.079857914, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001354903 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.794 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 7, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 5, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 2, 2024-09-15T22:07:45.795 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 178, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 89, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 603, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:45.796 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:45.797 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T22:07:45.798 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 999339, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 684, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:45.799 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 192, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 3092440, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 44442, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T22:07:45.800 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 11285, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 60139189, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 37, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 11285, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 11285, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 973.703181808, 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.086282958 2024-09-15T22:07:45.801 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 11285, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 14554 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 11285, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 1354, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 9931, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 372, 2024-09-15T22:07:45.802 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 1328, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 30, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 2382, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 114, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 1, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 5437, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 1324, 2024-09-15T22:07:45.803 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 2130, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T22:07:45.804 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 6, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T22:07:45.805 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 99, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T22:07:45.806 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 17, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 12, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 12, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 1162, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 12, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 256 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-09-15T22:07:45.807 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 836 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:45.808 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 52, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 7, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 4899, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 785, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 34491, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 10221118, 2024-09-15T22:07:45.809 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 34491, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 34491, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 10221118, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.810 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 11285, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 60238922, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 11285, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 60238922, 2024-09-15T22:07:45.811 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:45.812 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 11285, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 11285, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 11285, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 11285, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:45.813 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 785, 2024-09-15T22:07:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 99805, 2024-09-15T22:07:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 785, 2024-09-15T22:07:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:45.814 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 20, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 99805, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55f8d4990960": { 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:45.815 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 17389, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 57189932, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 17389, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2362, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 57189932, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:45.816 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.817 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:45.817 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T22:07:46.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:46 smithi028 ceph-mon[30969]: pgmap v321: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-15T22:07:46.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:46 smithi203 ceph-mon[35546]: pgmap v321: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-15T22:07:46.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:46 smithi174 ceph-mon[35244]: pgmap v321: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-15T22:07:47.122 INFO:tasks.check_counter:Daemon mds.b mds.exported=4 2024-09-15T22:07:47.122 INFO:tasks.check_counter:Daemon mds.b mds.imported=4 2024-09-15T22:07:47.122 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-09-15T22:07:47.122 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-09-15T22:07:47.122 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=3 2024-09-15T22:07:47.123 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-09-15T22:07:47.123 DEBUG:tasks.check_counter:Getting stats from e 2024-09-15T22:07:47.123 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.e perf dump 2024-09-15T22:07:47.388 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T22:07:47.961 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1230, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1210, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 22808617, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 445652, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.142211204, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.049268475, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.073140226, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.025595435, 2024-09-15T22:07:47.962 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1210, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.076133372, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000062920 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1530896, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 135280 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 323, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 501, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 16863101, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 544099, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 8, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.053165752, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.014363500, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.021896153, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.012731604, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 498, 2024-09-15T22:07:47.964 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.035534467, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000071354 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 440800 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 727, 2024-09-15T22:07:47.965 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 729, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 19625472, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 231193, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.070334181, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.015131296, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.030430577, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.019970762, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 727, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.028189203, 2024-09-15T22:07:47.966 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000038774 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T22:07:47.967 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3359, 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.098377680, 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000029287 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:47.968 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.969 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 2265, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 1458, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 807, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 90, 2024-09-15T22:07:47.970 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 2, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T22:07:47.971 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 4, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 234, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 7, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:47.972 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-09-15T22:07:47.973 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:47.974 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-09-15T22:07:47.975 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 17389, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-09-15T22:07:47.976 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 24, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 61361271, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 61384236, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 61384100, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.977 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 17389 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 798, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 1588, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 790, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 105, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 122, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 17, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 2265, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 2323, 2024-09-15T22:07:47.978 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 58, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 55780, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215328 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T22:07:47.979 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.980 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.981 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.982 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.983 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.984 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.985 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.986 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.987 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:47.988 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.989 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T22:07:47.990 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.991 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 7, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 5, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 5, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 3822, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T22:07:47.992 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:47.993 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:47.994 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 250563, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 1022, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T22:07:47.995 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 192, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 3795623, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 50249, 2024-09-15T22:07:47.996 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2024, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 26, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 4, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2024, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2024, 2024-09-15T22:07:47.997 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 120.971279301, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.059768418 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2024, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2025 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2024, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2024, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T22:07:47.998 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1241, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 780, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:47.999 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 2, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-09-15T22:07:48.000 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T22:07:48.001 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 99, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 11, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 8, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 8, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-09-15T22:07:48.002 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T22:07:48.003 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2276, 2024-09-15T22:07:48.004 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 59076528, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2276, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2276, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 59076528, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.005 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2024, 2024-09-15T22:07:48.006 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 57201042, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2024, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 57201042, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:48.007 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2024, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2024, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2024, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2024, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:48.008 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T22:07:48.009 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x56055fa50b40": { 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:48.010 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:48.011 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T22:07:48.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:48 smithi174 ceph-mon[35244]: pgmap v322: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-15T22:07:48.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:48 smithi028 ceph-mon[30969]: pgmap v322: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-15T22:07:48.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:48 smithi203 ceph-mon[35546]: pgmap v322: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-15T22:07:48.781 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-15T22:07:48.781 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-15T22:07:48.781 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=7 2024-09-15T22:07:48.781 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:48.781 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:48.781 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-09-15T22:07:48.781 DEBUG:tasks.check_counter:Getting stats from h 2024-09-15T22:07:48.781 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.h perf dump 2024-09-15T22:07:49.030 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T22:07:49.582 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T22:07:49.582 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:49.582 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1459, 2024-09-15T22:07:49.582 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1053, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 477368, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 601911, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 20, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.111856956, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.045509174, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.068549995, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.006956346, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1054, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.071457444, 2024-09-15T22:07:49.583 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000067796 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2291, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000403818, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000176 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 80, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 143, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 19915, 2024-09-15T22:07:49.584 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 332120, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 24, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.018974735, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.005137084, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.012109438, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000549560, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 140, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.030317407, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000216552 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.585 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 109, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000018775, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000172 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 127280 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 323, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 431, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1556248, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 501324, 2024-09-15T22:07:49.586 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 10, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 6, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.052447747, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.020699094, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.043874343, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000895740, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 428, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.029539787, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000069018 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 13, 2024-09-15T22:07:49.587 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000001703, 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000131 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1549040, 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 458736 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:49.588 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 121, 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.011639080, 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000096190 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 18, 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000833106, 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000046283 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.589 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 21, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 12, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 12, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.007955279, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000662939 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 9, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 11, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-09-15T22:07:49.590 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 11, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 0, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 1, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 10, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 10, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 33, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 2, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 48, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 23, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 8, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 6, 2024-09-15T22:07:49.591 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 3, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 12, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 3, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 8, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 12, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 4, 2024-09-15T22:07:49.592 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 234, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 7, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 12, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 12, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 10, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 7, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:49.593 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 10, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 2, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 9 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 5, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:49.594 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 4, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 4, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 3, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 36, 2024-09-15T22:07:49.595 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 3, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 43, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 43, 2024-09-15T22:07:49.596 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 2, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4281929, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 28, 2024-09-15T22:07:49.597 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.770073728, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.027502633 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 15, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 36, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 21, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 13, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 28, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 15, 2024-09-15T22:07:49.598 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 11, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 48, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 37, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 15, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 15, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 52960, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223520 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 32, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 3, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 21, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 16, 2024-09-15T22:07:49.599 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 13, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000184867, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000184867 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.600 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1, 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000279862, 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000279862 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:49.601 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.602 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.603 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T22:07:49.604 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.605 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 6, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.001423914, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000237319 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.606 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.607 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.608 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.609 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T22:07:49.610 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4, 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.006066636, 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001516659 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.611 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.612 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.612 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T22:07:49.612 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-09-15T22:07:49.612 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 3, 2024-09-15T22:07:49.612 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 2, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 1, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 8, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 8, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 626, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:49.613 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:49.614 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:49.615 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 256103, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 90, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:49.616 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 192, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 61200, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 238, 2024-09-15T22:07:49.617 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 105, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 101424, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 33, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 105, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 105, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.797974471, 2024-09-15T22:07:49.618 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.017123566 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 105, 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 134 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 105, 2024-09-15T22:07:49.619 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 13, 2024-09-15T22:07:49.620 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 92, 2024-09-15T22:07:49.620 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T22:07:49.620 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T22:07:49.620 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-09-15T22:07:49.620 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-09-15T22:07:49.620 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 1, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 29, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 13, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 20, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:49.621 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 12, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T22:07:49.622 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T22:07:49.623 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 99, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 11, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 10, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 10, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T22:07:49.624 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 25, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 4 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 39 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.625 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 8, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 4, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 8, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 4, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 1858, 2024-09-15T22:07:49.626 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1868348, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 1858, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 1858, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1868348, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 105, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 102052, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 105, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 102052, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.628 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 105, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 105, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 105, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 105, 2024-09-15T22:07:49.629 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 4, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 628, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 4, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:49.630 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 1, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 628, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x5583468ea960": { 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:49.631 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 43, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 87625, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 43, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 28, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 87625, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:49.632 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:49.633 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.633 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:49.633 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T22:07:50.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:50 smithi174 ceph-mon[35244]: pgmap v323: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-15T22:07:50.331 INFO:tasks.check_counter:Daemon mds.h mds.exported=3 2024-09-15T22:07:50.331 INFO:tasks.check_counter:Daemon mds.h mds.imported=3 2024-09-15T22:07:50.331 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=7 2024-09-15T22:07:50.331 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:50.331 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:50.331 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-09-15T22:07:50.331 DEBUG:tasks.check_counter:Getting stats from k 2024-09-15T22:07:50.332 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.k perf dump 2024-09-15T22:07:50.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:50 smithi028 ceph-mon[30969]: pgmap v323: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-15T22:07:50.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:50 smithi203 ceph-mon[35546]: pgmap v323: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-15T22:07:50.580 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2101, 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 2101, 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 598932, 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 667301, 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-09-15T22:07:51.131 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.158531770, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.048197974, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.042097858, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.056643768, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2100, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.068905852, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000032812 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.132 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 4, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 169, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 876, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 441850, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T22:07:51.133 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.010877225, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.006724922, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.002210536, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000054032, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 166, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.014767077, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000088958 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 441152 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 254, 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 236, 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1529068, 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 137550, 2024-09-15T22:07:51.135 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 3, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.043154347, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.015416120, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.041873632, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000017709, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 235, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.017825062, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000075851 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1528800, 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 137232 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3501, 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.084465916, 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000024126 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.138 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-09-15T22:07:51.139 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:51.140 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 1, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-09-15T22:07:51.141 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:51.142 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:51.143 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-09-15T22:07:51.144 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-09-15T22:07:51.145 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 10, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-09-15T22:07:51.146 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4197195, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 4197054, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.147 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 10 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 3, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 3, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 0, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 1, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 1, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 0, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 0, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 0, 2024-09-15T22:07:51.148 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 48048, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215328 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T22:07:51.149 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.151 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:51.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.153 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.154 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.155 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.156 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.157 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.158 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.159 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.160 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-09-15T22:07:51.161 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 5, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 3, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 4, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 3187, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:51.162 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:51.163 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:51.164 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 182081, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 48, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 192, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 8512, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 13, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T22:07:51.166 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2100, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 21, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 3, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2100, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2100, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 19.265726982, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.009174155 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2100, 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2101 2024-09-15T22:07:51.167 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2100, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2100, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1365, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 733, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T22:07:51.168 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 1, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T22:07:51.169 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T22:07:51.170 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 99, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 11, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 4, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 4, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-09-15T22:07:51.171 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:51.172 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2355, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1900942, 2024-09-15T22:07:51.173 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2355, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2355, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1900942, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2100, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2896, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2100, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2896, 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:51.175 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:51.176 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2100, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2100, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2100, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2100, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.177 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:51.178 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55cc862b2b40": { 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T22:07:51.179 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T22:07:51.180 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T22:07:51.932 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-09-15T22:07:51.932 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-09-15T22:07:51.932 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-09-15T22:07:51.932 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:51.933 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:51.933 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-09-15T22:07:51.933 DEBUG:tasks.check_counter:Getting stats from c 2024-09-15T22:07:51.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.c perf dump 2024-09-15T22:07:52.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:52 smithi203 ceph-mon[35546]: pgmap v324: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:52.234 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T22:07:52.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:52 smithi174 ceph-mon[35244]: pgmap v324: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:52.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:52 smithi028 ceph-mon[30969]: pgmap v324: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:52.958 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 75, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 91, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 16614, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 274134, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 27, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.016047412, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.002103358, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.009080156, 2024-09-15T22:07:52.959 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.001377527, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 89, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.015134207, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000170047 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 55, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000009568, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000173 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 2496, 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 5888 2024-09-15T22:07:52.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 44, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 206, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 12308, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 583233, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 20, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.019984464, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.007538595, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.009594616, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000675970, 2024-09-15T22:07:52.961 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 204, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.030012604, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000147120 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 18, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000002743, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000152 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 4272, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 444032 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.962 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1532, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1184, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1958833, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 588120, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.137696728, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.050915353, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.109309444, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.004754282, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1183, 2024-09-15T22:07:52.963 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.056625357, 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000047865 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2066, 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000433102, 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000209 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1552896, 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 142784 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-09-15T22:07:52.964 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 85, 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.005518857, 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000064927 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:52.965 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 16, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000463015, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000028938 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 7, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 3, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002759521, 2024-09-15T22:07:52.966 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000919840 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 4, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 14, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 1, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 2, 2024-09-15T22:07:52.967 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 11, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 11, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 17, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 3, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 17, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 10, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 3, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 1, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-09-15T22:07:52.968 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 1, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 3, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 1, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 2, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 3, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 1, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 7, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 3, 2024-09-15T22:07:52.969 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 3, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 3, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 2, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 3, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 2, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 2 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 2, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:52.971 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 1, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 1, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 3, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 25, 2024-09-15T22:07:52.972 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 1, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 19, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 19, 2024-09-15T22:07:52.973 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4224788, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-09-15T22:07:52.974 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 17, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.128563235, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.007562543 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 18, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 28, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 10, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 17, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 26, 2024-09-15T22:07:52.975 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 9, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 14, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 32, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 18, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 4, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 4, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 53104, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223520 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 10, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 1, 2024-09-15T22:07:52.976 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 7, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 25, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 4, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.977 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.978 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.979 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.980 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.981 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-09-15T22:07:52.981 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.981 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.981 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.981 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.982 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000749942, 2024-09-15T22:07:52.983 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000374971 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.984 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.986 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.987 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-09-15T22:07:52.988 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002009579, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002009579 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-09-15T22:07:52.989 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 4, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 2, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 2, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 2, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 1, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 596, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:52.990 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:52.991 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 255383, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 89, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:52.993 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 192, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 77648, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 280, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-09-15T22:07:52.994 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 79, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 43171, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 32, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 79, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 79, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.894978535, 2024-09-15T22:07:52.995 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.011328842 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 79, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 107 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 79, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 4, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 75, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-09-15T22:07:52.996 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 1, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 18, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 14, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 14, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 3, 2024-09-15T22:07:52.997 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-09-15T22:07:52.998 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-09-15T22:07:52.999 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 99, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 11, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 9, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 9, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 23, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 4 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-09-15T22:07:53.000 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 28 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 2, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 1, 2024-09-15T22:07:53.001 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 2, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 1, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1647, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1821159, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1647, 2024-09-15T22:07:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1647, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1821159, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 79, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 43328, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 79, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 43328, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:53.004 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 79, 2024-09-15T22:07:53.005 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 79, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 79, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 79, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:53.006 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 157, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 157, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:53.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55f57ffd6960": { 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 19, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 30484, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 19, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 17, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 30484, 2024-09-15T22:07:53.008 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:53.009 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:53.009 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:53.009 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:53.009 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:53.009 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:53.009 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-15T22:07:53.975 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-09-15T22:07:53.975 INFO:tasks.check_counter:Daemon mds.c mds.imported=1 2024-09-15T22:07:53.975 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=7 2024-09-15T22:07:53.975 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:53.975 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:53.975 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-09-15T22:07:53.975 DEBUG:tasks.check_counter:Getting stats from f 2024-09-15T22:07:53.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.f perf dump 2024-09-15T22:07:54.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:54 smithi203 ceph-mon[35546]: pgmap v325: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:07:54.231 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T22:07:54.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:54 smithi028 ceph-mon[30969]: pgmap v325: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:07:54.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:54 smithi174 ceph-mon[35244]: pgmap v325: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-09-15T22:07:54.801 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-15T22:07:54.801 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:54.801 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1181, 2024-09-15T22:07:54.801 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 971, 2024-09-15T22:07:54.801 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1842601, 2024-09-15T22:07:54.801 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 326333, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 7, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.115805711, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.041685333, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.089409019, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.004618804, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 970, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.052200295, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000053814 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:54.802 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1529, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000319564, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000209 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1540720, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 140752 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 48, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 211, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 12036, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 747863, 2024-09-15T22:07:54.803 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 33, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.022839399, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.007608709, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.012710005, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000186692, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 209, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.029393223, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000140637 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:54.804 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 43, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000004670, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000108 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 6000, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 444928 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 25, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 27, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 6224, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 58836, 2024-09-15T22:07:54.805 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 10, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.004256315, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.000348594, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.003052647, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000627734, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 25, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.017816759, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000712670 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-09-15T22:07:54.806 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000282, 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000070 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:54.807 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 42, 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001608812, 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000038305 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000010026, 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000005013 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 3, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 3, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000912346, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000304115 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:54.809 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 16, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 16, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 13, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 7, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 3, 2024-09-15T22:07:54.810 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 16, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 11, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 3, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-09-15T22:07:54.811 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 7, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:54.812 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 2, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 2, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-09-15T22:07:54.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 3, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 2, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 5, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-09-15T22:07:54.814 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 3, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 18, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:54.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 4, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 4, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-09-15T22:07:54.816 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4195426, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.004435480, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001478493 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.817 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 20, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 23, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 3, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 19, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 23, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 4, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 16, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 23, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 7, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-09-15T22:07:54.818 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 2, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 2, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 52928, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223520 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 5, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 3, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 4, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.819 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000201987, 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000201987 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000710359, 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000355179 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:54.821 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.822 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-09-15T22:07:54.823 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.824 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.825 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.826 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:54.827 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-09-15T22:07:54.829 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.830 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.831 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 0, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 1, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 0, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 2, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 2, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 85, 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:54.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:54.833 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:54.834 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:54.835 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 246135, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 77, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:54.836 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 192, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 85149, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 335, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-09-15T22:07:54.837 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 41, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 11569, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 41, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 41, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.216994793, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.005292555 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-09-15T22:07:54.838 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 41, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 65 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 41, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 0, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 41, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 0, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 3, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 7, 2024-09-15T22:07:54.839 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 6, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:54.840 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-09-15T22:07:54.841 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 99, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-09-15T22:07:54.842 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 11, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 9, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 9, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 17, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 2 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:54.843 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 21 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-09-15T22:07:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1250, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1730601, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1250, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:54.845 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1250, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1730601, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:54.846 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 41, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 11569, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 41, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 11569, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.847 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 41, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 41, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 41, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 41, 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:54.848 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:54.849 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x555d07474960": { 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:54.850 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 4, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1122, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 4, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 3, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1122, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:54.851 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.852 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:54.852 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-15T22:07:55.734 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-15T22:07:55.734 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-15T22:07:55.734 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=7 2024-09-15T22:07:55.734 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:55.735 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:55.735 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-09-15T22:07:55.735 DEBUG:tasks.check_counter:Getting stats from i 2024-09-15T22:07:55.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.i perf dump 2024-09-15T22:07:55.987 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T22:07:56.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:56 smithi203 ceph-mon[35546]: pgmap v326: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:56.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:56 smithi028 ceph-mon[30969]: pgmap v326: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:56.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:56 smithi174 ceph-mon[35244]: pgmap v326: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:56.761 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-15T22:07:56.761 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:56.761 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 56, 2024-09-15T22:07:56.761 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 217, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 13139, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 392257, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 15, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.023179861, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.008471627, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.009059129, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000692803, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 213, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.028884323, 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000135607 2024-09-15T22:07:56.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 67, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000015811, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000235 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 318464 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1243, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1017, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1867523, 2024-09-15T22:07:56.763 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 809558, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 27, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.129624156, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.043872369, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.104225293, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.005309613, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1018, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.058520033, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000057485 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:56.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1614, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000424710, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000263 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1559024, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 144320 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 40, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 100, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 11907, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 201199, 2024-09-15T22:07:56.765 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 14, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.012763467, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.003155841, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.007635637, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.001093184, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 98, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.013223226, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000134930 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 60, 2024-09-15T22:07:56.766 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000010974, 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000182 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 7648, 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 140176 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:56.767 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 56, 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002937349, 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000052452 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000007913, 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000003956 2024-09-15T22:07:56.768 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 15, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 7, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 7, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001699273, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000242753 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 8, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:56.769 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 16, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 16, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 14, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 6, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 2, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 2, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 3, 2024-09-15T22:07:56.770 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 28, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 20, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 4, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 2, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-09-15T22:07:56.771 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 4, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 234, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 7, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:56.772 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 7, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 3, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 7, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 1 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.773 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 3, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 2, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 4, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-09-15T22:07:56.774 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 3, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 21, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-09-15T22:07:56.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 10, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 10, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 2, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-09-15T22:07:56.776 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4197195, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 9, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.015473584, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001719287 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.777 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 20, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 23, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 3, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 19, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 23, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 4, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 16, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 22, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 6, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 2, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 5, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 3, 2024-09-15T22:07:56.778 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 54216, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223520 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 16, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 15, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 29, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.779 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001389649, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000277929 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.780 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000309624, 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000154812 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-09-15T22:07:56.781 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.782 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.783 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.784 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.785 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-09-15T22:07:56.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-09-15T22:07:56.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:56.790 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 5, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 3, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 2, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 5, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 1, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 614, 2024-09-15T22:07:56.791 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:56.792 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:56.793 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 270215, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 85, 2024-09-15T22:07:56.794 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 192, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-09-15T22:07:56.795 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 83405, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 338, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 52, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 14149, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 32, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 52, 2024-09-15T22:07:56.796 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 52, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.228994485, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004403740 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 52, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 79 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 52, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 0, 2024-09-15T22:07:56.797 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 52, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 9, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 9, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 6, 2024-09-15T22:07:56.798 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-09-15T22:07:56.799 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-09-15T22:07:56.800 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 99, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 11, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 10, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 10, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 23, 2024-09-15T22:07:56.801 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 2 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 24 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1335, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1753478, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1335, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1335, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1753478, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 52, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 14149, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 52, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 14149, 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:56.805 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:56.806 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 52, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 52, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 52, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 52, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:56.807 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.808 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x56287db9c960": { 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 10, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 2891, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 10, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 9, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2891, 2024-09-15T22:07:56.809 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:56.810 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:56.810 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:56.810 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:56.810 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.810 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:56.810 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-15T22:07:57.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:57 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:07:57.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:57 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:07:57.510 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:57 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T22:07:57.649 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-09-15T22:07:57.649 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-09-15T22:07:57.649 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-09-15T22:07:57.649 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:57.649 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:57.649 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-09-15T22:07:57.649 DEBUG:tasks.check_counter:Getting stats from l 2024-09-15T22:07:57.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph daemon mds.l perf dump 2024-09-15T22:07:58.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:58 smithi174 ceph-mon[35244]: pgmap v327: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:58.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:58 smithi028 ceph-mon[30969]: pgmap v327: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:58.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:58 smithi203 ceph-mon[35546]: pgmap v327: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:07:58.549 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T22:07:59.081 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 735, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 737, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 223517, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 233794, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.067549887, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.020335026, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.018747116, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.021373220, 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:59.082 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 735, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.031574257, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000042958 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T22:07:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 258, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 368, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1530018, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 456881, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.047492797, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.019336567, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.041712931, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000069834, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 367, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.031014335, 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000084507 2024-09-15T22:07:59.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1529504, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 456240 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1345, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1382, 2024-09-15T22:07:59.085 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 397714, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 551960, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.144534983, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.039462125, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.039662683, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.052714384, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1380, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.060821064, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000044073 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.086 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 125840 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:59.087 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3469, 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.105700392, 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000030469 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.088 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-09-15T22:07:59.089 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 2, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 2, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 2, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-09-15T22:07:59.090 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-09-15T22:07:59.091 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-09-15T22:07:59.092 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-09-15T22:07:59.093 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-09-15T22:07:59.094 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-09-15T22:07:59.095 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 19, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 4224788, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 4222183, 2024-09-15T22:07:59.096 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 19 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 4, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 18, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 14, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 2, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 15, 2024-09-15T22:07:59.097 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 13, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 2, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 17, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 15, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 48972, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215328 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-09-15T22:07:59.098 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-09-15T22:07:59.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.101 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-09-15T22:07:59.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.103 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.104 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.105 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-09-15T22:07:59.106 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.108 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.109 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.110 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 4, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 2, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 4, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 3168, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T22:07:59.111 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-09-15T22:07:59.112 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-09-15T22:07:59.113 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 190217, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 54, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-09-15T22:07:59.114 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 9512, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 192, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 15136, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 40, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.115 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 2077, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 3, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 2077, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2077, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 21.128493435, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.010172601 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-09-15T22:07:59.116 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2077, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2078 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 2077, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 2077, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 1340, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 735, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-09-15T22:07:59.117 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-09-15T22:07:59.118 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-09-15T22:07:59.119 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 99, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 11, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 5, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 5, 2024-09-15T22:07:59.120 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-09-15T22:07:59.121 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:59.122 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2334, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1925101, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2334, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2334, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1925101, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.123 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2077, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 30489, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2077, 2024-09-15T22:07:59.124 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 30489, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:59.125 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2077, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 2077, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2077, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2077, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:59.126 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55cb6a8feb40": { 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-09-15T22:07:59.128 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-09-15T22:07:59.129 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-09-15T22:07:59.129 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.129 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-15T22:07:59.129 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-15T22:07:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: pgmap v328: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-15T22:07:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: Unable to set osd_memory_target on smithi174 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T22:07:59.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:07:59 smithi203 ceph-mon[35546]: Unable to set osd_memory_target on smithi028 to 185689190: error parsing value: Value '185689190' is below minimum 939524096 2024-09-15T22:07:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: pgmap v328: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-15T22:07:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: Unable to set osd_memory_target on smithi174 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T22:07:59.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:07:59 smithi174 ceph-mon[35244]: Unable to set osd_memory_target on smithi028 to 185689190: error parsing value: Value '185689190' is below minimum 939524096 2024-09-15T22:07:59.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: pgmap v328: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-15T22:07:59.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi028 to 177.0M 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: Unable to set osd_memory_target on smithi174 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T22:07:59.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:07:59 smithi028 ceph-mon[30969]: Unable to set osd_memory_target on smithi028 to 185689190: error parsing value: Value '185689190' is below minimum 939524096 2024-09-15T22:07:59.951 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-09-15T22:07:59.951 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-09-15T22:07:59.951 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-09-15T22:07:59.951 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-09-15T22:07:59.951 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-09-15T22:07:59.952 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-09-15T22:07:59.952 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-15T22:07:59.962 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-15T22:07:59.962 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T22:07:59.962 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T22:07:59.989 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-15T22:07:59.989 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T22:08:00.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T22:08:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.830+0000 7f216bdef640 1 -- 172.21.15.28:0/3457407444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 msgr2=0x7f2164106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.830+0000 7f216bdef640 1 --2- 172.21.15.28:0/3457407444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f2164106e60 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f2154009930 tx=0x7f215402a420 comp rx=0 tx=0).stop 2024-09-15T22:08:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.830+0000 7f216bdef640 1 -- 172.21.15.28:0/3457407444 shutdown_connections 2024-09-15T22:08:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.830+0000 7f216bdef640 1 --2- 172.21.15.28:0/3457407444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2164101420 0x7f216410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.830+0000 7f216bdef640 1 --2- 172.21.15.28:0/3457407444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164100a80 0x7f2164100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.830+0000 7f216bdef640 1 --2- 172.21.15.28:0/3457407444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f2164106e60 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.830+0000 7f216bdef640 1 -- 172.21.15.28:0/3457407444 >> 172.21.15.28:0/3457407444 conn(0x7f21640fc7a0 msgr2=0x7f21640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.831+0000 7f216bdef640 1 -- 172.21.15.28:0/3457407444 shutdown_connections 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.831+0000 7f216bdef640 1 -- 172.21.15.28:0/3457407444 wait complete. 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.831+0000 7f216bdef640 1 Processor -- start 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.831+0000 7f216bdef640 1 -- start start 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216bdef640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2164100a80 0x7f2164199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216bdef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164101420 0x7f21641997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216bdef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f21641a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216bdef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2164110ef0 con 0x7f2164106a80 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216bdef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2164110d70 con 0x7f2164101420 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216bdef640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2164111070 con 0x7f2164100a80 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f21641a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f21641a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49348/0 (socket says 172.21.15.28:49348) 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 -- 172.21.15.28:0/442377768 learned_addr learned my addr 172.21.15.28:0/442377768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f2169b8b640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2164100a80 0x7f2164199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216938a640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164101420 0x7f21641997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 -- 172.21.15.28:0/442377768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2164100a80 msgr2=0x7f2164199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2164100a80 0x7f2164199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 -- 172.21.15.28:0/442377768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164101420 msgr2=0x7f21641997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164101420 0x7f21641997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:00.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 -- 172.21.15.28:0/442377768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21641a4840 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f2169b8b640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2164100a80 0x7f2164199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.832+0000 7f216a38c640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f21641a4300 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f215c009fd0 tx=0x7f215c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.833+0000 7f2162ffd640 1 -- 172.21.15.28:0/442377768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c011830 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.833+0000 7f2162ffd640 1 -- 172.21.15.28:0/442377768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215c016070 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.833+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21641a4a10 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.833+0000 7f2162ffd640 1 -- 172.21.15.28:0/442377768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c00f2e0 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.833+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21641a4f20 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.834+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2164100ee0 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.835+0000 7f2162ffd640 1 -- 172.21.15.28:0/442377768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f215c0119d0 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.836+0000 7f2162ffd640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f214c0a3430 0x7f214c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.836+0000 7f2169b8b640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f214c0a3430 0x7f214c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.836+0000 7f2162ffd640 1 -- 172.21.15.28:0/442377768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7f215c0cf6d0 con 0x7f2164106a80 2024-09-15T22:08:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.836+0000 7f2169b8b640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f214c0a3430 0x7f214c0a58f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2154006010 tx=0x7f2154005a20 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:00.840+0000 7f2162ffd640 1 -- 172.21.15.28:0/442377768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f215c09bdb0 con 0x7f2164106a80 2024-09-15T22:08:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.032+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f21641a51b0 con 0x7f2164106a80 2024-09-15T22:08:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.032+0000 7f2162ffd640 1 -- 172.21.15.28:0/442377768 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v99) ==== 81+0+941 (secure 0 0 0) 0x7f215c09b6b0 con 0x7f2164106a80 2024-09-15T22:08:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/972903625 2024-09-16T21:57:20.180486+0000 2024-09-15T22:08:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3676950981 2024-09-16T21:57:20.180486+0000 2024-09-15T22:08:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4057362541 2024-09-16T21:57:20.180486+0000 2024-09-15T22:08:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3052473616 2024-09-16T21:47:32.835958+0000 2024-09-15T22:08:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2627662465 2024-09-16T21:47:32.835958+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3052473616 2024-09-16T21:47:32.835958+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/123978199 2024-09-16T21:47:32.835958+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2625692755 2024-09-16T21:47:57.453457+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2625692755 2024-09-16T21:47:57.453457+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2709495038 2024-09-16T21:47:32.835958+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1395107553 2024-09-16T21:47:57.453457+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/711457183 2024-09-16T21:47:57.453457+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/671259623 2024-09-16T21:47:57.453457+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/955882315 2024-09-16T21:57:20.180486+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1670367809 2024-09-16T21:57:20.180486+0000 2024-09-15T22:08:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1670367809 2024-09-16T21:57:20.180486+0000 2024-09-15T22:08:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-09-15T22:08:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.035+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f214c0a3430 msgr2=0x7f214c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.035+0000 7f216bdef640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f214c0a3430 0x7f214c0a58f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2154006010 tx=0x7f2154005a20 comp rx=0 tx=0).stop 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.035+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 msgr2=0x7f21641a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.035+0000 7f216bdef640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f21641a4300 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f215c009fd0 tx=0x7f215c00b8e0 comp rx=0 tx=0).stop 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 shutdown_connections 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f214c0a3430 0x7f214c0a58f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2164106a80 0x7f21641a4300 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164101420 0x7f21641997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 --2- 172.21.15.28:0/442377768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2164100a80 0x7f2164199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 >> 172.21.15.28:0/442377768 conn(0x7f21640fc7a0 msgr2=0x7f21640fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 shutdown_connections 2024-09-15T22:08:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:01.036+0000 7f216bdef640 1 -- 172.21.15.28:0/442377768 wait complete. 2024-09-15T22:08:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: pgmap v329: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: Adjusting osd_memory_target on smithi203 to 176.0M 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: Unable to set osd_memory_target on smithi203 to 184563097: error parsing value: Value '184563097' is below minimum 939524096 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:01 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/442377768' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T22:08:01.656 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-15T22:08:01.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T22:08:01.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-15T22:08:01.707 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T22:08:01.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T22:08:01.708 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-15T22:08:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: pgmap v329: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:08:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: Adjusting osd_memory_target on smithi203 to 176.0M 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: Unable to set osd_memory_target on smithi203 to 184563097: error parsing value: Value '184563097' is below minimum 939524096 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:01 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/442377768' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T22:08:01.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T22:08:01.745 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: pgmap v329: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T22:08:01.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: Adjusting osd_memory_target on smithi203 to 176.0M 2024-09-15T22:08:01.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: Unable to set osd_memory_target on smithi203 to 184563097: error parsing value: Value '184563097' is below minimum 939524096 2024-09-15T22:08:01.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T22:08:01.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T22:08:01.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' 2024-09-15T22:08:01.761 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:01 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/442377768' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T22:08:01.828 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-15T22:08:01.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T22:08:01.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-15T22:08:01.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T22:08:01.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-15T22:08:02.005 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-15T22:08:02.017 INFO:tasks.cephadm:Teardown begin 2024-09-15T22:08:02.017 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-15T22:08:02.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:08:02.278 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 -- 172.21.15.28:0/3863373425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570106a80 msgr2=0x7fd570106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 --2- 172.21.15.28:0/3863373425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570106a80 0x7fd570106e60 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fd560009930 tx=0x7fd56002a420 comp rx=0 tx=0).stop 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 -- 172.21.15.28:0/3863373425 shutdown_connections 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 --2- 172.21.15.28:0/3863373425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570101420 0x7fd57010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 --2- 172.21.15.28:0/3863373425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd570100a80 0x7fd570100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 --2- 172.21.15.28:0/3863373425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570106a80 0x7fd570106e60 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 -- 172.21.15.28:0/3863373425 >> 172.21.15.28:0/3863373425 conn(0x7fd5700fc7a0 msgr2=0x7fd5700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 -- 172.21.15.28:0/3863373425 shutdown_connections 2024-09-15T22:08:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.848+0000 7fd576818640 1 -- 172.21.15.28:0/3863373425 wait complete. 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 Processor -- start 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 -- start start 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570100a80 0x7fd570198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd570101420 0x7fd570198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570106a80 0x7fd57019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd570110ed0 con 0x7fd570100a80 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd570110d50 con 0x7fd570101420 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd576818640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd570111050 con 0x7fd570106a80 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.849+0000 7fd56ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570100a80 0x7fd570198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570100a80 0x7fd570198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49370/0 (socket says 172.21.15.28:49370) 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 -- 172.21.15.28:0/2722186324 learned_addr learned my addr 172.21.15.28:0/2722186324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd574db5640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570106a80 0x7fd57019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 -- 172.21.15.28:0/2722186324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570106a80 msgr2=0x7fd57019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd570101420 0x7fd570198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570106a80 0x7fd57019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 -- 172.21.15.28:0/2722186324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd570101420 msgr2=0x7fd570198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd570101420 0x7fd570198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 -- 172.21.15.28:0/2722186324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57019d400 con 0x7fd570100a80 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd570101420 0x7fd570198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56ffff640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570100a80 0x7fd570198450 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd56002a940 tx=0x7fd560003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56d7fa640 1 -- 172.21.15.28:0/2722186324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56003f070 con 0x7fd570100a80 2024-09-15T22:08:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56d7fa640 1 -- 172.21.15.28:0/2722186324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd560031c90 con 0x7fd570100a80 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd57019d690 con 0x7fd570100a80 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd56d7fa640 1 -- 172.21.15.28:0/2722186324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560035030 con 0x7fd570100a80 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.850+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5701a4fd0 con 0x7fd570100a80 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.851+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5701021c0 con 0x7fd570100a80 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.853+0000 7fd56d7fa640 1 -- 172.21.15.28:0/2722186324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fd560042070 con 0x7fd570100a80 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.853+0000 7fd56d7fa640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd5580a3430 0x7fd5580a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.853+0000 7fd56d7fa640 1 -- 172.21.15.28:0/2722186324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9467+0+0 (secure 0 0 0) 0x7fd5600f0840 con 0x7fd570100a80 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.853+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd5580a3430 0x7fd5580a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.854+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd5580a3430 0x7fd5580a58f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd57006b7e0 tx=0x7fd564005d30 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:02.858+0000 7fd56d7fa640 1 -- 172.21.15.28:0/2722186324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd5600bce70 con 0x7fd570100a80 2024-09-15T22:08:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.069+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5701a51a0 con 0x7fd570100a80 2024-09-15T22:08:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.069+0000 7fd56d7fa640 1 -- 172.21.15.28:0/2722186324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11986 (secure 0 0 0) 0x7fd5600bc770 con 0x7fd570100a80 2024-09-15T22:08:03.071 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:08:03.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"btime":"2024-09-15T22:06:56:960477+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:06:56.716040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:08:03.072 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-09-15T22:08:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.071+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd5580a3430 msgr2=0x7fd5580a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.071+0000 7fd576818640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd5580a3430 0x7fd5580a58f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd57006b7e0 tx=0x7fd564005d30 comp rx=0 tx=0).stop 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.071+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570100a80 msgr2=0x7fd570198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.071+0000 7fd576818640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570100a80 0x7fd570198450 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fd56002a940 tx=0x7fd560003b90 comp rx=0 tx=0).stop 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 shutdown_connections 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fd5580a3430 0x7fd5580a58f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570106a80 0x7fd57019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd570101420 0x7fd570198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 --2- 172.21.15.28:0/2722186324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570100a80 0x7fd570198450 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 >> 172.21.15.28:0/2722186324 conn(0x7fd5700fc7a0 msgr2=0x7fd570104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 shutdown_connections 2024-09-15T22:08:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:03.072+0000 7fd576818640 1 -- 172.21.15.28:0/2722186324 wait complete. 2024-09-15T22:08:03.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:03 smithi174 ceph-mon[35244]: pgmap v330: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:08:03.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:03 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2722186324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:08:03.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:03 smithi028 ceph-mon[30969]: pgmap v330: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:08:03.806 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:03 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2722186324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:08:03.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T22:08:03.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:03 smithi203 ceph-mon[35546]: pgmap v330: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-09-15T22:08:03.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:03 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2722186324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:08:04.355 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:04.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:04 smithi174 ceph-mon[35244]: osdmap e100: 12 total, 12 up, 12 in 2024-09-15T22:08:04.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:04 smithi028 ceph-mon[30969]: osdmap e100: 12 total, 12 up, 12 in 2024-09-15T22:08:04.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:04 smithi203 ceph-mon[35546]: osdmap e100: 12 total, 12 up, 12 in 2024-09-15T22:08:05.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.396+0000 7f61bffb3640 1 -- 172.21.15.28:0/1281594352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109450 msgr2=0x7f61b8110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:05.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.396+0000 7f61bffb3640 1 --2- 172.21.15.28:0/1281594352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109450 0x7f61b8110250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f61b001acb0 tx=0x7f61b003b920 comp rx=0 tx=0).stop 2024-09-15T22:08:05.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.399+0000 7f61bffb3640 1 -- 172.21.15.28:0/1281594352 shutdown_connections 2024-09-15T22:08:05.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.399+0000 7f61bffb3640 1 --2- 172.21.15.28:0/1281594352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b811b140 0x7f61b811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.399+0000 7f61bffb3640 1 --2- 172.21.15.28:0/1281594352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61b8109830 0x7f61b8109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.399+0000 7f61bffb3640 1 --2- 172.21.15.28:0/1281594352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109450 0x7f61b8110250 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f61b001acb0 tx=0x7f61b003b920 comp rx=0 tx=0).stop 2024-09-15T22:08:05.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.399+0000 7f61bffb3640 1 -- 172.21.15.28:0/1281594352 >> 172.21.15.28:0/1281594352 conn(0x7f61b806e140 msgr2=0x7f61b806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:05.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.405+0000 7f61bffb3640 1 -- 172.21.15.28:0/1281594352 shutdown_connections 2024-09-15T22:08:05.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.407+0000 7f61bffb3640 1 -- 172.21.15.28:0/1281594352 wait complete. 2024-09-15T22:08:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.408+0000 7f61bffb3640 1 Processor -- start 2024-09-15T22:08:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.408+0000 7f61bffb3640 1 -- start start 2024-09-15T22:08:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.409+0000 7f61bffb3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109830 0x7f61b811abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.409+0000 7f61bffb3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61b811b140 0x7f61b8116c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.409+0000 7f61bffb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b8117320 0x7f61b81ab170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.409+0000 7f61bffb3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61b81133e0 con 0x7f61b8117320 2024-09-15T22:08:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.409+0000 7f61bffb3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8113260 con 0x7f61b8109830 2024-09-15T22:08:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.409+0000 7f61bffb3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8113560 con 0x7f61b811b140 2024-09-15T22:08:05.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.410+0000 7f61be550640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b8117320 0x7f61b81ab170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:05.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.410+0000 7f61be550640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b8117320 0x7f61b81ab170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49376/0 (socket says 172.21.15.28:49376) 2024-09-15T22:08:05.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.410+0000 7f61be550640 1 -- 172.21.15.28:0/3624566535 learned_addr learned my addr 172.21.15.28:0/3624566535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:05.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.410+0000 7f61be550640 1 -- 172.21.15.28:0/3624566535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61b811b140 msgr2=0x7f61b8116c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T22:08:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.411+0000 7f61bd54e640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61b811b140 0x7f61b8116c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.412+0000 7f61bdd4f640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109830 0x7f61b811abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.414+0000 7f61be550640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61b811b140 0x7f61b8116c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.414+0000 7f61be550640 1 -- 172.21.15.28:0/3624566535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109830 msgr2=0x7f61b811abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.414+0000 7f61be550640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109830 0x7f61b811abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.415+0000 7f61be550640 1 -- 172.21.15.28:0/3624566535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b81ab6b0 con 0x7f61b8117320 2024-09-15T22:08:05.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.415+0000 7f61be550640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b8117320 0x7f61b81ab170 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b3e0 tx=0x7f61b400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.416+0000 7f61aeffd640 1 -- 172.21.15.28:0/3624566535 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4015020 con 0x7f61b8117320 2024-09-15T22:08:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.416+0000 7f61aeffd640 1 -- 172.21.15.28:0/3624566535 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b4011d10 con 0x7f61b8117320 2024-09-15T22:08:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.416+0000 7f61aeffd640 1 -- 172.21.15.28:0/3624566535 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b400f2e0 con 0x7f61b8117320 2024-09-15T22:08:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.417+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b81ab880 con 0x7f61b8117320 2024-09-15T22:08:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.417+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b81abc80 con 0x7f61b8117320 2024-09-15T22:08:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.418+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b8073b00 con 0x7f61b8117320 2024-09-15T22:08:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.425+0000 7f61aeffd640 1 -- 172.21.15.28:0/3624566535 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f61b4018020 con 0x7f61b8117320 2024-09-15T22:08:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.426+0000 7f61aeffd640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f61a40a35c0 0x7f61a40a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.427+0000 7f61aeffd640 1 -- 172.21.15.28:0/3624566535 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9555+0+0 (secure 0 0 0) 0x7f61b40cf970 con 0x7f61b8117320 2024-09-15T22:08:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.434+0000 7f61aeffd640 1 -- 172.21.15.28:0/3624566535 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f61b401f020 con 0x7f61b8117320 2024-09-15T22:08:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.435+0000 7f61bdd4f640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f61a40a35c0 0x7f61a40a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.454+0000 7f61bdd4f640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f61a40a35c0 0x7f61a40a5a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f61b0000f80 tx=0x7f61b001a0a0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:05.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:05 smithi028 ceph-mon[30969]: pgmap v332: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:08:05.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:05 smithi174 ceph-mon[35244]: pgmap v332: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:08:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.852+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61b81abfa0 con 0x7f61b8117320 2024-09-15T22:08:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.854+0000 7f61aeffd640 1 -- 172.21.15.28:0/3624566535 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11986 (secure 0 0 0) 0x7f61b409bff0 con 0x7f61b8117320 2024-09-15T22:08:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:08:05.856 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"btime":"2024-09-15T22:06:56:960477+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2109233987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2109233987},{"type":"v1","addr":"172.21.15.28:6835","nonce":2109233987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2070322956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2070322956},{"type":"v1","addr":"172.21.15.28:6837","nonce":2070322956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T21:54:59.048835+0000","modified":"2024-09-15T22:06:56.716040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24485,"mds_1":24481,"mds_2":24460,"mds_3":24458,"mds_4":24454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3176985638","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3176985638},{"type":"v1","addr":"172.21.15.28:6839","nonce":3176985638}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/2641693590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2641693590},{"type":"v1","addr":"172.21.15.28:6841","nonce":2641693590}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2800102614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2800102614},{"type":"v1","addr":"172.21.15.203:6833","nonce":2800102614}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/1988552520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1988552520},{"type":"v1","addr":"172.21.15.174:6833","nonce":1988552520}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/280309786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":280309786},{"type":"v1","addr":"172.21.15.174:6835","nonce":280309786}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.203:6835/1524895518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1524895518},{"type":"v1","addr":"172.21.15.203:6835","nonce":1524895518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6837/3327498469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3327498469},{"type":"v1","addr":"172.21.15.174:6837","nonce":3327498469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":17,"addr":"172.21.15.203:6837/803844131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":803844131},{"type":"v1","addr":"172.21.15.203:6837","nonce":803844131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":15,"addr":"172.21.15.203:6839/3242294678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3242294678},{"type":"v1","addr":"172.21.15.203:6839","nonce":3242294678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24485":{"gid":24485,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6839/3735159621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3735159621},{"type":"v1","addr":"172.21.15.174:6839","nonce":3735159621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24485,"qdb_cluster":[24485,24481,24460,24458,24454]},"id":1}]} 2024-09-15T22:08:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-09-15T22:08:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.860+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f61a40a35c0 msgr2=0x7f61a40a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.860+0000 7f61bffb3640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f61a40a35c0 0x7f61a40a5a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f61b0000f80 tx=0x7f61b001a0a0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.860+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b8117320 msgr2=0x7f61b81ab170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.860+0000 7f61bffb3640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b8117320 0x7f61b81ab170 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b3e0 tx=0x7f61b400b8a0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.864+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 shutdown_connections 2024-09-15T22:08:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.864+0000 7f61bffb3640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f61a40a35c0 0x7f61a40a5a80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.864+0000 7f61bffb3640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b8117320 0x7f61b81ab170 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.864+0000 7f61bffb3640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61b811b140 0x7f61b8116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.864+0000 7f61bffb3640 1 --2- 172.21.15.28:0/3624566535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b8109830 0x7f61b811abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.864+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 >> 172.21.15.28:0/3624566535 conn(0x7f61b806e140 msgr2=0x7f61b8110000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:05.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.866+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 shutdown_connections 2024-09-15T22:08:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:05.867+0000 7f61bffb3640 1 -- 172.21.15.28:0/3624566535 wait complete. 2024-09-15T22:08:05.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:05 smithi203 ceph-mon[35546]: pgmap v332: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:08:06.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:06 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3624566535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:08:06.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T22:08:06.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:06 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3624566535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:08:06.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:06 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3624566535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T22:08:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:07.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.677+0000 7f5fae8e4640 1 -- 172.21.15.28:0/2335306542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa8103670 msgr2=0x7f5fa810bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:07.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.677+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/2335306542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa8103670 0x7f5fa810bb60 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009900 tx=0x7f5f9c02a4a0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.678+0000 7f5fae8e4640 1 -- 172.21.15.28:0/2335306542 shutdown_connections 2024-09-15T22:08:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.678+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/2335306542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa810c180 0x7f5fa810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.678+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/2335306542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa8103670 0x7f5fa810bb60 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.678+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/2335306542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa8102d50 0x7f5fa8103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.678+0000 7f5fae8e4640 1 -- 172.21.15.28:0/2335306542 >> 172.21.15.28:0/2335306542 conn(0x7f5fa80fc7d0 msgr2=0x7f5fa80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.678+0000 7f5fae8e4640 1 -- 172.21.15.28:0/2335306542 shutdown_connections 2024-09-15T22:08:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.678+0000 7f5fae8e4640 1 -- 172.21.15.28:0/2335306542 wait complete. 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.679+0000 7f5fae8e4640 1 Processor -- start 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.679+0000 7f5fae8e4640 1 -- start start 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fae8e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa8102d50 0x7f5fa81a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fae8e4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa8103670 0x7f5fa81a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fae8e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa810c180 0x7f5fa810acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fae8e4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8111fc0 con 0x7f5fa810c180 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fae8e4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8111e40 con 0x7f5fa8102d50 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fa77fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa8103670 0x7f5fa81a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fa77fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa8103670 0x7f5fa81a4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.28:47390/0 (socket says 172.21.15.28:47390) 2024-09-15T22:08:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fa77fe640 1 -- 172.21.15.28:0/3250440680 learned_addr learned my addr 172.21.15.28:0/3250440680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fae8e4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8112140 con 0x7f5fa8103670 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5face81640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa810c180 0x7f5fa810acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.680+0000 7f5fa7fff640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa8102d50 0x7f5fa81a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.681+0000 7f5fa77fe640 1 -- 172.21.15.28:0/3250440680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa8102d50 msgr2=0x7f5fa81a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.681+0000 7f5fa77fe640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa8102d50 0x7f5fa81a4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.681+0000 7f5fa77fe640 1 -- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa810c180 msgr2=0x7f5fa810acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.681+0000 7f5fa77fe640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa810c180 0x7f5fa810acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.681+0000 7f5fa77fe640 1 -- 172.21.15.28:0/3250440680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa810b420 con 0x7f5fa8103670 2024-09-15T22:08:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.681+0000 7f5fa77fe640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa8103670 0x7f5fa81a4c70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c005ab0 tx=0x7f5f9c005240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.682+0000 7f5fa57fa640 1 -- 172.21.15.28:0/3250440680 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c040030 con 0x7f5fa8103670 2024-09-15T22:08:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.682+0000 7f5fa57fa640 1 -- 172.21.15.28:0/3250440680 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f9c03c920 con 0x7f5fa8103670 2024-09-15T22:08:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.682+0000 7f5fa57fa640 1 -- 172.21.15.28:0/3250440680 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c040030 con 0x7f5fa8103670 2024-09-15T22:08:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.682+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa810b6b0 con 0x7f5fa8103670 2024-09-15T22:08:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.682+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa810ba80 con 0x7f5fa8103670 2024-09-15T22:08:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.684+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa8103130 con 0x7f5fa8103670 2024-09-15T22:08:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.685+0000 7f5fa57fa640 1 -- 172.21.15.28:0/3250440680 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f5f9c043030 con 0x7f5fa8103670 2024-09-15T22:08:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.686+0000 7f5fa57fa640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5f900a37b0 0x7f5f900a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.686+0000 7f5fa7fff640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5f900a37b0 0x7f5f900a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.686+0000 7f5fa57fa640 1 -- 172.21.15.28:0/3250440680 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9555+0+0 (secure 0 0 0) 0x7f5f9c041070 con 0x7f5fa8103670 2024-09-15T22:08:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.687+0000 7f5fa7fff640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5f900a37b0 0x7f5f900a5c70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5fa810be30 tx=0x7f5fa0009500 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.692+0000 7f5fa57fa640 1 -- 172.21.15.28:0/3250440680 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5f9c0bc790 con 0x7f5fa8103670 2024-09-15T22:08:07.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:07 smithi203 ceph-mon[35546]: pgmap v333: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:08:07.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:07 smithi174 ceph-mon[35244]: pgmap v333: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:08:07.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:07 smithi028 ceph-mon[30969]: pgmap v333: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 31 op/s 2024-09-15T22:08:07.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.992+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fa81ab750 con 0x7f5fa8103670 2024-09-15T22:08:07.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.993+0000 7f5fa57fa640 1 -- 172.21.15.28:0/3250440680 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+25459 (secure 0 0 0) 0x7f5f9c0bc090 con 0x7f5fa8103670 2024-09-15T22:08:07.996 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-15T22:08:07.996 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":100,"fsid":"e78b7764-73ab-11ef-bceb-c7b262605968","created":"2024-09-15T21:47:10.448749+0000","modified":"2024-09-15T22:08:03.472068+0000","last_up_change":"2024-09-15T21:54:06.164653+0000","last_in_change":"2024-09-15T21:53:41.973820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T21:50:13.957276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T21:54:54.805516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T21:54:57.410111+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":16,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T21:55:03.884115+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":16,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5066bae-2a3f-4b91-9da9-39d341d3257e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6803","nonce":896252218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6805","nonce":896252218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6809","nonce":896252218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":896252218},{"type":"v1","addr":"172.21.15.28:6807","nonce":896252218}]},"public_addr":"172.21.15.28:6803/896252218","cluster_addr":"172.21.15.28:6805/896252218","heartbeat_back_addr":"172.21.15.28:6809/896252218","heartbeat_front_addr":"172.21.15.28:6807/896252218","state":["exists","up"]},{"osd":1,"uuid":"b85e5570-ddd1-4e02-858e-948a4517801e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6801","nonce":4110616785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6803","nonce":4110616785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6807","nonce":4110616785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":4110616785},{"type":"v1","addr":"172.21.15.174:6805","nonce":4110616785}]},"public_addr":"172.21.15.174:6801/4110616785","cluster_addr":"172.21.15.174:6803/4110616785","heartbeat_back_addr":"172.21.15.174:6807/4110616785","heartbeat_front_addr":"172.21.15.174:6805/4110616785","state":["exists","up"]},{"osd":2,"uuid":"41e4d0e4-7b72-4ad8-9c2e-a260d638777f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6801","nonce":2523083594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6803","nonce":2523083594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6807","nonce":2523083594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2523083594},{"type":"v1","addr":"172.21.15.203:6805","nonce":2523083594}]},"public_addr":"172.21.15.203:6801/2523083594","cluster_addr":"172.21.15.203:6803/2523083594","heartbeat_back_addr":"172.21.15.203:6807/2523083594","heartbeat_front_addr":"172.21.15.203:6805/2523083594","state":["exists","up"]},{"osd":3,"uuid":"bd998a1a-b9b4-4950-8925-5b3d2da0e73c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6811","nonce":1752867147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6813","nonce":1752867147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6817","nonce":1752867147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1752867147},{"type":"v1","addr":"172.21.15.28:6815","nonce":1752867147}]},"public_addr":"172.21.15.28:6811/1752867147","cluster_addr":"172.21.15.28:6813/1752867147","heartbeat_back_addr":"172.21.15.28:6817/1752867147","heartbeat_front_addr":"172.21.15.28:6815/1752867147","state":["exists","up"]},{"osd":4,"uuid":"7d6dc76a-d3bc-435b-afc1-87f35ab0e2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6809","nonce":3085836897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6811","nonce":3085836897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6815","nonce":3085836897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3085836897},{"type":"v1","addr":"172.21.15.174:6813","nonce":3085836897}]},"public_addr":"172.21.15.174:6809/3085836897","cluster_addr":"172.21.15.174:6811/3085836897","heartbeat_back_addr":"172.21.15.174:6815/3085836897","heartbeat_front_addr":"172.21.15.174:6813/3085836897","state":["exists","up"]},{"osd":5,"uuid":"40516595-c52a-4503-b534-2b5c855d431e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6809","nonce":3676469409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6811","nonce":3676469409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6815","nonce":3676469409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3676469409},{"type":"v1","addr":"172.21.15.203:6813","nonce":3676469409}]},"public_addr":"172.21.15.203:6809/3676469409","cluster_addr":"172.21.15.203:6811/3676469409","heartbeat_back_addr":"172.21.15.203:6815/3676469409","heartbeat_front_addr":"172.21.15.203:6813/3676469409","state":["exists","up"]},{"osd":6,"uuid":"4db3191e-2eb9-4ce3-b007-11d4305d0c91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6819","nonce":908626673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6821","nonce":908626673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6825","nonce":908626673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":908626673},{"type":"v1","addr":"172.21.15.28:6823","nonce":908626673}]},"public_addr":"172.21.15.28:6819/908626673","cluster_addr":"172.21.15.28:6821/908626673","heartbeat_back_addr":"172.21.15.28:6825/908626673","heartbeat_front_addr":"172.21.15.28:6823/908626673","state":["exists","up"]},{"osd":7,"uuid":"c6259a57-5604-481c-b739-f187239955d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6817","nonce":1679850033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6819","nonce":1679850033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6823","nonce":1679850033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1679850033},{"type":"v1","addr":"172.21.15.174:6821","nonce":1679850033}]},"public_addr":"172.21.15.174:6817/1679850033","cluster_addr":"172.21.15.174:6819/1679850033","heartbeat_back_addr":"172.21.15.174:6823/1679850033","heartbeat_front_addr":"172.21.15.174:6821/1679850033","state":["exists","up"]},{"osd":8,"uuid":"e17e4b72-4e4d-4a52-a8d9-d771d4d22ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6817","nonce":3629574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6819","nonce":3629574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6823","nonce":3629574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3629574},{"type":"v1","addr":"172.21.15.203:6821","nonce":3629574}]},"public_addr":"172.21.15.203:6817/3629574","cluster_addr":"172.21.15.203:6819/3629574","heartbeat_back_addr":"172.21.15.203:6823/3629574","heartbeat_front_addr":"172.21.15.203:6821/3629574","state":["exists","up"]},{"osd":9,"uuid":"fbe08e2a-856e-4836-bfa3-fa269e447a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6827","nonce":306652678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6829","nonce":306652678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6833","nonce":306652678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":306652678},{"type":"v1","addr":"172.21.15.28:6831","nonce":306652678}]},"public_addr":"172.21.15.28:6827/306652678","cluster_addr":"172.21.15.28:6829/306652678","heartbeat_back_addr":"172.21.15.28:6833/306652678","heartbeat_front_addr":"172.21.15.28:6831/306652678","state":["exists","up"]},{"osd":10,"uuid":"ff5afcc0-0e93-459e-87a8-1a34428aef91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6825","nonce":2504623164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6827","nonce":2504623164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6831","nonce":2504623164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":2504623164},{"type":"v1","addr":"172.21.15.174:6829","nonce":2504623164}]},"public_addr":"172.21.15.174:6825/2504623164","cluster_addr":"172.21.15.174:6827/2504623164","heartbeat_back_addr":"172.21.15.174:6831/2504623164","heartbeat_front_addr":"172.21.15.174:6829/2504623164","state":["exists","up"]},{"osd":11,"uuid":"3393ee20-fde1-48f1-b9f8-407c434f1386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6825","nonce":2779316485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6827","nonce":2779316485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6831","nonce":2779316485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2779316485},{"type":"v1","addr":"172.21.15.203:6829","nonce":2779316485}]},"public_addr":"172.21.15.203:6825/2779316485","cluster_addr":"172.21.15.203:6827/2779316485","heartbeat_back_addr":"172.21.15.203:6831/2779316485","heartbeat_front_addr":"172.21.15.203:6829/2779316485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:21.466159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:49:46.459948+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:10.983348+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:50:37.228218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:02.697153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:28.944773+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:51:55.125562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:20.954722+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:52:46.674843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:12.276052+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:53:38.066538+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:54:04.272192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/972903625":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/3676950981":"2024-09-16T21:57:20.180486+0000","172.21.15.28:0/4057362541":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/2627662465":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6801/3052473616":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/123978199":"2024-09-16T21:47:32.835958+0000","172.21.15.28:6800/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:6801/2625692755":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/2709495038":"2024-09-16T21:47:32.835958+0000","172.21.15.28:0/1395107553":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/711457183":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/671259623":"2024-09-16T21:47:57.453457+0000","172.21.15.28:0/955882315":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6800/1670367809":"2024-09-16T21:57:20.180486+0000","172.21.15.28:6801/1670367809":"2024-09-16T21:57:20.180486+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":16,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":16,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":16,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":16,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T22:08:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5f900a37b0 msgr2=0x7f5f900a5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5f900a37b0 0x7f5f900a5c70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5fa810be30 tx=0x7f5fa0009500 comp rx=0 tx=0).stop 2024-09-15T22:08:07.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa8103670 msgr2=0x7f5fa81a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa8103670 0x7f5fa81a4c70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c005ab0 tx=0x7f5f9c005240 comp rx=0 tx=0).stop 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 shutdown_connections 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f5f900a37b0 0x7f5f900a5c70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa810c180 0x7f5fa810acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fa8103670 0x7f5fa81a4c70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 --2- 172.21.15.28:0/3250440680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5fa8102d50 0x7f5fa81a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 >> 172.21.15.28:0/3250440680 conn(0x7f5fa80fc7d0 msgr2=0x7f5fa810d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:07.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 shutdown_connections 2024-09-15T22:08:07.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:07.996+0000 7f5fae8e4640 1 -- 172.21.15.28:0/3250440680 wait complete. 2024-09-15T22:08:08.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:08 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/3250440680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T22:08:08.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:08 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/3250440680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T22:08:08.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:08 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/3250440680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T22:08:09.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs fail cephfs 2024-09-15T22:08:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:09.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:09 smithi028 ceph-mon[30969]: pgmap v334: 129 pgs: 12 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 93 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-09-15T22:08:09.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:09 smithi203 ceph-mon[35546]: pgmap v334: 129 pgs: 12 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 93 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-09-15T22:08:10.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:09 smithi174 ceph-mon[35244]: pgmap v334: 129 pgs: 12 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 93 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-09-15T22:08:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.071+0000 7fac30aa8640 1 -- 172.21.15.28:0/568490143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c10c170 msgr2=0x7fac2c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.071+0000 7fac30aa8640 1 --2- 172.21.15.28:0/568490143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c10c170 0x7fac2c10e610 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fac24009900 tx=0x7fac2402a480 comp rx=0 tx=0).stop 2024-09-15T22:08:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.071+0000 7fac30aa8640 1 -- 172.21.15.28:0/568490143 shutdown_connections 2024-09-15T22:08:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.071+0000 7fac30aa8640 1 --2- 172.21.15.28:0/568490143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c10c170 0x7fac2c10e610 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.071+0000 7fac30aa8640 1 --2- 172.21.15.28:0/568490143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac2c103660 0x7fac2c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.071+0000 7fac30aa8640 1 --2- 172.21.15.28:0/568490143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2c102d40 0x7fac2c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.071+0000 7fac30aa8640 1 -- 172.21.15.28:0/568490143 >> 172.21.15.28:0/568490143 conn(0x7fac2c0fc7a0 msgr2=0x7fac2c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.072+0000 7fac30aa8640 1 -- 172.21.15.28:0/568490143 shutdown_connections 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.072+0000 7fac30aa8640 1 -- 172.21.15.28:0/568490143 wait complete. 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.072+0000 7fac30aa8640 1 Processor -- start 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.072+0000 7fac30aa8640 1 -- start start 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.072+0000 7fac30aa8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac2c102d40 0x7fac2c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.072+0000 7fac30aa8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c103660 0x7fac2c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.072+0000 7fac30aa8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2c10c170 0x7fac2c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac30aa8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c111f70 con 0x7fac2c103660 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac30aa8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c111df0 con 0x7fac2c102d40 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac30aa8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c1120f0 con 0x7fac2c10c170 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c103660 0x7fac2c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c103660 0x7fac2c198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49422/0 (socket says 172.21.15.28:49422) 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 -- 172.21.15.28:0/1907371146 learned_addr learned my addr 172.21.15.28:0/1907371146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac2a59c640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac2c102d40 0x7fac2c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac2ad9d640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2c10c170 0x7fac2c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 -- 172.21.15.28:0/1907371146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2c10c170 msgr2=0x7fac2c19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2c10c170 0x7fac2c19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 -- 172.21.15.28:0/1907371146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac2c102d40 msgr2=0x7fac2c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac2c102d40 0x7fac2c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 -- 172.21.15.28:0/1907371146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2c19d3c0 con 0x7fac2c103660 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac29d9b640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c103660 0x7fac2c198950 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fac2000d740 tx=0x7fac2000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac1b7fe640 1 -- 172.21.15.28:0/1907371146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac20017070 con 0x7fac2c103660 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2c19d6b0 con 0x7fac2c103660 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac1b7fe640 1 -- 172.21.15.28:0/1907371146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac2000de50 con 0x7fac2c103660 2024-09-15T22:08:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac1b7fe640 1 -- 172.21.15.28:0/1907371146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2000f620 con 0x7fac2c103660 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.073+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2c1a4f90 con 0x7fac2c103660 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.074+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac2c103120 con 0x7fac2c103660 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.076+0000 7fac1b7fe640 1 -- 172.21.15.28:0/1907371146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fac20009c30 con 0x7fac2c103660 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.077+0000 7fac1b7fe640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac140a33b0 0x7fac140a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.077+0000 7fac2a59c640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac140a33b0 0x7fac140a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.077+0000 7fac1b7fe640 1 -- 172.21.15.28:0/1907371146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9555+0+0 (secure 0 0 0) 0x7fac200cfc10 con 0x7fac2c103660 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.077+0000 7fac2a59c640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac140a33b0 0x7fac140a5870 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fac1c005fd0 tx=0x7fac1c009680 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.081+0000 7fac1b7fe640 1 -- 172.21.15.28:0/1907371146 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fac2009bb90 con 0x7fac2c103660 2024-09-15T22:08:10.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.291+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fac2c1a51c0 con 0x7fac2c103660 2024-09-15T22:08:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.294+0000 7fac1b7fe640 1 -- 172.21.15.28:0/1907371146 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) ==== 161+0+0 (secure 0 0 0) 0x7fac2009fe90 con 0x7fac2c103660 2024-09-15T22:08:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-15T22:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.297+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac140a33b0 msgr2=0x7fac140a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.297+0000 7fac30aa8640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac140a33b0 0x7fac140a5870 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fac1c005fd0 tx=0x7fac1c009680 comp rx=0 tx=0).stop 2024-09-15T22:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.297+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c103660 msgr2=0x7fac2c198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.297+0000 7fac30aa8640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c103660 0x7fac2c198950 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fac2000d740 tx=0x7fac2000dc00 comp rx=0 tx=0).stop 2024-09-15T22:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.297+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 shutdown_connections 2024-09-15T22:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.297+0000 7fac30aa8640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fac140a33b0 0x7fac140a5870 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.297+0000 7fac30aa8640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2c10c170 0x7fac2c19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.298+0000 7fac30aa8640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac2c103660 0x7fac2c198950 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.298+0000 7fac30aa8640 1 --2- 172.21.15.28:0/1907371146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fac2c102d40 0x7fac2c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.298+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 >> 172.21.15.28:0/1907371146 conn(0x7fac2c0fc7a0 msgr2=0x7fac2c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.299+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 shutdown_connections 2024-09-15T22:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:10.299+0000 7fac30aa8640 1 -- 172.21.15.28:0/1907371146 wait complete. 2024-09-15T22:08:10.671 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a[71640]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.671 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a[71640]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mds[71656]: -- 172.21.15.28:0/943117240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559e3151d860 con 0x559e314fb400 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a[71640]: starting mds.a at 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a[30965]: 2024-09-15T22:08:10.292+0000 7fd760de6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1907371146' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mon[30969]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mon[30969]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/1907371146' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mon[30969]: osdmap e101: 12 total, 12 up, 12 in 2024-09-15T22:08:10.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mon[30969]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-15T22:08:10.673 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g[71303]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.673 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g[71303]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.673 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-mds[71311]: -- 172.21.15.28:0/2824083295 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625fa12fc20 con 0x5625fa10d400 2024-09-15T22:08:10.673 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:08:10 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g[71303]: starting mds.g at 2024-09-15T22:08:10.718 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l[65122]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.718 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l[65122]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.718 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mds[65126]: -- 172.21.15.203:0/2176332532 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6c8047860 con 0x55c6c8024800 2024-09-15T22:08:10.718 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l[65122]: starting mds.l at 2024-09-15T22:08:10.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1907371146' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mon[35546]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mon[35546]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/1907371146' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mon[35546]: osdmap e101: 12 total, 12 up, 12 in 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mon[35546]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i[66025]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i[66025]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mds[66029]: -- 172.21.15.203:0/720910402 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a23485d860 con 0x55a23483a800 2024-09-15T22:08:10.719 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i[66025]: starting mds.i at 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c[65725]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c[65725]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mds[65729]: -- 172.21.15.203:0/3342840537 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5575897ab860 con 0x557589789400 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c[65725]: starting mds.c at 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f[65422]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f[65422]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-mds[65426]: -- 172.21.15.203:0/3323979453 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56532b1c5c20 con 0x56532b1a3000 2024-09-15T22:08:10.720 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:08:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f[65422]: starting mds.f at 2024-09-15T22:08:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1907371146' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T22:08:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mon[35244]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T22:08:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mon[35244]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T22:08:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/1907371146' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T22:08:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mon[35244]: osdmap e101: 12 total, 12 up, 12 in 2024-09-15T22:08:10.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mon[35244]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-15T22:08:10.761 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.761 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.761 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: starting mds.b at 2024-09-15T22:08:10.761 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h[65655]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.761 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h[65655]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.761 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mds[65659]: -- 172.21.15.174:0/3871887897 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e23cf1f860 con 0x55e23cefd000 2024-09-15T22:08:10.761 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h[65655]: starting mds.h at 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e[65351]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e[65351]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mds[65355]: -- 172.21.15.174:0/1897245912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bac7bcb860 con 0x55bac7ba8800 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e[65351]: starting mds.e at 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k[65038]: ignoring --setuser ceph since I am not root 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k[65038]: ignoring --setgroup ceph since I am not root 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-mds[65042]: -- 172.21.15.174:0/4160548978 <== mon.2 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec43911860 con 0x55ec438ef000 2024-09-15T22:08:10.762 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:08:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k[65038]: starting mds.k at 2024-09-15T22:08:11.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-15T22:08:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:11.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: pgmap v336: 129 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 111 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6832/1575681602,v1:172.21.15.203:6834/1575681602] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6833/2113299161,v1:172.21.15.203:6835/2113299161] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6835/1504212416,v1:172.21.15.174:6838/1504212416] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6834/1249094162,v1:172.21.15.174:6839/1249094162] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6837/211812609,v1:172.21.15.203:6839/211812609] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6832/4166593002,v1:172.21.15.174:6836/4166593002] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.203:6836/2094356611,v1:172.21.15.203:6838/2094356611] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.174:6833/2717399800,v1:172.21.15.174:6837/2717399800] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.28:6838/111343716,v1:172.21.15.28:6840/111343716] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: mds.? [v2:172.21.15.28:6839/2807829645,v1:172.21.15.28:6841/2807829645] up:boot 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T22:08:11.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T22:08:11.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:11 smithi028 ceph-mon[30969]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 -- 172.21.15.28:0/3489395387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010102570 msgr2=0x7fa0101029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 --2- 172.21.15.28:0/3489395387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010102570 0x7fa0101029b0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa000009900 tx=0x7fa00002a480 comp rx=0 tx=0).stop 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 -- 172.21.15.28:0/3489395387 shutdown_connections 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 --2- 172.21.15.28:0/3489395387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010102ef0 0x7fa01010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 --2- 172.21.15.28:0/3489395387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010102570 0x7fa0101029b0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 --2- 172.21.15.28:0/3489395387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa010100e20 0x7fa010101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 -- 172.21.15.28:0/3489395387 >> 172.21.15.28:0/3489395387 conn(0x7fa0100fc7d0 msgr2=0x7fa0100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 -- 172.21.15.28:0/3489395387 shutdown_connections 2024-09-15T22:08:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 -- 172.21.15.28:0/3489395387 wait complete. 2024-09-15T22:08:11.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.773+0000 7fa017899640 1 Processor -- start 2024-09-15T22:08:11.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa017899640 1 -- start start 2024-09-15T22:08:11.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa017899640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010100e20 0x7fa010197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:11.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa017899640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010102570 0x7fa010197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa017899640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa010102ef0 0x7fa010197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa017899640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa010110fc0 con 0x7fa010100e20 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa017899640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa010110e40 con 0x7fa010102ef0 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa017899640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa010111140 con 0x7fa010102570 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010100e20 0x7fa010197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010100e20 0x7fa010197220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44164/0 (socket says 172.21.15.28:44164) 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 -- 172.21.15.28:0/699903212 learned_addr learned my addr 172.21.15.28:0/699903212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015e36640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa010102ef0 0x7fa010197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa014e34640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010102570 0x7fa010197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 -- 172.21.15.28:0/699903212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010102570 msgr2=0x7fa010197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010102570 0x7fa010197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 -- 172.21.15.28:0/699903212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa010102ef0 msgr2=0x7fa010197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa010102ef0 0x7fa010197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.774+0000 7fa015635640 1 -- 172.21.15.28:0/699903212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0101a6a90 con 0x7fa010100e20 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.775+0000 7fa015635640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010100e20 0x7fa010197220 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00b4a0 tx=0x7fa00c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.775+0000 7fa0067fc640 1 -- 172.21.15.28:0/699903212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c014070 con 0x7fa010100e20 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.775+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0101a6d80 con 0x7fa010100e20 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.775+0000 7fa0067fc640 1 -- 172.21.15.28:0/699903212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa00c010af0 con 0x7fa010100e20 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.775+0000 7fa0067fc640 1 -- 172.21.15.28:0/699903212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c017d10 con 0x7fa010100e20 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.775+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0101a7230 con 0x7fa010100e20 2024-09-15T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.776+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa010101200 con 0x7fa010100e20 2024-09-15T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.777+0000 7fa0067fc640 1 -- 172.21.15.28:0/699903212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fa00c017070 con 0x7fa010100e20 2024-09-15T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.778+0000 7fa0067fc640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ffc0a3430 0x7f9ffc0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.778+0000 7fa014e34640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ffc0a3430 0x7f9ffc0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.778+0000 7fa0067fc640 1 -- 172.21.15.28:0/699903212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9897+0+0 (secure 0 0 0) 0x7fa00c0cf3f0 con 0x7fa010100e20 2024-09-15T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.778+0000 7fa014e34640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ffc0a3430 0x7f9ffc0a58f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa000009230 tx=0x7fa0000058d0 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.782+0000 7fa0067fc640 1 -- 172.21.15.28:0/699903212 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa00c09b2d0 con 0x7fa010100e20 2024-09-15T22:08:11.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: pgmap v336: 129 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 111 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-15T22:08:11.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6832/1575681602,v1:172.21.15.203:6834/1575681602] up:boot 2024-09-15T22:08:11.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6833/2113299161,v1:172.21.15.203:6835/2113299161] up:boot 2024-09-15T22:08:11.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6835/1504212416,v1:172.21.15.174:6838/1504212416] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6834/1249094162,v1:172.21.15.174:6839/1249094162] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6837/211812609,v1:172.21.15.203:6839/211812609] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6832/4166593002,v1:172.21.15.174:6836/4166593002] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.203:6836/2094356611,v1:172.21.15.203:6838/2094356611] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.174:6833/2717399800,v1:172.21.15.174:6837/2717399800] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.28:6838/111343716,v1:172.21.15.28:6840/111343716] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: mds.? [v2:172.21.15.28:6839/2807829645,v1:172.21.15.28:6841/2807829645] up:boot 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T22:08:11.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T22:08:11.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T22:08:11.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T22:08:11.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T22:08:11.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T22:08:11.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T22:08:11.970 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:11 smithi203 ceph-mon[35546]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T22:08:11.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:11.996+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa01002d7c0 con 0x7fa010100e20 2024-09-15T22:08:12.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: pgmap v336: 129 pgs: 8 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 111 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-15T22:08:12.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6832/1575681602,v1:172.21.15.203:6834/1575681602] up:boot 2024-09-15T22:08:12.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6833/2113299161,v1:172.21.15.203:6835/2113299161] up:boot 2024-09-15T22:08:12.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6835/1504212416,v1:172.21.15.174:6838/1504212416] up:boot 2024-09-15T22:08:12.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6834/1249094162,v1:172.21.15.174:6839/1249094162] up:boot 2024-09-15T22:08:12.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6837/211812609,v1:172.21.15.203:6839/211812609] up:boot 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6832/4166593002,v1:172.21.15.174:6836/4166593002] up:boot 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.203:6836/2094356611,v1:172.21.15.203:6838/2094356611] up:boot 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.174:6833/2717399800,v1:172.21.15.174:6837/2717399800] up:boot 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.28:6838/111343716,v1:172.21.15.28:6840/111343716] up:boot 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: mds.? [v2:172.21.15.28:6839/2807829645,v1:172.21.15.28:6841/2807829645] up:boot 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T22:08:12.011 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T22:08:12.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T22:08:12.012 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:11 smithi174 ceph-mon[35244]: from='mgr.15330 172.21.15.28:0/4086532146' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T22:08:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.519+0000 7fa0067fc640 1 -- 172.21.15.28:0/699903212 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7fa00c09f5d0 con 0x7fa010100e20 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ffc0a3430 msgr2=0x7f9ffc0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ffc0a3430 0x7f9ffc0a58f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa000009230 tx=0x7fa0000058d0 comp rx=0 tx=0).stop 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010100e20 msgr2=0x7fa010197220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010100e20 0x7fa010197220 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00b4a0 tx=0x7fa00c00b960 comp rx=0 tx=0).stop 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 shutdown_connections 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f9ffc0a3430 0x7f9ffc0a58f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa010102ef0 0x7fa010197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010102570 0x7fa010197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 --2- 172.21.15.28:0/699903212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010100e20 0x7fa010197220 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 >> 172.21.15.28:0/699903212 conn(0x7fa0100fc7d0 msgr2=0x7fa0100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 shutdown_connections 2024-09-15T22:08:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:12.522+0000 7fa017899640 1 -- 172.21.15.28:0/699903212 wait complete. 2024-09-15T22:08:12.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:12 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/699903212' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T22:08:12.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:12 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/699903212' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T22:08:13.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:12 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/699903212' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T22:08:13.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-15T22:08:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:13.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:13 smithi028 ceph-mon[30969]: pgmap v337: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 572 B/s wr, 11 op/s 2024-09-15T22:08:13.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:13 smithi028 ceph-mon[30969]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T22:08:13.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:13 smithi028 ceph-mon[30969]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T22:08:13.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:13 smithi028 ceph-mon[30969]: Cluster is now healthy 2024-09-15T22:08:13.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:13 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/699903212' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T22:08:13.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:13 smithi028 ceph-mon[30969]: fsmap 12 up:standby 2024-09-15T22:08:13.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:13 smithi203 ceph-mon[35546]: pgmap v337: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 572 B/s wr, 11 op/s 2024-09-15T22:08:13.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:13 smithi203 ceph-mon[35546]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T22:08:13.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:13 smithi203 ceph-mon[35546]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T22:08:13.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:13 smithi203 ceph-mon[35546]: Cluster is now healthy 2024-09-15T22:08:13.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:13 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/699903212' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T22:08:13.969 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:13 smithi203 ceph-mon[35546]: fsmap 12 up:standby 2024-09-15T22:08:14.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:13 smithi174 ceph-mon[35244]: pgmap v337: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 572 B/s wr, 11 op/s 2024-09-15T22:08:14.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:13 smithi174 ceph-mon[35244]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T22:08:14.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:13 smithi174 ceph-mon[35244]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T22:08:14.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:13 smithi174 ceph-mon[35244]: Cluster is now healthy 2024-09-15T22:08:14.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:13 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/699903212' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T22:08:14.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:13 smithi174 ceph-mon[35244]: fsmap 12 up:standby 2024-09-15T22:08:15.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:15 smithi028 ceph-mon[30969]: pgmap v338: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 9 op/s 2024-09-15T22:08:15.968 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:15 smithi203 ceph-mon[35546]: pgmap v338: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 9 op/s 2024-09-15T22:08:16.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:15 smithi174 ceph-mon[35244]: pgmap v338: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 9 op/s 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 -- 172.21.15.28:0/1592199446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 msgr2=0x7f484c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 --2- 172.21.15.28:0/1592199446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c107b80 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4840009930 tx=0x7f484002a440 comp rx=0 tx=0).stop 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 -- 172.21.15.28:0/1592199446 shutdown_connections 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 --2- 172.21.15.28:0/1592199446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f484c108290 0x7f484c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 --2- 172.21.15.28:0/1592199446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c107b80 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 --2- 172.21.15.28:0/1592199446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f484c102d90 0x7f484c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 -- 172.21.15.28:0/1592199446 >> 172.21.15.28:0/1592199446 conn(0x7f484c0fc7d0 msgr2=0x7f484c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 -- 172.21.15.28:0/1592199446 shutdown_connections 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.316+0000 7f485133a640 1 -- 172.21.15.28:0/1592199446 wait complete. 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 Processor -- start 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 -- start start 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f484c102d90 0x7f484c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f484c108290 0x7f484c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f484c1120f0 con 0x7f484c103740 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f484c111f70 con 0x7f484c108290 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f485133a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f484c112270 con 0x7f484c102d90 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f484a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f484a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44174/0 (socket says 172.21.15.28:44174) 2024-09-15T22:08:16.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f484a7fc640 1 -- 172.21.15.28:0/2323415964 learned_addr learned my addr 172.21.15.28:0/2323415964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.317+0000 7f484affd640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f484c102d90 0x7f484c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484b7fe640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f484c108290 0x7f484c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484a7fc640 1 -- 172.21.15.28:0/2323415964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f484c102d90 msgr2=0x7f484c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484a7fc640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f484c102d90 0x7f484c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484a7fc640 1 -- 172.21.15.28:0/2323415964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f484c108290 msgr2=0x7f484c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484a7fc640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f484c108290 0x7f484c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484a7fc640 1 -- 172.21.15.28:0/2323415964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484c19d620 con 0x7f484c103740 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484b7fe640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f484c108290 0x7f484c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f484a7fc640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c198bb0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4840005940 tx=0x7f48400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f4833fff640 1 -- 172.21.15.28:0/2323415964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484003f030 con 0x7f484c103740 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f4833fff640 1 -- 172.21.15.28:0/2323415964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4840003ce0 con 0x7f484c103740 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484c19d8b0 con 0x7f484c103740 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f4833fff640 1 -- 172.21.15.28:0/2323415964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4840035070 con 0x7f484c103740 2024-09-15T22:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.318+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f484c1a51f0 con 0x7f484c103740 2024-09-15T22:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.319+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f484c06a7f0 con 0x7f484c103740 2024-09-15T22:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.321+0000 7f4833fff640 1 -- 172.21.15.28:0/2323415964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7f4840042030 con 0x7f484c103740 2024-09-15T22:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.321+0000 7f4833fff640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48340a33b0 0x7f48340a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.321+0000 7f484affd640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48340a33b0 0x7f48340a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.321+0000 7f4833fff640 1 -- 172.21.15.28:0/2323415964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9897+0+0 (secure 0 0 0) 0x7f48400f0a00 con 0x7f484c103740 2024-09-15T22:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.322+0000 7f484affd640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48340a33b0 0x7f48340a5870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f483c005fd0 tx=0x7f483c00b390 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.324+0000 7f4833fff640 1 -- 172.21.15.28:0/2323415964 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f48400bce80 con 0x7f484c103740 2024-09-15T22:08:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.503+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f484c103170 con 0x7f484c103740 2024-09-15T22:08:16.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.925+0000 7f4833fff640 1 -- 172.21.15.28:0/2323415964 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) ==== 181+0+0 (secure 0 0 0) 0x7f484c103170 con 0x7f484c103740 2024-09-15T22:08:16.928 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-15T22:08:16.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48340a33b0 msgr2=0x7f48340a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:16.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48340a33b0 0x7f48340a5870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f483c005fd0 tx=0x7f483c00b390 comp rx=0 tx=0).stop 2024-09-15T22:08:16.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 msgr2=0x7f484c198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:16.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c198bb0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f4840005940 tx=0x7f48400042f0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 shutdown_connections 2024-09-15T22:08:16.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7f48340a33b0 0x7f48340a5870 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f484c108290 0x7f484c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f484c103740 0x7f484c198bb0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 --2- 172.21.15.28:0/2323415964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f484c102d90 0x7f484c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 >> 172.21.15.28:0/2323415964 conn(0x7f484c0fc7d0 msgr2=0x7f484c105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.928+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 shutdown_connections 2024-09-15T22:08:16.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:16.929+0000 7f485133a640 1 -- 172.21.15.28:0/2323415964 wait complete. 2024-09-15T22:08:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:16 smithi203 ceph-mon[35546]: osdmap e102: 12 total, 12 up, 12 in 2024-09-15T22:08:17.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:16 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2323415964' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:17.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:16 smithi174 ceph-mon[35244]: osdmap e102: 12 total, 12 up, 12 in 2024-09-15T22:08:17.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:16 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2323415964' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:17.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:16 smithi028 ceph-mon[30969]: osdmap e102: 12 total, 12 up, 12 in 2024-09-15T22:08:17.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:16 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2323415964' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:17.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-15T22:08:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:17 smithi028 ceph-mon[30969]: pgmap v340: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 127 B/s wr, 4 op/s 2024-09-15T22:08:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:17 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2323415964' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:17 smithi028 ceph-mon[30969]: osdmap e103: 12 total, 12 up, 12 in 2024-09-15T22:08:18.180 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:17 smithi174 ceph-mon[35244]: pgmap v340: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 127 B/s wr, 4 op/s 2024-09-15T22:08:18.180 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:17 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2323415964' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:18.180 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:17 smithi174 ceph-mon[35244]: osdmap e103: 12 total, 12 up, 12 in 2024-09-15T22:08:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:17 smithi203 ceph-mon[35546]: pgmap v340: 129 pgs: 129 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 127 B/s wr, 4 op/s 2024-09-15T22:08:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:17 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2323415964' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:18.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:17 smithi203 ceph-mon[35546]: osdmap e103: 12 total, 12 up, 12 in 2024-09-15T22:08:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.644+0000 7fce8fbad640 1 -- 172.21.15.28:0/1281793319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88106f10 msgr2=0x7fce8810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.644+0000 7fce8fbad640 1 --2- 172.21.15.28:0/1281793319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88106f10 0x7fce8810ea90 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fce78009ca0 tx=0x7fce7802a8c0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.644+0000 7fce8fbad640 1 -- 172.21.15.28:0/1281793319 shutdown_connections 2024-09-15T22:08:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.644+0000 7fce8fbad640 1 --2- 172.21.15.28:0/1281793319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88106f10 0x7fce8810ea90 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.644+0000 7fce8fbad640 1 --2- 172.21.15.28:0/1281793319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce88102790 0x7fce881067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.644+0000 7fce8fbad640 1 --2- 172.21.15.28:0/1281793319 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce88101de0 0x7fce881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 -- 172.21.15.28:0/1281793319 >> 172.21.15.28:0/1281793319 conn(0x7fce880fd8c0 msgr2=0x7fce880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 -- 172.21.15.28:0/1281793319 shutdown_connections 2024-09-15T22:08:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 -- 172.21.15.28:0/1281793319 wait complete. 2024-09-15T22:08:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 Processor -- start 2024-09-15T22:08:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 -- start start 2024-09-15T22:08:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88101de0 0x7fce881983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce88102790 0x7fce88198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce88106f10 0x7fce8819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce88111fe0 con 0x7fce88101de0 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.645+0000 7fce8fbad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fce88111e60 con 0x7fce88106f10 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8fbad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fce88112160 con 0x7fce88102790 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88101de0 0x7fce881983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88101de0 0x7fce881983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44198/0 (socket says 172.21.15.28:44198) 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 -- 172.21.15.28:0/2174132626 learned_addr learned my addr 172.21.15.28:0/2174132626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 -- 172.21.15.28:0/2174132626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce88102790 msgr2=0x7fce88198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce88102790 0x7fce88198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 -- 172.21.15.28:0/2174132626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce88106f10 msgr2=0x7fce8819cc90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce88106f10 0x7fce8819cc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 -- 172.21.15.28:0/2174132626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce8819d370 con 0x7fce88101de0 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8d949640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88101de0 0x7fce881983c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fce8000b4a0 tx=0x7fce8000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce86ffd640 1 -- 172.21.15.28:0/2174132626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce80014070 con 0x7fce88101de0 2024-09-15T22:08:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce8819d660 con 0x7fce88101de0 2024-09-15T22:08:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce86ffd640 1 -- 172.21.15.28:0/2174132626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce80010af0 con 0x7fce88101de0 2024-09-15T22:08:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.646+0000 7fce86ffd640 1 -- 172.21.15.28:0/2174132626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce80017d10 con 0x7fce88101de0 2024-09-15T22:08:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.647+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce881a4f40 con 0x7fce88101de0 2024-09-15T22:08:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.647+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce881021c0 con 0x7fce88101de0 2024-09-15T22:08:18.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.649+0000 7fce86ffd640 1 -- 172.21.15.28:0/2174132626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fce80017070 con 0x7fce88101de0 2024-09-15T22:08:18.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.649+0000 7fce86ffd640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fce700a33b0 0x7fce700a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:18.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.649+0000 7fce8d148640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fce700a33b0 0x7fce700a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:18.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.650+0000 7fce86ffd640 1 -- 172.21.15.28:0/2174132626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9284+0+0 (secure 0 0 0) 0x7fce800cf7a0 con 0x7fce88101de0 2024-09-15T22:08:18.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.650+0000 7fce8d148640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fce700a33b0 0x7fce700a5870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fce88108880 tx=0x7fce7c005c70 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.652+0000 7fce86ffd640 1 -- 172.21.15.28:0/2174132626 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fce8009bf90 con 0x7fce88101de0 2024-09-15T22:08:18.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.829+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fce88199720 con 0x7fce88101de0 2024-09-15T22:08:18.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.931+0000 7fce86ffd640 1 -- 172.21.15.28:0/2174132626 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) ==== 169+0+0 (secure 0 0 0) 0x7fce8009b890 con 0x7fce88101de0 2024-09-15T22:08:18.934 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-15T22:08:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.934+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fce700a33b0 msgr2=0x7fce700a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.934+0000 7fce8fbad640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fce700a33b0 0x7fce700a5870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fce88108880 tx=0x7fce7c005c70 comp rx=0 tx=0).stop 2024-09-15T22:08:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.934+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88101de0 msgr2=0x7fce881983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:18.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.934+0000 7fce8fbad640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88101de0 0x7fce881983c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fce8000b4a0 tx=0x7fce8000b960 comp rx=0 tx=0).stop 2024-09-15T22:08:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.935+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 shutdown_connections 2024-09-15T22:08:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.935+0000 7fce8fbad640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fce700a33b0 0x7fce700a5870 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.935+0000 7fce8fbad640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce88106f10 0x7fce8819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.935+0000 7fce8fbad640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce88102790 0x7fce88198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.935+0000 7fce8fbad640 1 --2- 172.21.15.28:0/2174132626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce88101de0 0x7fce881983c0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.935+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 >> 172.21.15.28:0/2174132626 conn(0x7fce880fd8c0 msgr2=0x7fce880ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:18.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.935+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 shutdown_connections 2024-09-15T22:08:18.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:18.936+0000 7fce8fbad640 1 -- 172.21.15.28:0/2174132626 wait complete. 2024-09-15T22:08:19.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:18 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2174132626' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:18 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2174132626' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:19.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:18 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2174132626' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:19.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid e78b7764-73ab-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-15T22:08:19.840 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T22:08:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:19 smithi203 ceph-mon[35546]: pgmap v342: 97 pgs: 97 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T22:08:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:19 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/2174132626' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:20.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:19 smithi203 ceph-mon[35546]: osdmap e104: 12 total, 12 up, 12 in 2024-09-15T22:08:20.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:19 smithi028 ceph-mon[30969]: pgmap v342: 97 pgs: 97 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T22:08:20.228 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:19 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/2174132626' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:20.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:19 smithi028 ceph-mon[30969]: osdmap e104: 12 total, 12 up, 12 in 2024-09-15T22:08:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:19 smithi174 ceph-mon[35244]: pgmap v342: 97 pgs: 97 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T22:08:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:19 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/2174132626' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:20.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:19 smithi174 ceph-mon[35244]: osdmap e104: 12 total, 12 up, 12 in 2024-09-15T22:08:20.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.312+0000 7fba3d2b8640 1 -- 172.21.15.28:0/2337555110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38108290 msgr2=0x7fba3810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:20.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.312+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/2337555110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38108290 0x7fba3810ec30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fba30009cd0 tx=0x7fba3002a8e0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 -- 172.21.15.28:0/2337555110 shutdown_connections 2024-09-15T22:08:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/2337555110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38108290 0x7fba3810ec30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/2337555110 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba38103740 0x7fba38107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/2337555110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba38102d90 0x7fba38103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 -- 172.21.15.28:0/2337555110 >> 172.21.15.28:0/2337555110 conn(0x7fba380fc7d0 msgr2=0x7fba380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:20.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 -- 172.21.15.28:0/2337555110 shutdown_connections 2024-09-15T22:08:20.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 -- 172.21.15.28:0/2337555110 wait complete. 2024-09-15T22:08:20.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 Processor -- start 2024-09-15T22:08:20.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.313+0000 7fba3d2b8640 1 -- start start 2024-09-15T22:08:20.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba3d2b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38102d90 0x7fba38198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:20.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba3d2b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba38103740 0x7fba38198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:20.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba3d2b8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba38108290 0x7fba3819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:20.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba3d2b8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba38112090 con 0x7fba38102d90 2024-09-15T22:08:20.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba3d2b8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fba38111f10 con 0x7fba38108290 2024-09-15T22:08:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba3d2b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba38112210 con 0x7fba38103740 2024-09-15T22:08:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38102d90 0x7fba38198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba367fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba38103740 0x7fba38198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38102d90 0x7fba38198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44210/0 (socket says 172.21.15.28:44210) 2024-09-15T22:08:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba36ffd640 1 -- 172.21.15.28:0/420361660 learned_addr learned my addr 172.21.15.28:0/420361660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-15T22:08:20.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba377fe640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba38108290 0x7fba3819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:20.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba367fc640 1 -- 172.21.15.28:0/420361660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba38108290 msgr2=0x7fba3819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:20.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba367fc640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba38108290 0x7fba3819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba367fc640 1 -- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38102d90 msgr2=0x7fba38198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:20.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba367fc640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38102d90 0x7fba38198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba367fc640 1 -- 172.21.15.28:0/420361660 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3819d530 con 0x7fba38103740 2024-09-15T22:08:20.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba367fc640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba38103740 0x7fba38198ac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00e9c0 tx=0x7fba2c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:20.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.314+0000 7fba1ffff640 1 -- 172.21.15.28:0/420361660 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00ccc0 con 0x7fba38103740 2024-09-15T22:08:20.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.315+0000 7fba1ffff640 1 -- 172.21.15.28:0/420361660 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba2c005020 con 0x7fba38103740 2024-09-15T22:08:20.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.315+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3819d820 con 0x7fba38103740 2024-09-15T22:08:20.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.315+0000 7fba1ffff640 1 -- 172.21.15.28:0/420361660 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c004a00 con 0x7fba38103740 2024-09-15T22:08:20.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.315+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba381a5100 con 0x7fba38103740 2024-09-15T22:08:20.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.315+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba38103170 con 0x7fba38103740 2024-09-15T22:08:20.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.317+0000 7fba1ffff640 1 -- 172.21.15.28:0/420361660 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138486+0+0 (secure 0 0 0) 0x7fba2c0051d0 con 0x7fba38103740 2024-09-15T22:08:20.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.318+0000 7fba1ffff640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fba200a33b0 0x7fba200a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T22:08:20.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.318+0000 7fba36ffd640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fba200a33b0 0x7fba200a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T22:08:20.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.318+0000 7fba1ffff640 1 -- 172.21.15.28:0/420361660 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8754+0+0 (secure 0 0 0) 0x7fba2c0cfde0 con 0x7fba38103740 2024-09-15T22:08:20.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.318+0000 7fba36ffd640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fba200a33b0 0x7fba200a5870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba28005fd0 tx=0x7fba2800b480 comp rx=0 tx=0).ready entity=mgr.15330 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T22:08:20.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.321+0000 7fba1ffff640 1 -- 172.21.15.28:0/420361660 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fba2c09c700 con 0x7fba38103740 2024-09-15T22:08:20.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.499+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fba381998e0 con 0x7fba38103740 2024-09-15T22:08:20.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.948+0000 7fba1ffff640 1 -- 172.21.15.28:0/420361660 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v105) ==== 178+0+0 (secure 0 0 0) 0x7fba381998e0 con 0x7fba38103740 2024-09-15T22:08:20.952 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fba200a33b0 msgr2=0x7fba200a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fba200a33b0 0x7fba200a5870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba28005fd0 tx=0x7fba2800b480 comp rx=0 tx=0).stop 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba38103740 msgr2=0x7fba38198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba38103740 0x7fba38198ac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00e9c0 tx=0x7fba2c00ee80 comp rx=0 tx=0).stop 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 shutdown_connections 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:6800/1380427157,v1:172.21.15.28:6801/1380427157] conn(0x7fba200a33b0 0x7fba200a5870 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fba28005fd0 tx=0x7fba2800b480 comp rx=0 tx=0).stop 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba38108290 0x7fba3819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba38103740 0x7fba38198ac0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 --2- 172.21.15.28:0/420361660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba38102d90 0x7fba38198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T22:08:20.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.952+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 >> 172.21.15.28:0/420361660 conn(0x7fba380fc7d0 msgr2=0x7fba38106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T22:08:20.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.953+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 shutdown_connections 2024-09-15T22:08:20.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-15T22:08:20.953+0000 7fba3d2b8640 1 -- 172.21.15.28:0/420361660 wait complete. 2024-09-15T22:08:21.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:20 smithi203 ceph-mon[35546]: from='client.? 172.21.15.28:0/420361660' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:21.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:20 smithi203 ceph-mon[35546]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:21.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:20 smithi174 ceph-mon[35244]: from='client.? 172.21.15.28:0/420361660' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:21.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:20 smithi174 ceph-mon[35244]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:21.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:20 smithi028 ceph-mon[30969]: from='client.? 172.21.15.28:0/420361660' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:21.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:20 smithi028 ceph-mon[30969]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T22:08:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:21 smithi203 ceph-mon[35546]: pgmap v344: 33 pgs: 33 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T22:08:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:21 smithi203 ceph-mon[35546]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:22.218 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:21 smithi203 ceph-mon[35546]: osdmap e105: 12 total, 12 up, 12 in 2024-09-15T22:08:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:21 smithi174 ceph-mon[35244]: pgmap v344: 33 pgs: 33 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T22:08:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:21 smithi174 ceph-mon[35244]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:22.260 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:21 smithi174 ceph-mon[35244]: osdmap e105: 12 total, 12 up, 12 in 2024-09-15T22:08:22.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:21 smithi028 ceph-mon[30969]: pgmap v344: 33 pgs: 33 active+clean; 15 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T22:08:22.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:21 smithi028 ceph-mon[30969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T22:08:22.276 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:21 smithi028 ceph-mon[30969]: osdmap e105: 12 total, 12 up, 12 in 2024-09-15T22:08:22.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T22:08:22.362 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T22:08:22.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T22:08:22.599 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-15T22:08:22.599 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T22:08:22.638 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T22:08:22.675 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T22:08:22.705 INFO:tasks.cephadm:Stopping all daemons... 2024-09-15T22:08:22.705 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-15T22:08:22.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a 2024-09-15T22:08:23.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:22 smithi028 systemd[1]: Stopping Ceph mon.a for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:23.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:23 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a[30965]: 2024-09-15T22:08:23.222+0000 7fd7665f1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:08:23.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:23 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a[30965]: 2024-09-15T22:08:23.222+0000 7fd7665f1640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-15T22:08:23.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:23 smithi028 podman[111868]: 2024-09-15 22:08:23.43079433 +0000 UTC m=+0.363656760 container died ac21064d7e14df08b7a48a1791ce380ea06ec39fb586131c2b056ccb7fe24f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-09-15T22:08:23.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:23 smithi028 podman[111868]: 2024-09-15 22:08:23.821401604 +0000 UTC m=+0.754264031 container cleanup ac21064d7e14df08b7a48a1791ce380ea06ec39fb586131c2b056ccb7fe24f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:08:23.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:23 smithi028 bash[111868]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a 2024-09-15T22:08:24.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service' 2024-09-15T22:08:24.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:24 smithi028 podman[111880]: 2024-09-15 22:08:24.140131478 +0000 UTC m=+0.706024833 container remove ac21064d7e14df08b7a48a1791ce380ea06ec39fb586131c2b056ccb7fe24f4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True) 2024-09-15T22:08:24.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:24 smithi028 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-15T22:08:24.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:24 smithi028 systemd[1]: Stopped Ceph mon.a for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T22:08:24.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Sep 15 22:08:24 smithi028 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.a.service: Consumed 22.220s CPU time. 2024-09-15T22:08:24.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:24.430 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-15T22:08:24.431 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-15T22:08:24.431 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.b 2024-09-15T22:08:24.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:24 smithi174 systemd[1]: Stopping Ceph mon.b for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:27 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-b[35240]: 2024-09-15T22:08:27.288+0000 7f1ee3b94640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:08:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:27 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-b[35240]: 2024-09-15T22:08:27.288+0000 7f1ee3b94640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-15T22:08:27.760 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:27 smithi174 podman[72064]: 2024-09-15 22:08:27.462503748 +0000 UTC m=+1.834829389 container died 210357fd4b1e686dab048d3df132f815b418165ecc734f961df06859e3a6425d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-15T22:08:33.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:32 smithi174 podman[72064]: 2024-09-15 22:08:32.539354712 +0000 UTC m=+6.911680350 container cleanup 210357fd4b1e686dab048d3df132f815b418165ecc734f961df06859e3a6425d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-b, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:08:33.010 INFO:journalctl@ceph.mon.b.smithi174.stdout:Sep 15 22:08:32 smithi174 bash[72064]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-b 2024-09-15T22:08:33.334 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.b.service' 2024-09-15T22:08:33.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:33.381 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-15T22:08:33.381 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-15T22:08:33.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.c 2024-09-15T22:08:33.718 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:33 smithi203 systemd[1]: Stopping Ceph mon.c for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:34.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:33 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-c[35541]: 2024-09-15T22:08:33.817+0000 7fda5f379640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:08:34.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:33 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-c[35541]: 2024-09-15T22:08:33.817+0000 7fda5f379640 -1 mon.c@1(probing) e3 *** Got Signal Terminated *** 2024-09-15T22:08:34.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:33 smithi203 podman[72174]: 2024-09-15 22:08:33.87897142 +0000 UTC m=+0.189845134 container died af6e405c8f2fc797c80f24c330a4ea038e03347b240c1da49413070d088517f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e) 2024-09-15T22:08:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:34 smithi203 podman[72174]: 2024-09-15 22:08:34.310108892 +0000 UTC m=+0.620982587 container cleanup af6e405c8f2fc797c80f24c330a4ea038e03347b240c1da49413070d088517f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-15T22:08:34.468 INFO:journalctl@ceph.mon.c.smithi203.stdout:Sep 15 22:08:34 smithi203 bash[72174]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mon-c 2024-09-15T22:08:34.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mon.c.service' 2024-09-15T22:08:34.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:34.849 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-15T22:08:34.849 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-15T22:08:34.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.x 2024-09-15T22:08:35.171 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 22:08:34 smithi028 systemd[1]: Stopping Ceph mgr.x for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:36.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 22:08:36 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x[31198]: 2024-09-15T22:08:36.563+0000 7fb9a3d72640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:08:36.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 22:08:36 smithi028 podman[111985]: 2024-09-15 22:08:36.604891873 +0000 UTC m=+1.557377149 container died c43e87b526bf4acc75aa920549c4f22375d342913565265d1f35a09f2a907315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-15T22:08:38.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 22:08:38 smithi028 podman[111985]: 2024-09-15 22:08:38.796305628 +0000 UTC m=+3.748790890 container cleanup c43e87b526bf4acc75aa920549c4f22375d342913565265d1f35a09f2a907315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-15T22:08:38.921 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Sep 15 22:08:38 smithi028 bash[111985]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-x 2024-09-15T22:08:39.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-15T22:08:39.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:39.256 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-15T22:08:39.256 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-15T22:08:39.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.y 2024-09-15T22:08:39.599 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 22:08:39 smithi174 systemd[1]: Stopping Ceph mgr.y for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:39.921 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 22:08:39 smithi174 podman[72185]: 2024-09-15 22:08:39.59807116 +0000 UTC m=+0.142362941 container died 539fe850ed94c0f77d00967966b124d01d33e21eabc78a0d3b9e6af3dcf295d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:08:40.260 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 22:08:40 smithi174 podman[72185]: 2024-09-15 22:08:40.028030137 +0000 UTC m=+0.572321913 container cleanup 539fe850ed94c0f77d00967966b124d01d33e21eabc78a0d3b9e6af3dcf295d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:08:40.260 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 22:08:40 smithi174 bash[72185]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y 2024-09-15T22:08:40.514 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 22:08:40 smithi174 podman[72197]: 2024-09-15 22:08:40.262863165 +0000 UTC m=+0.659036063 container remove 539fe850ed94c0f77d00967966b124d01d33e21eabc78a0d3b9e6af3dcf295d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-15T22:08:40.514 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Sep 15 22:08:40 smithi174 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-15T22:08:40.516 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-15T22:08:40.552 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:40.552 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-15T22:08:40.553 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-15T22:08:40.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.z 2024-09-15T22:08:40.906 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:40 smithi203 systemd[1]: Stopping Ceph mgr.z for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:41.218 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:40 smithi203 podman[72296]: 2024-09-15 22:08:40.904889957 +0000 UTC m=+0.152605491 container died 26da5c3b0f95d7e4106d5441fc018cfc17c829ae14bd1a66ab74213183408ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0) 2024-09-15T22:08:41.570 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:41 smithi203 podman[72296]: 2024-09-15 22:08:41.333298634 +0000 UTC m=+0.581014168 container cleanup 26da5c3b0f95d7e4106d5441fc018cfc17c829ae14bd1a66ab74213183408ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-15T22:08:41.570 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:41 smithi203 bash[72296]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z 2024-09-15T22:08:41.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.z.service' 2024-09-15T22:08:41.827 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:41 smithi203 podman[72308]: 2024-09-15 22:08:41.569139513 +0000 UTC m=+0.663124214 container remove 26da5c3b0f95d7e4106d5441fc018cfc17c829ae14bd1a66ab74213183408ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:08:41.827 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:41 smithi203 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-09-15T22:08:41.827 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:41 smithi203 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.z.service: Failed with result 'exit-code'. 2024-09-15T22:08:41.827 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:41 smithi203 systemd[1]: Stopped Ceph mgr.z for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T22:08:41.827 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Sep 15 22:08:41 smithi203 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mgr.z.service: Consumed 11.527s CPU time. 2024-09-15T22:08:41.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:41.841 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-15T22:08:41.841 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-15T22:08:41.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.0 2024-09-15T22:08:42.169 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:41 smithi028 systemd[1]: Stopping Ceph osd.0 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:42.421 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:42 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0[44848]: 2024-09-15T22:08:42.167+0000 7f73f1c9d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:08:42.421 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:42 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0[44848]: 2024-09-15T22:08:42.167+0000 7f73f1c9d640 -1 osd.0 105 *** Got signal Terminated *** 2024-09-15T22:08:42.421 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:42 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0[44848]: 2024-09-15T22:08:42.167+0000 7f73f1c9d640 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:08:48.103 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:47 smithi028 podman[112106]: 2024-09-15 22:08:47.823645741 +0000 UTC m=+5.774829549 container died 72ea0fd6497628fc44e053b957733cab62c2cbc677ec42cc30202e0ea0445503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:08:48.379 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:48 smithi028 podman[112106]: 2024-09-15 22:08:48.168002557 +0000 UTC m=+6.119186366 container cleanup 72ea0fd6497628fc44e053b957733cab62c2cbc677ec42cc30202e0ea0445503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:08:48.379 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:48 smithi028 bash[112106]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0 2024-09-15T22:08:48.671 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:48 smithi028 podman[112123]: 2024-09-15 22:08:48.378161796 +0000 UTC m=+0.544455251 container remove 72ea0fd6497628fc44e053b957733cab62c2cbc677ec42cc30202e0ea0445503 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, io.buildah.version=1.37.2) 2024-09-15T22:08:49.041 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:48 smithi028 podman[112213]: 2024-09-15 22:08:48.686978977 +0000 UTC m=+0.016724635 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:08:49.041 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:48 smithi028 podman[112213]: 2024-09-15 22:08:48.812431736 +0000 UTC m=+0.142177393 container create 7697ecae8ea2260a97a532528d6e3c069bab8654273017732a249758cd42af38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-15T22:08:49.422 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:49 smithi028 podman[112213]: 2024-09-15 22:08:49.039925256 +0000 UTC m=+0.369670914 container init 7697ecae8ea2260a97a532528d6e3c069bab8654273017732a249758cd42af38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-15T22:08:49.422 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:49 smithi028 podman[112213]: 2024-09-15 22:08:49.043436542 +0000 UTC m=+0.373182200 container start 7697ecae8ea2260a97a532528d6e3c069bab8654273017732a249758cd42af38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:08:49.422 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:49 smithi028 podman[112213]: 2024-09-15 22:08:49.173438326 +0000 UTC m=+0.503184028 container attach 7697ecae8ea2260a97a532528d6e3c069bab8654273017732a249758cd42af38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-15T22:08:49.761 INFO:journalctl@ceph.osd.0.smithi028.stdout:Sep 15 22:08:49 smithi028 podman[112213]: 2024-09-15 22:08:49.458119144 +0000 UTC m=+0.787864809 container died 7697ecae8ea2260a97a532528d6e3c069bab8654273017732a249758cd42af38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-15T22:08:50.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.0.service' 2024-09-15T22:08:50.116 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:50.116 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-15T22:08:50.117 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-15T22:08:50.117 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.1 2024-09-15T22:08:50.510 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:50 smithi174 systemd[1]: Stopping Ceph osd.1 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:51.010 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:50 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1[39711]: 2024-09-15T22:08:50.573+0000 7fe092480640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:08:51.010 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:50 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1[39711]: 2024-09-15T22:08:50.573+0000 7fe092480640 -1 osd.1 105 *** Got signal Terminated *** 2024-09-15T22:08:51.010 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:50 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1[39711]: 2024-09-15T22:08:50.573+0000 7fe092480640 -1 osd.1 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:08:56.510 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:56 smithi174 podman[72312]: 2024-09-15 22:08:56.244300039 +0000 UTC m=+5.931897050 container died b9c849c2afe6553b9f27e12fb7cd37e22607f086d6b4a99198a7ffcaffdf4a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-15T22:08:56.943 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:56 smithi174 podman[72312]: 2024-09-15 22:08:56.657391174 +0000 UTC m=+6.344988193 container cleanup b9c849c2afe6553b9f27e12fb7cd37e22607f086d6b4a99198a7ffcaffdf4a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-15T22:08:56.943 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:56 smithi174 bash[72312]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1 2024-09-15T22:08:57.251 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:56 smithi174 podman[72340]: 2024-09-15 22:08:56.942390162 +0000 UTC m=+0.690854997 container remove b9c849c2afe6553b9f27e12fb7cd37e22607f086d6b4a99198a7ffcaffdf4a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:08:57.510 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:57 smithi174 podman[72430]: 2024-09-15 22:08:57.15299813 +0000 UTC m=+0.016148344 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:08:57.510 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:57 smithi174 podman[72430]: 2024-09-15 22:08:57.285244618 +0000 UTC m=+0.148394824 container create 28946cd3bdc0be801114848c7e522e725ec3cce5ae41405288865527c2c25b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-09-15T22:08:57.810 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:57 smithi174 podman[72430]: 2024-09-15 22:08:57.529622269 +0000 UTC m=+0.392772474 container init 28946cd3bdc0be801114848c7e522e725ec3cce5ae41405288865527c2c25b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2) 2024-09-15T22:08:57.810 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:57 smithi174 podman[72430]: 2024-09-15 22:08:57.532955273 +0000 UTC m=+0.396105485 container start 28946cd3bdc0be801114848c7e522e725ec3cce5ae41405288865527c2c25b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:08:57.810 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:57 smithi174 podman[72430]: 2024-09-15 22:08:57.637963652 +0000 UTC m=+0.501113866 container attach 28946cd3bdc0be801114848c7e522e725ec3cce5ae41405288865527c2c25b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:08:58.176 INFO:journalctl@ceph.osd.1.smithi174.stdout:Sep 15 22:08:57 smithi174 podman[72430]: 2024-09-15 22:08:57.809727858 +0000 UTC m=+0.672878064 container died 28946cd3bdc0be801114848c7e522e725ec3cce5ae41405288865527c2c25b10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-15T22:08:58.501 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.1.service' 2024-09-15T22:08:58.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:08:58.545 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-15T22:08:58.545 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-15T22:08:58.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.2 2024-09-15T22:08:58.932 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:08:58 smithi203 systemd[1]: Stopping Ceph osd.2 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:08:59.218 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:08:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2[40488]: 2024-09-15T22:08:58.937+0000 7f0647d03640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:08:59.218 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:08:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2[40488]: 2024-09-15T22:08:58.937+0000 7f0647d03640 -1 osd.2 105 *** Got signal Terminated *** 2024-09-15T22:08:59.218 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:08:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2[40488]: 2024-09-15T22:08:58.937+0000 7f0647d03640 -1 osd.2 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:04.807 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:04 smithi203 podman[72422]: 2024-09-15 22:09:04.541328984 +0000 UTC m=+5.794520581 container died c6ee242e6dbded3e3c345750def6a7a10265e935628a5a273433bd819840ba85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-15T22:09:05.208 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:04 smithi203 podman[72422]: 2024-09-15 22:09:04.914302642 +0000 UTC m=+6.167494236 container cleanup c6ee242e6dbded3e3c345750def6a7a10265e935628a5a273433bd819840ba85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-15T22:09:05.209 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:04 smithi203 bash[72422]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2 2024-09-15T22:09:05.468 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:05 smithi203 podman[72449]: 2024-09-15 22:09:05.207737591 +0000 UTC m=+0.661547253 container remove c6ee242e6dbded3e3c345750def6a7a10265e935628a5a273433bd819840ba85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-15T22:09:05.863 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:05 smithi203 podman[72538]: 2024-09-15 22:09:05.426728904 +0000 UTC m=+0.015872395 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:09:05.863 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:05 smithi203 podman[72538]: 2024-09-15 22:09:05.567196356 +0000 UTC m=+0.156339834 container create c933788b17b6327f0fbf06bf2f894bd40504e945b3b5262df773f26c0e0b56ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-15T22:09:05.863 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:05 smithi203 podman[72538]: 2024-09-15 22:09:05.861738731 +0000 UTC m=+0.450882209 container init c933788b17b6327f0fbf06bf2f894bd40504e945b3b5262df773f26c0e0b56ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:09:06.137 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:05 smithi203 podman[72538]: 2024-09-15 22:09:05.865147268 +0000 UTC m=+0.454290754 container start c933788b17b6327f0fbf06bf2f894bd40504e945b3b5262df773f26c0e0b56ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:09:06.138 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:05 smithi203 podman[72538]: 2024-09-15 22:09:05.961920602 +0000 UTC m=+0.551064097 container attach c933788b17b6327f0fbf06bf2f894bd40504e945b3b5262df773f26c0e0b56ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e) 2024-09-15T22:09:06.468 INFO:journalctl@ceph.osd.2.smithi203.stdout:Sep 15 22:09:06 smithi203 podman[72538]: 2024-09-15 22:09:06.137153632 +0000 UTC m=+0.726297113 container died c933788b17b6327f0fbf06bf2f894bd40504e945b3b5262df773f26c0e0b56ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-15T22:09:06.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.2.service' 2024-09-15T22:09:06.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:09:06.873 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-15T22:09:06.874 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-15T22:09:06.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.3 2024-09-15T22:09:07.171 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:06 smithi028 systemd[1]: Stopping Ceph osd.3 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:09:07.671 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:07 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3[51427]: 2024-09-15T22:09:07.235+0000 7fbc34855640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:09:07.671 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:07 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3[51427]: 2024-09-15T22:09:07.235+0000 7fbc34855640 -1 osd.3 105 *** Got signal Terminated *** 2024-09-15T22:09:07.671 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:07 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3[51427]: 2024-09-15T22:09:07.235+0000 7fbc34855640 -1 osd.3 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:13.073 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:12 smithi028 podman[112315]: 2024-09-15 22:09:12.760348116 +0000 UTC m=+5.676908765 container died e13beb93320fefbde570f07ec02b454c7d758ad436c3a6d38ee9d6aa1436b8a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-15T22:09:13.421 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:13 smithi028 podman[112315]: 2024-09-15 22:09:13.254939125 +0000 UTC m=+6.171499760 container cleanup e13beb93320fefbde570f07ec02b454c7d758ad436c3a6d38ee9d6aa1436b8a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-15T22:09:13.421 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:13 smithi028 bash[112315]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3 2024-09-15T22:09:13.884 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:13 smithi028 podman[112330]: 2024-09-15 22:09:13.531683115 +0000 UTC m=+0.769345440 container remove e13beb93320fefbde570f07ec02b454c7d758ad436c3a6d38ee9d6aa1436b8a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-15T22:09:14.171 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:13 smithi028 podman[112418]: 2024-09-15 22:09:13.786179504 +0000 UTC m=+0.016372882 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:09:14.171 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:13 smithi028 podman[112418]: 2024-09-15 22:09:13.916854671 +0000 UTC m=+0.147048039 container create 927ccf450317fa807dc8c8a2421c5e54203f95638e7d7ac728b5be4a350cb5a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-15T22:09:14.671 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:14 smithi028 podman[112418]: 2024-09-15 22:09:14.177128152 +0000 UTC m=+0.407321521 container init 927ccf450317fa807dc8c8a2421c5e54203f95638e7d7ac728b5be4a350cb5a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-15T22:09:14.671 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:14 smithi028 podman[112418]: 2024-09-15 22:09:14.180514184 +0000 UTC m=+0.410707553 container start 927ccf450317fa807dc8c8a2421c5e54203f95638e7d7ac728b5be4a350cb5a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-15T22:09:14.672 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:14 smithi028 podman[112418]: 2024-09-15 22:09:14.277210447 +0000 UTC m=+0.507403824 container attach 927ccf450317fa807dc8c8a2421c5e54203f95638e7d7ac728b5be4a350cb5a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:09:14.672 INFO:journalctl@ceph.osd.3.smithi028.stdout:Sep 15 22:09:14 smithi028 podman[112418]: 2024-09-15 22:09:14.376697009 +0000 UTC m=+0.606890378 container died 927ccf450317fa807dc8c8a2421c5e54203f95638e7d7ac728b5be4a350cb5a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-15T22:09:15.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.3.service' 2024-09-15T22:09:15.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:09:15.118 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-15T22:09:15.118 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-15T22:09:15.118 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.4 2024-09-15T22:09:15.438 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:15 smithi174 systemd[1]: Stopping Ceph osd.4 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:09:15.760 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4[46611]: 2024-09-15T22:09:15.436+0000 7f9312550640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:09:15.760 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4[46611]: 2024-09-15T22:09:15.436+0000 7f9312550640 -1 osd.4 105 *** Got signal Terminated *** 2024-09-15T22:09:15.760 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:15 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4[46611]: 2024-09-15T22:09:15.436+0000 7f9312550640 -1 osd.4 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:21.260 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:20 smithi174 podman[72533]: 2024-09-15 22:09:20.947937455 +0000 UTC m=+5.628186654 container died 5197eab51244e8664ad8eaa3fdf2a0a2af38c8361f2b89752744377deb497ebd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-15T22:09:21.614 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:21 smithi174 podman[72533]: 2024-09-15 22:09:21.377875997 +0000 UTC m=+6.058125185 container cleanup 5197eab51244e8664ad8eaa3fdf2a0a2af38c8361f2b89752744377deb497ebd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-15T22:09:21.614 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:21 smithi174 bash[72533]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4 2024-09-15T22:09:22.010 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:21 smithi174 podman[72549]: 2024-09-15 22:09:21.612843797 +0000 UTC m=+0.662245032 container remove 5197eab51244e8664ad8eaa3fdf2a0a2af38c8361f2b89752744377deb497ebd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-15T22:09:22.450 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:22 smithi174 podman[72652]: 2024-09-15 22:09:22.038350379 +0000 UTC m=+0.017676805 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:09:22.450 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:22 smithi174 podman[72652]: 2024-09-15 22:09:22.168316786 +0000 UTC m=+0.147643191 container create 75c6db9d279023658d57b2e6736d5d99782c3859e18aedbf13644d67935de771 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-15T22:09:22.760 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:22 smithi174 podman[72652]: 2024-09-15 22:09:22.449474595 +0000 UTC m=+0.428801003 container init 75c6db9d279023658d57b2e6736d5d99782c3859e18aedbf13644d67935de771 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:09:22.760 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:22 smithi174 podman[72652]: 2024-09-15 22:09:22.453004633 +0000 UTC m=+0.432331041 container start 75c6db9d279023658d57b2e6736d5d99782c3859e18aedbf13644d67935de771 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-15T22:09:22.761 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:22 smithi174 podman[72652]: 2024-09-15 22:09:22.549769548 +0000 UTC m=+0.529095954 container attach 75c6db9d279023658d57b2e6736d5d99782c3859e18aedbf13644d67935de771 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-15T22:09:22.761 INFO:journalctl@ceph.osd.4.smithi174.stdout:Sep 15 22:09:22 smithi174 podman[72652]: 2024-09-15 22:09:22.644039052 +0000 UTC m=+0.623365457 container died 75c6db9d279023658d57b2e6736d5d99782c3859e18aedbf13644d67935de771 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=squid-8921c8e, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:09:23.342 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.4.service' 2024-09-15T22:09:23.383 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:09:23.383 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-15T22:09:23.383 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-15T22:09:23.384 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.5 2024-09-15T22:09:23.705 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:23 smithi203 systemd[1]: Stopping Ceph osd.5 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:09:23.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:23 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5[47404]: 2024-09-15T22:09:23.702+0000 7f2b6e148640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:09:23.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:23 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5[47404]: 2024-09-15T22:09:23.702+0000 7f2b6e148640 -1 osd.5 105 *** Got signal Terminated *** 2024-09-15T22:09:23.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:23 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5[47404]: 2024-09-15T22:09:23.702+0000 7f2b6e148640 -1 osd.5 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:29.651 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:29 smithi203 podman[72640]: 2024-09-15 22:09:29.329428368 +0000 UTC m=+5.748764117 container died bc631adc11eba22127a790268c3f724b0d494ff1fb5210a400e47e3256defb53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-15T22:09:29.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:29 smithi203 podman[72640]: 2024-09-15 22:09:29.762125877 +0000 UTC m=+6.181461622 container cleanup bc631adc11eba22127a790268c3f724b0d494ff1fb5210a400e47e3256defb53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True) 2024-09-15T22:09:29.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:29 smithi203 bash[72640]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5 2024-09-15T22:09:30.447 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:30 smithi203 podman[72657]: 2024-09-15 22:09:30.00546603 +0000 UTC m=+0.667782898 container remove bc631adc11eba22127a790268c3f724b0d494ff1fb5210a400e47e3256defb53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:09:30.702 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:30 smithi203 podman[72755]: 2024-09-15 22:09:30.348981644 +0000 UTC m=+0.016658067 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:09:30.702 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:30 smithi203 podman[72755]: 2024-09-15 22:09:30.473136379 +0000 UTC m=+0.140812788 container create 5e5147fcd9a71eedfc754b992bfe7e80efc228201e63892a33b9f1875019c107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-8921c8e) 2024-09-15T22:09:30.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:30 smithi203 podman[72755]: 2024-09-15 22:09:30.701046746 +0000 UTC m=+0.368723155 container init 5e5147fcd9a71eedfc754b992bfe7e80efc228201e63892a33b9f1875019c107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-15T22:09:30.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:30 smithi203 podman[72755]: 2024-09-15 22:09:30.704876869 +0000 UTC m=+0.372553279 container start 5e5147fcd9a71eedfc754b992bfe7e80efc228201e63892a33b9f1875019c107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-15T22:09:30.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:30 smithi203 podman[72755]: 2024-09-15 22:09:30.809529954 +0000 UTC m=+0.477207531 container attach 5e5147fcd9a71eedfc754b992bfe7e80efc228201e63892a33b9f1875019c107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-15T22:09:30.968 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 15 22:09:30 smithi203 podman[72755]: 2024-09-15 22:09:30.890691084 +0000 UTC m=+0.558367507 container died 5e5147fcd9a71eedfc754b992bfe7e80efc228201e63892a33b9f1875019c107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-15T22:09:31.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.5.service' 2024-09-15T22:09:31.628 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:09:31.628 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-15T22:09:31.628 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-15T22:09:31.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.6 2024-09-15T22:09:31.921 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:31 smithi028 systemd[1]: Stopping Ceph osd.6 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:09:32.421 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6[57911]: 2024-09-15T22:09:31.957+0000 7fe26ab34640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:09:32.421 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6[57911]: 2024-09-15T22:09:31.957+0000 7fe26ab34640 -1 osd.6 105 *** Got signal Terminated *** 2024-09-15T22:09:32.422 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6[57911]: 2024-09-15T22:09:31.957+0000 7fe26ab34640 -1 osd.6 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:38.002 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:37 smithi028 podman[112522]: 2024-09-15 22:09:37.678821252 +0000 UTC m=+5.839686418 container died 25b3e578e2e486c6746131f9499a014259d779e8054f853874261fb070a31ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-15T22:09:38.411 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:38 smithi028 podman[112522]: 2024-09-15 22:09:38.208675773 +0000 UTC m=+6.369540929 container cleanup 25b3e578e2e486c6746131f9499a014259d779e8054f853874261fb070a31ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:09:38.411 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:38 smithi028 bash[112522]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6 2024-09-15T22:09:38.671 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:38 smithi028 podman[112538]: 2024-09-15 22:09:38.410068451 +0000 UTC m=+0.725927349 container remove 25b3e578e2e486c6746131f9499a014259d779e8054f853874261fb070a31ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=squid-8921c8e) 2024-09-15T22:09:39.131 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:38 smithi028 podman[112630]: 2024-09-15 22:09:38.738266051 +0000 UTC m=+0.016569225 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:09:39.131 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:38 smithi028 podman[112630]: 2024-09-15 22:09:38.852697979 +0000 UTC m=+0.131001186 container create a1268495633fe59589dcde68131b3d1f87869f63190e63997d475a297fe9c957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:09:39.131 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:39 smithi028 podman[112630]: 2024-09-15 22:09:39.130004291 +0000 UTC m=+0.408307467 container init a1268495633fe59589dcde68131b3d1f87869f63190e63997d475a297fe9c957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-15T22:09:39.421 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:39 smithi028 podman[112630]: 2024-09-15 22:09:39.133421749 +0000 UTC m=+0.411724914 container start a1268495633fe59589dcde68131b3d1f87869f63190e63997d475a297fe9c957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:09:39.422 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:39 smithi028 podman[112630]: 2024-09-15 22:09:39.213597696 +0000 UTC m=+0.491900876 container attach a1268495633fe59589dcde68131b3d1f87869f63190e63997d475a297fe9c957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-15T22:09:39.422 INFO:journalctl@ceph.osd.6.smithi028.stdout:Sep 15 22:09:39 smithi028 podman[112630]: 2024-09-15 22:09:39.334969986 +0000 UTC m=+0.613273152 container died a1268495633fe59589dcde68131b3d1f87869f63190e63997d475a297fe9c957 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:09:39.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.6.service' 2024-09-15T22:09:40.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:09:40.030 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-15T22:09:40.030 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-15T22:09:40.030 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.7 2024-09-15T22:09:40.415 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:40 smithi174 systemd[1]: Stopping Ceph osd.7 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:09:40.760 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:40 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7[53595]: 2024-09-15T22:09:40.413+0000 7f4aced03640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:09:40.760 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:40 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7[53595]: 2024-09-15T22:09:40.413+0000 7f4aced03640 -1 osd.7 105 *** Got signal Terminated *** 2024-09-15T22:09:40.760 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:40 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7[53595]: 2024-09-15T22:09:40.413+0000 7f4aced03640 -1 osd.7 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:46.260 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:45 smithi174 podman[72757]: 2024-09-15 22:09:45.976088657 +0000 UTC m=+5.751264578 container died 6a9447c583404432f88ae98cc8949d01672f5564b80b6bb104774fab0cc688c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:09:46.607 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:46 smithi174 podman[72757]: 2024-09-15 22:09:46.404614954 +0000 UTC m=+6.179790874 container cleanup 6a9447c583404432f88ae98cc8949d01672f5564b80b6bb104774fab0cc688c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-15T22:09:46.607 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:46 smithi174 bash[72757]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7 2024-09-15T22:09:47.010 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:46 smithi174 podman[72774]: 2024-09-15 22:09:46.606024777 +0000 UTC m=+0.622430087 container remove 6a9447c583404432f88ae98cc8949d01672f5564b80b6bb104774fab0cc688c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-15T22:09:47.326 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:47 smithi174 podman[72860]: 2024-09-15 22:09:46.915730695 +0000 UTC m=+0.015836697 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:09:47.327 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:47 smithi174 podman[72860]: 2024-09-15 22:09:47.023173684 +0000 UTC m=+0.123279687 container create 6c34418302a4cfc65710bdc3c6ce94ee13496f1959e868cc79603354bbe4830b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-15T22:09:47.760 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:47 smithi174 podman[72860]: 2024-09-15 22:09:47.325971923 +0000 UTC m=+0.426077926 container init 6c34418302a4cfc65710bdc3c6ce94ee13496f1959e868cc79603354bbe4830b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2) 2024-09-15T22:09:47.760 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:47 smithi174 podman[72860]: 2024-09-15 22:09:47.329459049 +0000 UTC m=+0.429565057 container start 6c34418302a4cfc65710bdc3c6ce94ee13496f1959e868cc79603354bbe4830b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-15T22:09:47.761 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:47 smithi174 podman[72860]: 2024-09-15 22:09:47.45129017 +0000 UTC m=+0.551396173 container attach 6c34418302a4cfc65710bdc3c6ce94ee13496f1959e868cc79603354bbe4830b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-15T22:09:47.761 INFO:journalctl@ceph.osd.7.smithi174.stdout:Sep 15 22:09:47 smithi174 podman[72860]: 2024-09-15 22:09:47.515304534 +0000 UTC m=+0.615410537 container died 6c34418302a4cfc65710bdc3c6ce94ee13496f1959e868cc79603354bbe4830b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-15T22:09:48.191 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.7.service' 2024-09-15T22:09:48.230 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:09:48.231 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-15T22:09:48.231 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-15T22:09:48.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.8 2024-09-15T22:09:48.468 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:48 smithi203 systemd[1]: Stopping Ceph osd.8 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:09:48.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:48 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:48.119+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:48.968 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:48 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T22:09:48.641+0000 7f4545db2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:09:48.968 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:48 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T22:09:48.641+0000 7f4545db2640 -1 osd.8 105 *** Got signal Terminated *** 2024-09-15T22:09:48.968 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:48 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T22:09:48.641+0000 7f4545db2640 -1 osd.8 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:49.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:48 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:48.702+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:49.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:49 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:49.120+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:50.111 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:49 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T22:09:49.823+0000 7f453e3c0640 -1 osd.8 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.847254+0000 front 2024-09-15T22:08:46.847091+0000 (oldest deadline 2024-09-15T22:09:49.746600+0000) 2024-09-15T22:09:50.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:49 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:49.739+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:50.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:50 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:50.109+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:51.081 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:50 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T22:09:50.784+0000 7f453e3c0640 -1 osd.8 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.847254+0000 front 2024-09-15T22:08:46.847091+0000 (oldest deadline 2024-09-15T22:09:49.746600+0000) 2024-09-15T22:09:51.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:50 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:50.736+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:51.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:51 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:51.080+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:52.082 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:51 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T22:09:51.750+0000 7f453e3c0640 -1 osd.8 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.847254+0000 front 2024-09-15T22:08:46.847091+0000 (oldest deadline 2024-09-15T22:09:49.746600+0000) 2024-09-15T22:09:52.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:51 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:51.733+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:52.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:52 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:52.081+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:53.068 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:52 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8[53884]: 2024-09-15T22:09:52.743+0000 7f453e3c0640 -1 osd.8 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.847254+0000 front 2024-09-15T22:08:46.847091+0000 (oldest deadline 2024-09-15T22:09:49.746600+0000) 2024-09-15T22:09:53.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:52 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:52.767+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:53.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:53 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:53.067+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:54.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:53 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:53.763+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:54.290 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:54 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:54.022+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:54.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:09:54 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:09:54.037+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:09:54.577 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:54 smithi203 podman[72859]: 2024-09-15 22:09:54.289212113 +0000 UTC m=+5.864278063 container died 31fed15665c45299cbe0e8f26ba018c7589b718266fc2ea24870d24a8c5236d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-15T22:09:54.968 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:54 smithi203 podman[72859]: 2024-09-15 22:09:54.683562742 +0000 UTC m=+6.258628687 container cleanup 31fed15665c45299cbe0e8f26ba018c7589b718266fc2ea24870d24a8c5236d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-15T22:09:54.968 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:54 smithi203 bash[72859]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8 2024-09-15T22:09:55.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:54 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:54.720+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:55.344 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:54 smithi203 podman[72875]: 2024-09-15 22:09:54.985086712 +0000 UTC m=+0.691252683 container remove 31fed15665c45299cbe0e8f26ba018c7589b718266fc2ea24870d24a8c5236d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:09:55.345 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:55.052+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:55.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:09:55 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:09:55.061+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:09:55.614 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:55 smithi203 podman[72962]: 2024-09-15 22:09:55.246260237 +0000 UTC m=+0.015590265 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:09:55.615 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:55 smithi203 podman[72962]: 2024-09-15 22:09:55.369310513 +0000 UTC m=+0.138640527 container create 159237998a47b71ea290b7198ceee2453dd76f309dbd247b930780695b3e5e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=) 2024-09-15T22:09:55.615 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:55 smithi203 podman[72962]: 2024-09-15 22:09:55.613697495 +0000 UTC m=+0.383027510 container init 159237998a47b71ea290b7198ceee2453dd76f309dbd247b930780695b3e5e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-09-15T22:09:55.968 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:55 smithi203 podman[72962]: 2024-09-15 22:09:55.617329845 +0000 UTC m=+0.386659865 container start 159237998a47b71ea290b7198ceee2453dd76f309dbd247b930780695b3e5e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-15T22:09:55.969 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:55 smithi203 podman[72962]: 2024-09-15 22:09:55.713701191 +0000 UTC m=+0.483031214 container attach 159237998a47b71ea290b7198ceee2453dd76f309dbd247b930780695b3e5e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8-deactivate, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:09:55.969 INFO:journalctl@ceph.osd.8.smithi203.stdout:Sep 15 22:09:55 smithi203 podman[72962]: 2024-09-15 22:09:55.80818009 +0000 UTC m=+0.577510116 container died 159237998a47b71ea290b7198ceee2453dd76f309dbd247b930780695b3e5e61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-15T22:09:56.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:55 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:55.699+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:56.468 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:56 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:56.021+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:56.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:09:56 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:09:56.085+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:09:56.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.8.service' 2024-09-15T22:09:56.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:09:56.558 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-15T22:09:56.558 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-15T22:09:56.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.9 2024-09-15T22:09:56.862 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:56 smithi028 systemd[1]: Stopping Ceph osd.9 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:09:56.862 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:56.721+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:56.862 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:56.860+0000 7fe0f109f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:09:56.862 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:56.860+0000 7fe0f109f640 -1 osd.9 105 *** Got signal Terminated *** 2024-09-15T22:09:56.863 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:56 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:56.860+0000 7fe0f109f640 -1 osd.9 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:09:57.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:56 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:56.978+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:57.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:09:57 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:09:57.105+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:09:58.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:57 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:57.687+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:58.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:57 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:57.948+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:58.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:09:58 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:09:58.129+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:09:59.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:58 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:58.729+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:09:59.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:58 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:58.729+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:53.011934+0000 front 2024-09-15T22:08:53.011696+0000 (oldest deadline 2024-09-15T22:09:57.711698+0000) 2024-09-15T22:09:59.217 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:58 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:58.924+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:09:59.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:09:59 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:09:59.088+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:09:59.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:09:59 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:09:59.088+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:00.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:59 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:59.742+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:10:00.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:09:59 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:09:59.742+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:53.011934+0000 front 2024-09-15T22:08:53.011696+0000 (oldest deadline 2024-09-15T22:09:57.711698+0000) 2024-09-15T22:10:00.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:09:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:09:59.926+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:00.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:00 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:00.065+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:00.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:00 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:00.065+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:01.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:10:00.706+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:10:01.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:00 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:10:00.706+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:53.011934+0000 front 2024-09-15T22:08:53.011696+0000 (oldest deadline 2024-09-15T22:09:57.711698+0000) 2024-09-15T22:10:01.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:00 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:00.899+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:01.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:01.015+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:01.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:01 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:01.015+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:02.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:01 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:10:01.715+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:45.510238+0000 front 2024-09-15T22:08:45.510151+0000 (oldest deadline 2024-09-15T22:09:48.410075+0000) 2024-09-15T22:10:02.171 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:01 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9[64443]: 2024-09-15T22:10:01.715+0000 7fe0e96ad640 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:53.011934+0000 front 2024-09-15T22:08:53.011696+0000 (oldest deadline 2024-09-15T22:09:57.711698+0000) 2024-09-15T22:10:02.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:01 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:01.866+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:02.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:02 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:01.999+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:02.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:02 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:01.999+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:02.944 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:02 smithi028 podman[112733]: 2024-09-15 22:10:02.670991337 +0000 UTC m=+5.897101970 container died 331bff8d956ae9e207e2b52b06b29e1a47c5109e83b88e1d55bc8ec10609bcff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-15T22:10:03.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:02 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:02.818+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:03.259 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:02 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:02.966+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:03.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:02 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:02.966+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:03.313 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 podman[112733]: 2024-09-15 22:10:03.077472764 +0000 UTC m=+6.303583428 container cleanup 331bff8d956ae9e207e2b52b06b29e1a47c5109e83b88e1d55bc8ec10609bcff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-15T22:10:03.313 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 bash[112733]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9 2024-09-15T22:10:03.630 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 podman[112749]: 2024-09-15 22:10:03.31208835 +0000 UTC m=+0.635254483 container remove 331bff8d956ae9e207e2b52b06b29e1a47c5109e83b88e1d55bc8ec10609bcff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-15T22:10:03.921 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 podman[112838]: 2024-09-15 22:10:03.532348282 +0000 UTC m=+0.016743290 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:10:03.922 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 podman[112838]: 2024-09-15 22:10:03.671141936 +0000 UTC m=+0.155536931 container create 894bc1f778896d6f179fa1e1f3999b659341be308f39dfd0028346102d39e41e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9-deactivate, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:10:03.922 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 podman[112838]: 2024-09-15 22:10:03.865089815 +0000 UTC m=+0.349484813 container init 894bc1f778896d6f179fa1e1f3999b659341be308f39dfd0028346102d39e41e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-15T22:10:03.922 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 podman[112838]: 2024-09-15 22:10:03.868607609 +0000 UTC m=+0.353002613 container start 894bc1f778896d6f179fa1e1f3999b659341be308f39dfd0028346102d39e41e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-15T22:10:04.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:03 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:03.804+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:04.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:03 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:03.988+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:04.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:03 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:03.988+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:04.361 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:03 smithi028 podman[112838]: 2024-09-15 22:10:03.948329951 +0000 UTC m=+0.432724960 container attach 894bc1f778896d6f179fa1e1f3999b659341be308f39dfd0028346102d39e41e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-15T22:10:04.362 INFO:journalctl@ceph.osd.9.smithi028.stdout:Sep 15 22:10:04 smithi028 podman[112838]: 2024-09-15 22:10:04.065997726 +0000 UTC m=+0.550392732 container died 894bc1f778896d6f179fa1e1f3999b659341be308f39dfd0028346102d39e41e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:10:04.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.9.service' 2024-09-15T22:10:04.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:04.776 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-15T22:10:04.776 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-15T22:10:04.776 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.10 2024-09-15T22:10:05.075 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:04 smithi174 systemd[1]: Stopping Ceph osd.10 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:05.075 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:04 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:04.971+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:05.075 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:04 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:04.971+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:05.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:04 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:04.800+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:05.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:05 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:05.073+0000 7f77aa3a6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:05.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:05 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:05.073+0000 7f77aa3a6640 -1 osd.10 105 *** Got signal Terminated *** 2024-09-15T22:10:05.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:05 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:05.073+0000 7f77aa3a6640 -1 osd.10 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:10:06.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:05 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:05.822+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:06.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:05 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:05.984+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:06.260 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:05 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:05.984+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:07.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:06 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:06.784+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:07.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:06 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:06.784+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:07.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:07 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:07.018+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:07.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:07 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:07.019+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:07.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:07 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:07.019+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:02.887876+0000 front 2024-09-15T22:09:02.888122+0000 (oldest deadline 2024-09-15T22:10:06.987842+0000) 2024-09-15T22:10:08.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:07 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:07.805+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:08.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:07 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:07.805+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:08.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:08 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:08.012+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:08.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:08 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:08.012+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:08.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:08 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:08.012+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:02.887876+0000 front 2024-09-15T22:09:02.888122+0000 (oldest deadline 2024-09-15T22:10:06.987842+0000) 2024-09-15T22:10:09.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:08 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:08.759+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:09.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:08 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:08.759+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:09.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:09 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:09.014+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:09.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:09 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:09.014+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:09.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:09 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:09.014+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:02.887876+0000 front 2024-09-15T22:09:02.888122+0000 (oldest deadline 2024-09-15T22:10:06.987842+0000) 2024-09-15T22:10:10.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:09 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:09.791+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:10.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:09 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:09.791+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:10.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:10.058+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:46.085089+0000 front 2024-09-15T22:08:46.085116+0000 (oldest deadline 2024-09-15T22:09:53.085372+0000) 2024-09-15T22:10:10.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:10.058+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.174:6804 osd.1 since back 2024-09-15T22:08:54.786108+0000 front 2024-09-15T22:08:54.786130+0000 (oldest deadline 2024-09-15T22:09:58.786693+0000) 2024-09-15T22:10:10.510 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:10 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10[60315]: 2024-09-15T22:10:10.058+0000 7f77a69bc640 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:02.887876+0000 front 2024-09-15T22:09:02.888122+0000 (oldest deadline 2024-09-15T22:10:06.987842+0000) 2024-09-15T22:10:11.010 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:10 smithi174 podman[72977]: 2024-09-15 22:10:10.756660925 +0000 UTC m=+5.783948043 container died 925ed160639716d33a63e2bfade091e77df56394b1a49f406a1a3ad9f4c75fd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-15T22:10:11.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:10.801+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:11.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:10 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:10.801+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:11.425 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:11 smithi174 podman[72977]: 2024-09-15 22:10:11.13940473 +0000 UTC m=+6.166691857 container cleanup 925ed160639716d33a63e2bfade091e77df56394b1a49f406a1a3ad9f4c75fd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-09-15T22:10:11.425 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:11 smithi174 bash[72977]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10 2024-09-15T22:10:11.734 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:11 smithi174 podman[72994]: 2024-09-15 22:10:11.423905986 +0000 UTC m=+0.661728606 container remove 925ed160639716d33a63e2bfade091e77df56394b1a49f406a1a3ad9f4c75fd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-15T22:10:12.010 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:11 smithi174 podman[73081]: 2024-09-15 22:10:11.635831503 +0000 UTC m=+0.016067585 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:10:12.010 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:11 smithi174 podman[73081]: 2024-09-15 22:10:11.757904698 +0000 UTC m=+0.138140818 container create 29c64838b17186958168ac2e76535b6a5dbda0b77fb8441d4cdeb4f63ba55cc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e) 2024-09-15T22:10:12.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:11 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:11.847+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:12.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:11 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:11.847+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:12.488 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:12 smithi174 podman[73081]: 2024-09-15 22:10:12.010058189 +0000 UTC m=+0.390294272 container init 29c64838b17186958168ac2e76535b6a5dbda0b77fb8441d4cdeb4f63ba55cc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-15T22:10:12.488 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:12 smithi174 podman[73081]: 2024-09-15 22:10:12.013371185 +0000 UTC m=+0.393607264 container start 29c64838b17186958168ac2e76535b6a5dbda0b77fb8441d4cdeb4f63ba55cc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0) 2024-09-15T22:10:12.488 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:12 smithi174 podman[73081]: 2024-09-15 22:10:12.093407019 +0000 UTC m=+0.473643108 container attach 29c64838b17186958168ac2e76535b6a5dbda0b77fb8441d4cdeb4f63ba55cc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-15T22:10:12.488 INFO:journalctl@ceph.osd.10.smithi174.stdout:Sep 15 22:10:12 smithi174 podman[73081]: 2024-09-15 22:10:12.199222896 +0000 UTC m=+0.579459015 container died 29c64838b17186958168ac2e76535b6a5dbda0b77fb8441d4cdeb4f63ba55cc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-15T22:10:12.813 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.10.service' 2024-09-15T22:10:12.853 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:12.854 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-15T22:10:12.854 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-15T22:10:12.854 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.11 2024-09-15T22:10:13.189 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:12 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:12.855+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:13.189 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:12 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:12.855+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:13.189 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:12 smithi203 systemd[1]: Stopping Ceph osd.11 for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:13.190 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:13 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:13.186+0000 7f46d624d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:13.190 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:13 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:13.186+0000 7f46d624d640 -1 osd.11 105 *** Got signal Terminated *** 2024-09-15T22:10:13.190 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:13 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:13.186+0000 7f46d624d640 -1 osd.11 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T22:10:14.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:13 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:13.816+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:14.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:13 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:13.816+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:15.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:14 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:14.856+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:15.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:14 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:14.856+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:16.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:15.857+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:16.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:15 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:15.857+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:17.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:16 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:16.894+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:17.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:16 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:16.894+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:17.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:16 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:16.894+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-15T22:09:10.579907+0000 front 2024-09-15T22:09:10.579820+0000 (oldest deadline 2024-09-15T22:10:15.879824+0000) 2024-09-15T22:10:18.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:17 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:17.864+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-09-15T22:08:42.675298+0000 front 2024-09-15T22:08:42.675341+0000 (oldest deadline 2024-09-15T22:09:47.975348+0000) 2024-09-15T22:10:18.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:17 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:17.864+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-09-15T22:09:00.578736+0000 front 2024-09-15T22:09:00.578649+0000 (oldest deadline 2024-09-15T22:10:06.478787+0000) 2024-09-15T22:10:18.218 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:17 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11[60602]: 2024-09-15T22:10:17.864+0000 7f46d2863640 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-09-15T22:09:10.579907+0000 front 2024-09-15T22:09:10.579820+0000 (oldest deadline 2024-09-15T22:10:15.879824+0000) 2024-09-15T22:10:19.023 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:18 smithi203 podman[73079]: 2024-09-15 22:10:18.735524935 +0000 UTC m=+5.681729631 container died 9ca081ec77beeb9aa797b117c126b60abdbaed1c00651ee8d6661c31cf3fb098 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-15T22:10:19.390 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:19 smithi203 podman[73079]: 2024-09-15 22:10:19.112703352 +0000 UTC m=+6.058908055 container cleanup 9ca081ec77beeb9aa797b117c126b60abdbaed1c00651ee8d6661c31cf3fb098 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e) 2024-09-15T22:10:19.390 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:19 smithi203 bash[73079]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11 2024-09-15T22:10:19.718 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:19 smithi203 podman[73096]: 2024-09-15 22:10:19.38888689 +0000 UTC m=+0.651709509 container remove 9ca081ec77beeb9aa797b117c126b60abdbaed1c00651ee8d6661c31cf3fb098 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-15T22:10:20.027 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:19 smithi203 podman[73179]: 2024-09-15 22:10:19.621402379 +0000 UTC m=+0.015787390 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T22:10:20.027 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:19 smithi203 podman[73179]: 2024-09-15 22:10:19.723093377 +0000 UTC m=+0.117478415 container create 47a1714ae5494ddea4bbd7d935720f3afe961514fa53f3b954aa4d2023cdbd37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-15T22:10:20.027 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:19 smithi203 podman[73179]: 2024-09-15 22:10:19.942215938 +0000 UTC m=+0.336600931 container init 47a1714ae5494ddea4bbd7d935720f3afe961514fa53f3b954aa4d2023cdbd37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11-deactivate, RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-15T22:10:20.027 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:19 smithi203 podman[73179]: 2024-09-15 22:10:19.945553935 +0000 UTC m=+0.339938928 container start 47a1714ae5494ddea4bbd7d935720f3afe961514fa53f3b954aa4d2023cdbd37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2) 2024-09-15T22:10:20.028 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:20 smithi203 podman[73179]: 2024-09-15 22:10:20.025426151 +0000 UTC m=+0.419811144 container attach 47a1714ae5494ddea4bbd7d935720f3afe961514fa53f3b954aa4d2023cdbd37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:10:20.421 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:20 smithi203 podman[73179]: 2024-09-15 22:10:20.133092053 +0000 UTC m=+0.527477058 container died 47a1714ae5494ddea4bbd7d935720f3afe961514fa53f3b954aa4d2023cdbd37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T22:10:20.681 INFO:journalctl@ceph.osd.11.smithi203.stdout:Sep 15 22:10:20 smithi203 podman[73202]: 2024-09-15 22:10:20.662608413 +0000 UTC m=+0.523569570 container remove 47a1714ae5494ddea4bbd7d935720f3afe961514fa53f3b954aa4d2023cdbd37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-15T22:10:20.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@osd.11.service' 2024-09-15T22:10:20.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:20.727 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-15T22:10:20.727 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-15T22:10:20.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.a 2024-09-15T22:10:21.031 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:20 smithi028 systemd[1]: Stopping Ceph mds.a for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:21.301 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a[71640]: 2024-09-15T22:10:21.028+0000 7f2978f59640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:21.301 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:21 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a[71640]: 2024-09-15T22:10:21.028+0000 7f2978f59640 -1 mds.a *** got signal Terminated *** 2024-09-15T22:10:22.671 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:22 smithi028 podman[112941]: 2024-09-15 22:10:22.482065331 +0000 UTC m=+1.553562801 container died ea2f62eab5bd5c34bf4a032d1df55f86ac3fa6e9d55a41e79d268d02fe8efb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:10:23.098 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:22 smithi028 podman[112941]: 2024-09-15 22:10:22.920531343 +0000 UTC m=+1.992028857 container cleanup ea2f62eab5bd5c34bf4a032d1df55f86ac3fa6e9d55a41e79d268d02fe8efb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-15T22:10:23.098 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:22 smithi028 bash[112941]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a 2024-09-15T22:10:23.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.a.service' 2024-09-15T22:10:23.349 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:23 smithi028 podman[112965]: 2024-09-15 22:10:23.096829022 +0000 UTC m=+0.617527678 container remove ea2f62eab5bd5c34bf4a032d1df55f86ac3fa6e9d55a41e79d268d02fe8efb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-15T22:10:23.350 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:23 smithi028 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.a.service: Deactivated successfully. 2024-09-15T22:10:23.350 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:23 smithi028 systemd[1]: Stopped Ceph mds.a for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T22:10:23.350 INFO:journalctl@ceph.mds.a.smithi028.stdout:Sep 15 22:10:23 smithi028 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.a.service: Consumed 2.107s CPU time. 2024-09-15T22:10:23.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:23.363 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-15T22:10:23.364 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-15T22:10:23.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.d 2024-09-15T22:10:23.671 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 22:10:23 smithi028 systemd[1]: Stopping Ceph mds.d for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:24.171 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 22:10:23 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d[70423]: 2024-09-15T22:10:23.704+0000 7effc4e61640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:24.172 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 22:10:23 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d[70423]: 2024-09-15T22:10:23.704+0000 7effc4e61640 -1 mds.d *** got signal Terminated *** 2024-09-15T22:10:26.171 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 22:10:25 smithi028 podman[113071]: 2024-09-15 22:10:25.871177873 +0000 UTC m=+2.276597573 container died 8a8c18c74584c311d8862283477e8fd46cc49349160f12eb181c937b8c20bea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-15T22:10:26.619 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 22:10:26 smithi028 podman[113071]: 2024-09-15 22:10:26.357869971 +0000 UTC m=+2.763289669 container cleanup 8a8c18c74584c311d8862283477e8fd46cc49349160f12eb181c937b8c20bea4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T22:10:26.619 INFO:journalctl@ceph.mds.d.smithi028.stdout:Sep 15 22:10:26 smithi028 bash[113071]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-d 2024-09-15T22:10:26.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.d.service' 2024-09-15T22:10:26.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:26.837 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-15T22:10:26.837 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-15T22:10:26.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.g 2024-09-15T22:10:27.171 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:10:26 smithi028 systemd[1]: Stopping Ceph mds.g for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:27.671 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:10:27 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g[71303]: 2024-09-15T22:10:27.178+0000 7fa9a0ccd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:27.671 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:10:27 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g[71303]: 2024-09-15T22:10:27.178+0000 7fa9a0ccd640 -1 mds.g *** got signal Terminated *** 2024-09-15T22:10:30.747 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:10:30 smithi028 podman[113195]: 2024-09-15 22:10:30.473957134 +0000 UTC m=+3.405804109 container died 41ea9524f6700a2e2c40075c8c3ddf68bf853518e1004819a70727acffc0f032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-15T22:10:31.089 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:10:30 smithi028 podman[113195]: 2024-09-15 22:10:30.836524011 +0000 UTC m=+3.768370984 container cleanup 41ea9524f6700a2e2c40075c8c3ddf68bf853518e1004819a70727acffc0f032 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-15T22:10:31.089 INFO:journalctl@ceph.mds.g.smithi028.stdout:Sep 15 22:10:30 smithi028 bash[113195]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-g 2024-09-15T22:10:31.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.g.service' 2024-09-15T22:10:31.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:31.280 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-15T22:10:31.280 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-15T22:10:31.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.j 2024-09-15T22:10:31.617 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 22:10:31 smithi028 systemd[1]: Stopping Ceph mds.j for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:31.921 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 22:10:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j[70031]: 2024-09-15T22:10:31.615+0000 7f64240b7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:31.921 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 22:10:31 smithi028 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j[70031]: 2024-09-15T22:10:31.615+0000 7f64240b7640 -1 mds.j *** got signal Terminated *** 2024-09-15T22:10:34.597 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 22:10:34 smithi028 podman[113316]: 2024-09-15 22:10:34.313437998 +0000 UTC m=+2.808441756 container died 1d6c8bef2d45e40cf905199b3901d0d48a0e38e99e2d42a7c2fc9dd9f05eca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-15T22:10:34.921 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 22:10:34 smithi028 podman[113316]: 2024-09-15 22:10:34.704574525 +0000 UTC m=+3.199578282 container cleanup 1d6c8bef2d45e40cf905199b3901d0d48a0e38e99e2d42a7c2fc9dd9f05eca6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-15T22:10:34.921 INFO:journalctl@ceph.mds.j.smithi028.stdout:Sep 15 22:10:34 smithi028 bash[113316]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-j 2024-09-15T22:10:35.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.j.service' 2024-09-15T22:10:35.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:35.131 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-15T22:10:35.131 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-15T22:10:35.131 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.b 2024-09-15T22:10:35.426 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:10:35 smithi174 systemd[1]: Stopping Ceph mds.b for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:35.682 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:10:35 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: 2024-09-15T22:10:35.423+0000 7f27e54df640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:35.682 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:10:35 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b[65971]: 2024-09-15T22:10:35.423+0000 7f27e54df640 -1 mds.b *** got signal Terminated *** 2024-09-15T22:10:38.677 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:10:38 smithi174 podman[73187]: 2024-09-15 22:10:38.396348824 +0000 UTC m=+3.074980020 container died 52e7bc84e2d02d59323bdfd87073224872f4e1b412507311a309a9f1051cf652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-15T22:10:39.010 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:10:38 smithi174 podman[73187]: 2024-09-15 22:10:38.78199305 +0000 UTC m=+3.460624238 container cleanup 52e7bc84e2d02d59323bdfd87073224872f4e1b412507311a309a9f1051cf652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-15T22:10:39.010 INFO:journalctl@ceph.mds.b.smithi174.stdout:Sep 15 22:10:38 smithi174 bash[73187]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-b 2024-09-15T22:10:39.221 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.b.service' 2024-09-15T22:10:39.258 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:39.258 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-15T22:10:39.258 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-15T22:10:39.258 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.e 2024-09-15T22:10:39.578 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:10:39 smithi174 systemd[1]: Stopping Ceph mds.e for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:40.010 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:10:39 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e[65351]: 2024-09-15T22:10:39.577+0000 7f0fa4468640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:40.010 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:10:39 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e[65351]: 2024-09-15T22:10:39.577+0000 7f0fa4468640 -1 mds.e *** got signal Terminated *** 2024-09-15T22:10:42.670 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:10:42 smithi174 podman[73319]: 2024-09-15 22:10:42.393962557 +0000 UTC m=+2.920121654 container died 5a42363735971110ab1ff881b1d2c804e1fa3adf9518d9000e554d3959fccba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-15T22:10:43.010 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:10:42 smithi174 podman[73319]: 2024-09-15 22:10:42.768590223 +0000 UTC m=+3.294749366 container cleanup 5a42363735971110ab1ff881b1d2c804e1fa3adf9518d9000e554d3959fccba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-15T22:10:43.010 INFO:journalctl@ceph.mds.e.smithi174.stdout:Sep 15 22:10:42 smithi174 bash[73319]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-e 2024-09-15T22:10:43.208 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.e.service' 2024-09-15T22:10:43.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:43.242 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-15T22:10:43.242 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-15T22:10:43.242 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.h 2024-09-15T22:10:43.565 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:10:43 smithi174 systemd[1]: Stopping Ceph mds.h for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:44.010 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:10:43 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h[65655]: 2024-09-15T22:10:43.563+0000 7f8e7d09c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:44.010 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:10:43 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h[65655]: 2024-09-15T22:10:43.563+0000 7f8e7d09c640 -1 mds.h *** got signal Terminated *** 2024-09-15T22:10:46.510 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:10:46 smithi174 podman[73437]: 2024-09-15 22:10:46.389046561 +0000 UTC m=+2.921509076 container died 3fe66e52248602e426bee1abcf1f9d7019ceb8a47aafae54403795dfbbd09aa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-15T22:10:47.010 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:10:46 smithi174 podman[73437]: 2024-09-15 22:10:46.805807198 +0000 UTC m=+3.338269713 container cleanup 3fe66e52248602e426bee1abcf1f9d7019ceb8a47aafae54403795dfbbd09aa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-15T22:10:47.010 INFO:journalctl@ceph.mds.h.smithi174.stdout:Sep 15 22:10:46 smithi174 bash[73437]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-h 2024-09-15T22:10:47.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.h.service' 2024-09-15T22:10:47.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:47.197 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-15T22:10:47.197 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-15T22:10:47.198 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.k 2024-09-15T22:10:47.760 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:10:47 smithi174 systemd[1]: Stopping Ceph mds.k for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:47.760 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:10:47 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k[65038]: 2024-09-15T22:10:47.517+0000 7f4646037640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:47.760 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:10:47 smithi174 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k[65038]: 2024-09-15T22:10:47.517+0000 7f4646037640 -1 mds.k *** got signal Terminated *** 2024-09-15T22:10:50.685 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:10:50 smithi174 podman[73557]: 2024-09-15 22:10:50.40386678 +0000 UTC m=+2.992005415 container died 6eb2bd6a32e1cbaa70b209b8ded4ee6e90e4a34b439da74418349e4200a895e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e) 2024-09-15T22:10:51.010 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:10:50 smithi174 podman[73557]: 2024-09-15 22:10:50.76705543 +0000 UTC m=+3.355194055 container cleanup 6eb2bd6a32e1cbaa70b209b8ded4ee6e90e4a34b439da74418349e4200a895e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:10:51.010 INFO:journalctl@ceph.mds.k.smithi174.stdout:Sep 15 22:10:50 smithi174 bash[73557]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-k 2024-09-15T22:10:51.194 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.k.service' 2024-09-15T22:10:51.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:51.227 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-15T22:10:51.227 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-15T22:10:51.227 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.c 2024-09-15T22:10:51.530 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:51 smithi203 systemd[1]: Stopping Ceph mds.c for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:51.968 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:51 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c[65725]: 2024-09-15T22:10:51.528+0000 7f038429d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:51.968 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:51 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c[65725]: 2024-09-15T22:10:51.528+0000 7f038429d640 -1 mds.c *** got signal Terminated *** 2024-09-15T22:10:54.673 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:54 smithi203 podman[73282]: 2024-09-15 22:10:54.385827276 +0000 UTC m=+2.963499022 container died 5e9426b12798c48eb727c394d25a65cc079de29c3aa72b015b7da837a2151911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-15T22:10:54.968 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:54 smithi203 podman[73282]: 2024-09-15 22:10:54.829326021 +0000 UTC m=+3.406997768 container cleanup 5e9426b12798c48eb727c394d25a65cc079de29c3aa72b015b7da837a2151911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-8921c8e) 2024-09-15T22:10:54.968 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:54 smithi203 bash[73282]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c 2024-09-15T22:10:55.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.c.service' 2024-09-15T22:10:55.276 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:55 smithi203 podman[73307]: 2024-09-15 22:10:55.022411325 +0000 UTC m=+0.634780127 container remove 5e9426b12798c48eb727c394d25a65cc079de29c3aa72b015b7da837a2151911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T22:10:55.276 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:55 smithi203 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.c.service: Deactivated successfully. 2024-09-15T22:10:55.276 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:55 smithi203 systemd[1]: Stopped Ceph mds.c for e78b7764-73ab-11ef-bceb-c7b262605968. 2024-09-15T22:10:55.276 INFO:journalctl@ceph.mds.c.smithi203.stdout:Sep 15 22:10:55 smithi203 systemd[1]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.c.service: Consumed 2.073s CPU time. 2024-09-15T22:10:55.290 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:55.290 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-15T22:10:55.290 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-15T22:10:55.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.f 2024-09-15T22:10:55.626 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:10:55 smithi203 systemd[1]: Stopping Ceph mds.f for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:55.968 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:10:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f[65422]: 2024-09-15T22:10:55.624+0000 7f2d2171a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:55.968 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:10:55 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f[65422]: 2024-09-15T22:10:55.624+0000 7f2d2171a640 -1 mds.f *** got signal Terminated *** 2024-09-15T22:10:58.661 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:10:58 smithi203 podman[73417]: 2024-09-15 22:10:58.400316122 +0000 UTC m=+2.879736897 container died ca2aca01a6851e360c01779d61cf4f1c32f97196b6bca9b38ed2ac0f9ca2874b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_CLEAN=True) 2024-09-15T22:10:58.967 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:10:58 smithi203 podman[73417]: 2024-09-15 22:10:58.758467744 +0000 UTC m=+3.237888518 container cleanup ca2aca01a6851e360c01779d61cf4f1c32f97196b6bca9b38ed2ac0f9ca2874b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2) 2024-09-15T22:10:58.968 INFO:journalctl@ceph.mds.f.smithi203.stdout:Sep 15 22:10:58 smithi203 bash[73417]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-f 2024-09-15T22:10:59.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.f.service' 2024-09-15T22:10:59.265 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:10:59.265 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-15T22:10:59.265 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-15T22:10:59.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.i 2024-09-15T22:10:59.580 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:10:59 smithi203 systemd[1]: Stopping Ceph mds.i for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:10:59.968 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:10:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i[66025]: 2024-09-15T22:10:59.578+0000 7f38ee789640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:10:59.968 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:10:59 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i[66025]: 2024-09-15T22:10:59.578+0000 7f38ee789640 -1 mds.i *** got signal Terminated *** 2024-09-15T22:11:02.714 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:11:02 smithi203 podman[73537]: 2024-09-15 22:11:02.392339347 +0000 UTC m=+2.908322607 container died 5fa2b0e44dc8cd6406dd210017918e251e18098b87e11924058994aef07c95f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-15T22:11:02.968 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:11:02 smithi203 podman[73537]: 2024-09-15 22:11:02.796276366 +0000 UTC m=+3.312259634 container cleanup 5fa2b0e44dc8cd6406dd210017918e251e18098b87e11924058994aef07c95f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-15T22:11:02.968 INFO:journalctl@ceph.mds.i.smithi203.stdout:Sep 15 22:11:02 smithi203 bash[73537]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-i 2024-09-15T22:11:03.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.i.service' 2024-09-15T22:11:03.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:11:03.198 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-15T22:11:03.198 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-15T22:11:03.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.l 2024-09-15T22:11:03.468 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:11:03 smithi203 systemd[1]: Stopping Ceph mds.l for e78b7764-73ab-11ef-bceb-c7b262605968... 2024-09-15T22:11:03.820 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:11:03 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l[65122]: 2024-09-15T22:11:03.516+0000 7fc2f7830640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T22:11:03.820 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:11:03 smithi203 ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l[65122]: 2024-09-15T22:11:03.516+0000 7fc2f7830640 -1 mds.l *** got signal Terminated *** 2024-09-15T22:11:06.693 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:11:06 smithi203 podman[73659]: 2024-09-15 22:11:06.384699646 +0000 UTC m=+2.974534511 container died 7678aedb8bc26dbf6ad8078c4405550c1d7285855a397fac3c36ef0620397917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-15T22:11:06.968 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:11:06 smithi203 podman[73659]: 2024-09-15 22:11:06.775350499 +0000 UTC m=+3.365185367 container cleanup 7678aedb8bc26dbf6ad8078c4405550c1d7285855a397fac3c36ef0620397917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e) 2024-09-15T22:11:06.968 INFO:journalctl@ceph.mds.l.smithi203.stdout:Sep 15 22:11:06 smithi203 bash[73659]: ceph-e78b7764-73ab-11ef-bceb-c7b262605968-mds-l 2024-09-15T22:11:07.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e78b7764-73ab-11ef-bceb-c7b262605968@mds.l.service' 2024-09-15T22:11:07.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T22:11:07.189 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-15T22:11:07.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e78b7764-73ab-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T22:11:07.403 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T22:11:10.084 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e78b7764-73ab-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T22:11:10.287 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T22:11:12.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e78b7764-73ab-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T22:11:13.010 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T22:11:15.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T22:11:15.573 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T22:11:15.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T22:11:15.633 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-15T22:11:15.635 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi028/crash 2024-09-15T22:11:15.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash -- . 2024-09-15T22:11:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T22:11:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-09-15T22:11:15.682 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi174/crash 2024-09-15T22:11:15.683 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash -- . 2024-09-15T22:11:15.721 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T22:11:15.721 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-09-15T22:11:15.723 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi203/crash 2024-09-15T22:11:15.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash -- . 2024-09-15T22:11:15.782 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T22:11:15.782 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-09-15T22:11:15.783 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-15T22:11:15.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-15T22:11:15.872 INFO:tasks.cephadm:Compressing logs... 2024-09-15T22:11:15.873 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T22:11:15.875 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T22:11:15.877 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T22:11:15.907 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-15T22:11:15.908 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.10.log 2024-09-15T22:11:15.909 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-15T22:11:15.910 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-15T22:11:15.910 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T22:11:15.910 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.b.log 2024-09-15T22:11:15.911 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.k.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.h.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.e.log 2024-09-15T22:11:15.914 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T22:11:15.916 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T22:11:15.916 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T22:11:15.916 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.i.log 2024-09-15T22:11:15.916 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-15T22:11:15.925 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T22:11:15.925 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-15T22:11:15.925 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-15T22:11:15.925 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-15T22:11:15.925 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log 2024-09-15T22:11:15.925 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.l.log 2024-09-15T22:11:15.925 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-15T22:11:15.926 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.z.log 2024-09-15T22:11:15.926 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.c.log 2024-09-15T22:11:15.926 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.f.log 2024-09-15T22:11:15.926 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T22:11:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/ceph-client.admin.log 2024-09-15T22:11:15.927 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-15T22:11:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-15T22:11:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-15T22:11:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.log: /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T22:11:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T22:11:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.a.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.g.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.g.log: /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.d.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.j.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T22:11:15.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 86.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T22:11:15.929 INFO:teuthology.orchestra.run.smithi028.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-15T22:11:15.929 INFO:teuthology.orchestra.run.smithi028.stderr: 94.6% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.d.log.gz 2024-09-15T22:11:15.929 INFO:teuthology.orchestra.run.smithi028.stderr: 94.5% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.j.log.gz 2024-09-15T22:11:15.953 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T22:11:15.958 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.e.log: /var/log/ceph/cephadm.log: 81.8% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T22:11:15.973 INFO:teuthology.orchestra.run.smithi028.stderr: 84.4% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T22:11:16.003 INFO:teuthology.orchestra.run.smithi174.stderr: 91.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-15T22:11:16.039 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.f.log: /var/log/ceph/cephadm.log: 81.8% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T22:11:16.063 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T22:11:16.074 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T22:11:16.121 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T22:11:16.124 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T22:11:16.143 INFO:teuthology.orchestra.run.smithi203.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T22:11:16.227 INFO:teuthology.orchestra.run.smithi028.stderr: 94.2% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.g.log.gz 2024-09-15T22:11:16.340 INFO:teuthology.orchestra.run.smithi028.stderr: 94.0% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.a.log.gz 2024-09-15T22:11:16.356 INFO:teuthology.orchestra.run.smithi203.stderr: 94.0% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.l.log.gz 2024-09-15T22:11:16.496 INFO:teuthology.orchestra.run.smithi203.stderr: 86.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T22:11:16.572 INFO:teuthology.orchestra.run.smithi203.stderr: 90.4% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T22:11:16.612 INFO:teuthology.orchestra.run.smithi203.stderr: 92.0% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.z.log.gz 2024-09-15T22:11:16.661 INFO:teuthology.orchestra.run.smithi203.stderr: 92.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.c.log.gz 2024-09-15T22:11:16.718 INFO:teuthology.orchestra.run.smithi174.stderr: 91.0% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T22:11:16.729 INFO:teuthology.orchestra.run.smithi174.stderr: 86.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T22:11:16.736 INFO:teuthology.orchestra.run.smithi174.stderr: 93.2% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.h.log.gz 2024-09-15T22:11:16.741 INFO:teuthology.orchestra.run.smithi203.stderr: 92.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.f.log.gz 2024-09-15T22:11:17.042 INFO:teuthology.orchestra.run.smithi028.stderr: 94.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T22:11:17.224 INFO:teuthology.orchestra.run.smithi203.stderr: 92.6% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.i.log.gz 2024-09-15T22:11:17.448 INFO:teuthology.orchestra.run.smithi174.stderr: 94.5% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.k.log.gz 2024-09-15T22:11:17.928 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-15T22:11:18.693 INFO:teuthology.orchestra.run.smithi203.stderr: 91.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-15T22:11:20.302 INFO:teuthology.orchestra.run.smithi028.stderr: 90.4% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-15T22:11:24.740 INFO:teuthology.orchestra.run.smithi174.stderr: 94.2% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.b.log.gz 2024-09-15T22:11:30.025 INFO:teuthology.orchestra.run.smithi028.stderr: 90.4% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-15T22:11:35.920 INFO:teuthology.orchestra.run.smithi174.stderr: 95.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-mds.e.log.gz 2024-09-15T22:14:15.546 INFO:teuthology.orchestra.run.smithi174.stderr: 92.5% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-15T22:16:31.700 INFO:teuthology.orchestra.run.smithi174.stderr: 92.7% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-15T22:16:39.852 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-15T22:17:39.663 INFO:teuthology.orchestra.run.smithi174.stderr: 92.9% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-15T22:18:39.043 INFO:teuthology.orchestra.run.smithi174.stderr: 92.1% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-15T22:18:39.045 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-15T22:18:39.045 INFO:teuthology.orchestra.run.smithi174.stderr:real 7m23.153s 2024-09-15T22:18:39.045 INFO:teuthology.orchestra.run.smithi174.stderr:user 5m56.433s 2024-09-15T22:18:39.045 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m13.597s 2024-09-15T22:20:00.232 INFO:teuthology.orchestra.run.smithi203.stderr: 92.5% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-15T22:20:10.422 INFO:teuthology.orchestra.run.smithi203.stderr: 92.7% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-15T22:20:40.357 INFO:teuthology.orchestra.run.smithi028.stderr: 92.5% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-15T22:21:22.719 INFO:teuthology.orchestra.run.smithi203.stderr: 92.2% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-15T22:21:22.721 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T22:21:22.721 INFO:teuthology.orchestra.run.smithi203.stderr:real 10m6.829s 2024-09-15T22:21:22.721 INFO:teuthology.orchestra.run.smithi203.stderr:user 6m3.110s 2024-09-15T22:21:22.721 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m14.485s 2024-09-15T22:27:43.176 INFO:teuthology.orchestra.run.smithi028.stderr: 92.4% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-15T22:28:31.956 INFO:teuthology.orchestra.run.smithi028.stderr: 92.2% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-15T22:29:17.787 INFO:teuthology.orchestra.run.smithi028.stderr: 92.1% -- replaced with /var/log/ceph/e78b7764-73ab-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-15T22:29:17.822 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-09-15T22:29:17.823 INFO:teuthology.orchestra.run.smithi028.stderr:real 18m1.932s 2024-09-15T22:29:17.823 INFO:teuthology.orchestra.run.smithi028.stderr:user 8m4.517s 2024-09-15T22:29:17.823 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m20.003s 2024-09-15T22:29:17.823 INFO:tasks.cephadm:Archiving logs... 2024-09-15T22:29:17.825 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi028/log 2024-09-15T22:29:17.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T22:30:27.589 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi174/log 2024-09-15T22:30:27.590 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T22:31:22.863 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi203/log 2024-09-15T22:31:22.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T22:32:20.140 INFO:tasks.cephadm:Removing cluster... 2024-09-15T22:32:20.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e78b7764-73ab-11ef-bceb-c7b262605968 --force 2024-09-15T22:32:20.475 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T22:32:21.234 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e78b7764-73ab-11ef-bceb-c7b262605968 --force 2024-09-15T22:32:21.438 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T22:32:22.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e78b7764-73ab-11ef-bceb-c7b262605968 --force 2024-09-15T22:32:22.282 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: e78b7764-73ab-11ef-bceb-c7b262605968 2024-09-15T22:32:22.941 INFO:tasks.cephadm:Removing cephadm ... 2024-09-15T22:32:22.941 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T22:32:22.971 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T22:32:22.991 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T22:32:23.007 INFO:tasks.cephadm:Teardown complete 2024-09-15T22:32:23.007 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T22:32:23.019 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T22:32:23.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T22:32:23.022 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T22:32:23.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T22:32:23.377 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T22:32:23.388 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T22:32:23.388 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T22:32:23.391 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T22:32:23.393 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T22:32:23.406 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-15T22:32:23.408 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-09-15T22:32:23.409 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-09-15T22:32:23.433 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T22:32:23.433 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-09-15T22:32:23.433 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +1088us[+1102us] +/- 40ms 2024-09-15T22:32:23.434 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 36 -945us[ -931us] +/- 51ms 2024-09-15T22:32:23.434 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -602us[ -588us] +/- 62ms 2024-09-15T22:32:23.434 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T22:32:23.435 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T22:32:23.435 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-09-15T22:32:23.435 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv01.front.sepia.ceph.com 3 6 377 33 +1091us[+1095us] +/- 40ms 2024-09-15T22:32:23.435 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -958us[ -954us] +/- 51ms 2024-09-15T22:32:23.436 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 -630us[ -630us] +/- 62ms 2024-09-15T22:32:23.436 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T22:32:23.483 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T22:32:23.484 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-15T22:32:23.484 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 36 +1077us[+1090us] +/- 40ms 2024-09-15T22:32:23.484 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 -967us[ -953us] +/- 51ms 2024-09-15T22:32:23.484 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 95 -625us[ -609us] +/- 61ms 2024-09-15T22:32:23.484 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T22:32:23.503 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T22:32:23.514 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T22:32:23.515 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T22:32:23.525 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T22:32:23.707 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T22:32:23.820 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T22:32:23.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T22:32:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:32:24.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T22:32:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:32:24.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T22:32:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:32:24.091 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T22:32:24.102 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T22:32:24.113 INFO:teuthology.task.internal:Duration was 3130.934183 seconds 2024-09-15T22:32:24.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T22:32:24.124 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T22:32:24.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T22:32:24.127 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T22:32:24.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T22:32:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T22:32:24.200 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T22:32:24.213 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T22:32:24.567 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T22:32:24.567 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-09-15T22:32:24.567 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T22:32:24.593 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-09-15T22:32:24.593 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T22:32:24.623 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T22:32:24.623 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T22:32:24.652 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T22:32:24.652 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T22:32:24.655 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T22:32:24.666 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T22:32:24.719 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T22:32:24.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T22:32:24.731 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T22:32:24.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T22:32:25.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T22:32:25.877 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T22:32:25.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T22:32:25.954 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T22:32:25.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T22:32:26.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T22:32:26.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:32:26.051 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:32:26.053 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:32:26.080 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-09-15T22:32:26.088 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-09-15T22:32:26.117 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-09-15T22:32:26.137 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:32:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:32:26.174 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:32:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:32:26.189 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:32:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:32:26.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T22:32:26.217 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T22:32:26.218 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi028 2024-09-15T22:32:26.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T22:32:26.279 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi174 2024-09-15T22:32:26.279 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T22:32:26.316 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906192/remote/smithi203 2024-09-15T22:32:26.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T22:32:26.354 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T22:32:26.355 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T22:32:26.357 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T22:32:26.359 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T22:32:26.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T22:32:26.420 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T22:32:26.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T22:32:26.432 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T22:32:26.432 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T22:32:26.434 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T22:32:26.437 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T22:32:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 22:32 /home/ubuntu/cephtest 2024-09-15T22:32:26.450 INFO:teuthology.orchestra.run.smithi174.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 22:32 /home/ubuntu/cephtest 2024-09-15T22:32:26.462 INFO:teuthology.orchestra.run.smithi203.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 22:32 /home/ubuntu/cephtest 2024-09-15T22:32:26.472 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T22:32:26.489 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T22:32:26.865 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} duration: 3130.934183359146 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-15T22:32:26.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T22:32:26.953 INFO:teuthology.run:pass